single-file-core 1.6.0 → 1.6.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/core/helper.js +5 -2
  2. package/modules/css-fonts-minifier.js +34 -13
  3. package/modules/template-formatter.js +8 -2
  4. package/package.json +10 -2
  5. package/vendor/zip/zip.js +4 -0
  6. package/.github/workflows/publish.yml +0 -88
  7. package/.github/workflows/test.yml +0 -31
  8. package/.github/workflows/vendor.yml +0 -39
  9. package/css-tree-build/README.md +0 -39
  10. package/css-tree-build/build.js +0 -66
  11. package/css-tree-build/package-lock.json +0 -511
  12. package/css-tree-build/package.json +0 -13
  13. package/deno.lock +0 -19
  14. package/doc/assets/singlefile-archive-byte-map.svg +0 -276
  15. package/doc/assets/singlefile-archive-lenses.svg +0 -245
  16. package/doc/singlefile-archive.md +0 -1968
  17. package/eslint.config.mjs +0 -74
  18. package/test/capture/README.md +0 -74
  19. package/test/capture/canonical-link.js +0 -50
  20. package/test/capture/common.js +0 -73
  21. package/test/capture/deferred-content-options.js +0 -71
  22. package/test/capture/dom.js +0 -26
  23. package/test/capture/maff-metadata.js +0 -205
  24. package/test/capture/resource-cap.js +0 -79
  25. package/test/capture/script-uri-sanitization.js +0 -82
  26. package/test/fixtures/README.md +0 -18
  27. package/test/fixtures/classic-digest.html +0 -5
  28. package/test/fixtures/generate-fixtures.sh +0 -31
  29. package/test/fixtures/multi-page-dedup.zip.html +0 -70
  30. package/test/fixtures/multi-page.zip.html +0 -26
  31. package/test/fixtures/single-page.zip.html +0 -11
  32. package/test/fixtures/site/multi-page/alpha.html +0 -2
  33. package/test/fixtures/site/multi-page/beta.html +0 -2
  34. package/test/fixtures/site/multi-page/index.html +0 -2
  35. package/test/fixtures/site/multi-page/sub/gamma.html +0 -2
  36. package/test/fixtures/site/multi-page-dedup/index.html +0 -2
  37. package/test/fixtures/site/multi-page-dedup/logo.png +0 -0
  38. package/test/fixtures/site/multi-page-dedup/one.html +0 -2
  39. package/test/fixtures/site/multi-page-dedup/shared.css +0 -3
  40. package/test/fixtures/site/multi-page-dedup/two.html +0 -2
  41. package/test/run.js +0 -109
  42. package/test/sfz-harness/README.md +0 -87
  43. package/test/sfz-harness/adopted-stylesheets-hook.js +0 -240
  44. package/test/sfz-harness/byte-map.js +0 -137
  45. package/test/sfz-harness/charset-round-trip.js +0 -161
  46. package/test/sfz-harness/check-determinism.js +0 -57
  47. package/test/sfz-harness/common.js +0 -154
  48. package/test/sfz-harness/content-type-sniffing.js +0 -83
  49. package/test/sfz-harness/css-fonts-minifier.js +0 -235
  50. package/test/sfz-harness/css-property-filter.js +0 -113
  51. package/test/sfz-harness/dom-stub.js +0 -13
  52. package/test/sfz-harness/entry-compression.js +0 -77
  53. package/test/sfz-harness/filename-characters.js +0 -55
  54. package/test/sfz-harness/filename-max-length.js +0 -69
  55. package/test/sfz-harness/font-face-composite.js +0 -135
  56. package/test/sfz-harness/format-rules.js +0 -617
  57. package/test/sfz-harness/gen-e2e-page.js +0 -14
  58. package/test/sfz-harness/inlined-functions.js +0 -82
  59. package/test/sfz-harness/option-wiring.js +0 -60
  60. package/test/sfz-harness/pages-archive.js +0 -318
  61. package/test/sfz-harness/pages-router.js +0 -143
  62. package/test/sfz-harness/relocation-cost.js +0 -94
  63. package/test/sfz-harness/search-triggers.js +0 -35
  64. package/test/sfz-harness/smoke.js +0 -10
  65. package/test/sfz-harness/stored-trigger.js +0 -95
  66. package/test/sfz-harness/trigger-seeds.json +0 -120
  67. package/test/sfz-harness/zip64.js +0 -77
  68. package/zip-build/README.md +0 -56
  69. package/zip-build/lib/fflate-streams.js +0 -47
  70. package/zip-build/lib/fflate.js +0 -2692
  71. package/zip-build/lib/zip-vendor-worker.js +0 -3
  72. package/zip-build/lib/zip-vendor.js +0 -3
  73. package/zip-build/lib/zip.js +0 -25
  74. package/zip-build/package-lock.json +0 -665
  75. package/zip-build/package.json +0 -15
  76. package/zip-build/reserved-property-names.json +0 -10630
  77. package/zip-build/rollup.config.js +0 -61
@@ -1,1968 +0,0 @@
1
- # The SingleFile archive format
2
-
3
- **Status: draft.** This document specifies the SingleFile archive, the polyglot file
4
- format produced by [SingleFile](https://github.com/gildas-lormeau/SingleFile) when it
5
- saves a page as a ZIP archive. It is written against the reference implementation,
6
- [single-file-core](https://github.com/gildas-lormeau/single-file-core) 1.5.120
7
- (`processors/compression/`), and every byte-level statement has been verified on
8
- generated specimen files.
9
-
10
- The key words MUST, MUST NOT, SHOULD and MAY are to be interpreted as
11
- described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119) when, and only when,
12
- they appear in all capitals.
13
-
14
- Three kinds of statement appear throughout:
15
-
16
- - **Format requirements**, in RFC 2119 capitals. Their subject is a writer producing an
17
- archive, or a *conforming reader of this format* — third-party software that reads
18
- SingleFile archives (§7). It is never the stock HTML, ZIP, PDF or PNG readers the
19
- document describes: those cannot be placed under an obligation by this document, and
20
- a requirement that appears to do so is describing what a writer must produce so that
21
- their existing behavior lands correctly (§4).
22
- - **Reference behavior** — what `createArchive()` does where the format allows other
23
- choices. Always marked as the reference writer's, and never a requirement.
24
- - **Measurements** — observed behavior of specific third-party software, with the
25
- version and the specimen it was measured on. §8 collects them; where one appears
26
- inline it is the evidence for a claim, not a guarantee about software in general.
27
-
28
- ## 1. Introduction
29
-
30
- A SingleFile archive is **one byte string that is simultaneously a valid document in
31
- several formats**. Every archive is a valid ZIP file containing the saved page and its
32
- resources. Depending on the options used to produce it, the same byte string is also:
33
-
34
- - a valid **HTML page** that extracts and displays the archived page when opened in a
35
- browser, with no external dependency;
36
- - a valid **PNG image**, typically a screenshot of the page, though the format does not
37
- require the image to depict it;
38
- - a valid **PDF document**, typically a rendering of the page, though the format does
39
- not require the document to depict it either.
40
-
41
- Each format's reader accepts the file as a complete document of its own format and
42
- silently ignores the bytes that belong to the other formats. The word is *accepts*,
43
- not *conforms*: several faces lean on reader tolerances the target standards do not
44
- promise (§1.1). The large payloads are stored once and
45
- shared: the archive entries, the PDF document and the PNG pixel data are single
46
- regions that several readers reach, not per-face copies. The polyglot works by
47
- *partitioning* the file into regions and arranging each region so that every reader
48
- either interprets it or skips it. What the optional features add is never a copy of
49
- another face's payload: the text body (an optional plain-text copy of the page stored
50
- in the HTML face for text tools and indexers, §4.6) repeats the page text, and the
51
- image and PDF a writer supplies are separate documents, whether or not they depict the
52
- archived page.
53
-
54
- ### 1.1 Design goals
55
-
56
- The format exists to keep saved pages readable for as long as possible, with as
57
- little software as possible. Each way of opening the file has a simpler fallback:
58
-
59
- 1. In a JavaScript-capable browser, the file opens and displays the page.
60
- 2. When extraction fails, the file displays an error message with recovery
61
- instructions. Without JavaScript, it renders as a blank page: the document body
62
- is hidden by construction, so the browser displays neither the page nor raw
63
- archive bytes (§4.1).
64
- 3. Renamed to `.zip`, the file opens in a ZIP tool; the page and each resource are
65
- ordinary entries. Two measured readers refuse a self-extracting variant even from
66
- seekable input; the ranking below names them. A forward-only reader refuses it too,
67
- and is a non-goal (§1.2).
68
- 4. Renamed to `.pdf` or `.png` (when those faces are present), the file opens in a PDF
69
- viewer or an image viewer.
70
-
71
- Three consequences shape everything below:
72
-
73
- - **The HTML face depends on the ZIP face.** The HTML bootstrap extracts the page
74
- *from the ZIP structure of its own file*; the ZIP face is not an export feature, it
75
- is the storage layer the HTML face reads from.
76
- - **Readers must need no cooperation.** Every face works with stock, unmodified
77
- readers. The format relies on two kinds of reader behavior: rules the target
78
- format actually defines (such as HTML's parsing and error-recovery rules, §4.1),
79
- and *customary tolerances* — behaviors that are near-universal in practice but
80
- that no standard promises, such as the backward scan ZIP readers use to find the
81
- End Of Central Directory record (§4.2, §5.2), the PDF header scan and
82
- trailing-data tolerance (§4.3), and PNG decoders' indifference to what an ancillary
83
- `tEXt` chunk contains (§4.4). The compatibility appendix records the customary
84
- tolerances' real-world support (§8).
85
- - **The faces are not equally durable.** They rest on different amounts of unpromised
86
- behavior, and "the file is valid in four formats" is not four guarantees. A decision
87
- about what to rely on years from now should follow this order:
88
- 1. **The ZIP face** is the one to trust, and the only one the format calls storage.
89
- A variant with no other face is an ordinary ZIP file. The self-extracting variants
90
- add prepended and appended bytes, which cost two measured readers: `ditto`, which
91
- requires a local header at offset 0, and `java.util.zip`, which rejects undeclared
92
- trailing bytes. A writer can fix the second case by declaring those bytes as the
93
- archive comment (§4.2).
94
- 2. **The HTML face** rests mostly on *normative* behavior: HTML's tokenizer states
95
- and error recovery are specified, and the blank-page backstop of §4.1 is ordinary
96
- CSS. It is exposed on two other counts: it depends on the ZIP face beneath it, and
97
- a browser must run its script for the page to appear.
98
- 3. **The PDF and PNG faces are conveniences.** Both rest entirely on tolerances no
99
- standard promises: the PDF header scan, where PDFium already enforces its
100
- 1024-byte window exactly with no margin, and PNG decoders' indifference to a
101
- `tEXt` chunk holding bytes the format does not permit. They are worth having
102
- because they cost nothing the other faces need. For preservation purposes they
103
- are exports that happen to share the file; they are not archival copies.
104
-
105
- ### 1.2 Non-goals
106
-
107
- - **Forward-only ZIP parsers.** Every archive with a face requires central-directory-driven
108
- reading, since the entries are then preceded by non-ZIP bytes. The variant with no other
109
- face is an ordinary ZIP file and streams from offset 0 like any other (§8.1, class A);
110
- parsers that require that are out of scope for the rest (§7).
111
- - **In-place modification by generic ZIP tools.** The face invariants are global:
112
- the writer picks each hiding tag only after checking the exact bytes it must hide,
113
- the recovery payload of universal mode contains a checksum of the ZIP region
114
- without its comment-length field, and the PDF and PNG structures wrap the
115
- archive (§5.4). A tool
116
- that adds, removes or recompresses entries invalidates them, and most rewriters drop
117
- the prepended and appended regions outright. A generically rewritten file keeps at
118
- best its ZIP face. Editing an archive means producing a new one through the writer
119
- rules (§6).
120
- - **Multi-page archives.** The reference implementation can bundle several saved
121
- pages into one archive behind a routing bootstrap (`multiPageArchive`). This
122
- version of the document specifies single-page archives only; the multi-page
123
- layout is out of scope, and so are the regions it adds to the prologue, which this
124
- document does not describe.
125
- - **Confidentiality outside the ZIP entries.** A password encrypts ZIP entry contents
126
- only (AES). The PDF and PNG faces render the page content and are plaintext by
127
- design; the writer withholds what it can without breaking a face, as described in
128
- §5.6. The embedded PDF MAY itself be a PDF-encrypted document, since the format is
129
- agnostic to the PDF's content, but the reference writer does not produce one.
130
-
131
- ### 1.3 Terminology
132
-
133
- | Term | Meaning |
134
- |---|---|
135
- | **face** | One of the formats the file is valid in: HTML, ZIP, PNG, PDF. |
136
- | **region** | A byte range with a single producer, named in §3. Regions are the units the rest of this document reasons about; a region can appear in several pieces — `html-prologue` resumes after the embedded PDF document in the PDF variants, and after the `tEXt "ZIP"` chunk header in the PNG ones, so with all four faces it comes in three. |
137
- | **universal mode** | The variant whose HTML face can extract the archive from the *parsed page text*, the text and comment nodes the HTML parser produced, and therefore needs no access to its own raw bytes. Named "universal" because it works from any location, including the `file:` protocol. |
138
- | **wrapper tag** | The HTML construct that hides a binary region from the HTML parser, `<!--`…`-->` by default (§5.1). |
139
- | **appended data** | Bytes after the ZIP End Of Central Directory record. A reader tolerates them as far back as its EOCD scan reaches, and how far that is varies by an order of magnitude: 65557 bytes from the end of the file for Python `zipfile` (the 22-byte record plus the 65535-byte maximum comment length), but 16383 for libarchive and 32768 for perl `Archive::Zip` (§8.1). No reader's window is guaranteed, so a writer keeps its own narrower budget (§5.2). The format's one hard limit is the 65535-byte comment field, and it binds only a run the writer declares (§4.2). It may be left undeclared or declared as the archive comment; both forms are valid ZIP and readers MUST accept both (§4.2). The recovery payload can be computed before that choice is made because it stops two bytes short of the record, excluding its comment-length field (see *recovered range* below). |
140
- | **ZIP region** | The contiguous byte range holding the archive proper: from the first local file header the ZIP writer emitted through the last byte of the End Of Central Directory record. It spans the `zip-entries`, `pdf-central-record` (when present) and `central-directory · eocd` blocks of §3, and in the HTML variants it is the content of the last wrapper, exactly so on the element rungs and preceded by the `sfz-data` identifier on the comment rung, which the extractor steps over. It does **not** include `pdf-local-header` or the PDF document, which sit earlier in the file. |
141
- | **archive** | The *logical* ZIP file: the set of entries the central directory describes, wherever their bytes lie. This is distinct from the ZIP region above, which is a contiguous byte range. Every entry but one has its bytes inside the region; `page.pdf` is the deliberate exception, an entry of the archive whose local header and data sit before the region (§4.2). "Archive" in this document always means the logical file, "ZIP region" always the byte range, and the two differ only in the PDF-with-HTML variants. |
142
- | **recovered range** | What the universal extractor reproduces (§4.5): the ZIP region minus its last two bytes, the comment-length field of the End Of Central Directory record. That field is the one part of the record whose value depends on what follows the region, so leaving it out is what lets a writer decide the appended-data form after the recovery payload is final (§4.2). The extractor supplies the two bytes itself, as zeroes — the recovered range carries no comment. |
143
- | **reference writer** | `createArchive()` in single-file-core `processors/compression/compression.js`. |
144
- | **bootstrap** | The inline script in the HTML face that locates, extracts and displays the archived page. |
145
- | **`sfz` identifiers** | Three byte-level identifiers carrying the `sfz` prefix matter to this document: `data-sfz`, `<sfz-extra-data>` and `sfz-data`. A conforming file may hold others the format says nothing about — the reference bootstrap gives its own status messages `sfz`-prefixed ids, which no reader has any reason to look for. The prefix is inherited from SingleFileZ, the browser extension the format originated in (since merged into SingleFile), and is kept unchanged for compatibility with existing files. They are wire identifiers, not the format's name. Two of the three have a role in the format: `<sfz-extra-data>` is the element carrying the recovery payload, and `sfz-data` is the identifier the universal extractor addresses the ZIP region with — an `id` attribute on the wrapper element, or the first characters of the wrapper comment's data (§4.5). `data-sfz` is a marker the reference writer happens to put on the root element; this document mentions it only where it describes bytes those files contain. |
146
-
147
- ## 2. Variants: composing faces
148
-
149
- Every SingleFile archive has the ZIP face. The other faces are enabled independently by writer
150
- options, and compose. Each row of the table lists its complete option set; every row
151
- also implies `compressContent`, the option that stores the page and its resources as
152
- ZIP entries and so makes the output an archive at all:
153
-
154
- | Writer options (core names) | HTML | PNG | PDF | Extension | Specimen |
155
- |---|---|---|---|---|---|
156
- | *(none)* | — | — | — | `.zip` | pure zip |
157
- | `selfExtractingArchive` | ✔ | — | — | `.zip.html` | plain |
158
- | `selfExtractingArchive`, `extractDataFromPage` | ✔ universal | — | — | `.u.zip.html` | universal |
159
- | `selfExtractingArchive`, `extractDataFromPage`, `embeddedImage` | ✔ universal | ✔ | — | `.u.zip.html` | png |
160
- | `selfExtractingArchive`, `extractDataFromPage`, `embeddedPdf` | ✔ universal | — | ✔ | `.u.zip.html` | pdf |
161
- | `selfExtractingArchive`, `extractDataFromPage`, `embeddedImage`, `embeddedPdf` | ✔ universal | ✔ | ✔ | `.u.zip.html` | png-pdf |
162
- | `embeddedPdf` | — | — | ✔ | `.zip` | zip-pdf |
163
- | `embeddedImage` | — | ✔ | — | `.zip` | zip-png |
164
- | `embeddedImage`, `embeddedPdf` | — | ✔ | ✔ | `.zip` | zip-png-pdf |
165
-
166
- `extractDataFromPage` is orthogonal to the PNG and PDF faces. The PNG and PDF rows
167
- include it because the clients producing those variants enable it by default, but
168
- `embeddedImage` and `embeddedPdf` compose with a non-universal self-extracting file
169
- just as well; the extension is then `.zip.html`. The one interaction: `page.pdf` lies
170
- outside the ZIP region (§1.3), so the page-text extraction path does not recover it
171
- with the rest. The extractor therefore filters that entry out unconditionally, on
172
- every acquisition path including the ones that read raw bytes and could return it
173
- (§4.5). In the last three rows there is no
174
- HTML face, so the option does not apply. The *Specimen* column names the measured
175
- reference files this document cites; §8 records how to regenerate them.
176
-
177
- Other writer options shape the file without adding a face: `preventAppendedData`,
178
- `declareAppendedData` and `maxAppendedDataLength` (§4.2, §5.2), `includeBOM` (§3.1),
179
- `insertTextBody` (§4.6),
180
- `password` (§5.6), `createRootDirectory` (§7.1), and the head-element switches
181
- `insertCanonicalLink`, `insertMetaNoIndex` and `insertMetaCSP` (§3.1).
182
-
183
- Notes on composition:
184
-
185
- - **Universal mode requires the HTML face** (it is a property of the bootstrap) and is
186
- independent of PNG/PDF. The command-line client enables it by default whenever it
187
- produces a self-extracting file; the browser extension exposes it as the
188
- "self-extracting ZIP (universal)" file format.
189
- - **The extra-data payload is what separates the two self-extracting types in
190
- practice.** A plain file has no way of its own to reach its bytes from a `file:`
191
- URL: the bootstrap has no payload to rebuild the archive from and does not attempt
192
- a self-read there (§4.1), so it goes to its error message. Opening one from disk
193
- needs cooperating software that reads the file and hands it to the bootstrap — the
194
- SingleFile extension does this once granted file access ("Allow access to file
195
- URLs" in Chrome, Edge and Brave; "Disable Local File Restrictions" in Safari). Over
196
- HTTP the plain file is self-sufficient, fetching its own URL. A
197
- universal file instead carries the extra-data payload (§3.1, §5.5), so the bootstrap can
198
- rebuild the archive from the parsed page text with no access to the raw bytes. Such a
199
- file opens from disk in any browser, with no setting and no assistance.
200
- - **All four faces at once is a supported combination**: the PDF document rides
201
- inside the HTML head, which itself rides inside the PNG's first `tEXt` chunk
202
- (the all-four-faces row of the byte map).
203
- - **Faces without HTML** are plain polyglots with no self-extraction and no markup:
204
- the ZIP data is appended after a raw PDF document (which is then plain prepended
205
- data), or wrapped in PNG chunks — or both, the PNG chunk layout carrying the PDF
206
- document as the data of its own `tEXt` chunk (keyword `PDF`) placed right after
207
- `IHDR`, which keeps `%PDF-` within the header scan window (§4.3). In none of
208
- these variants is `page.pdf` an archive entry.
209
-
210
- ### 2.1 The charset rule
211
-
212
- The HTML face declares `<meta charset=utf-8>` when universal mode is off. When
213
- universal mode is on it declares a single-byte charset instead — `windows-1252` in the
214
- reference writer. The declaration MUST appear within the first 1024 bytes of the file
215
- so the parser's encoding prescan finds it. That bound is the HTML standard's own
216
- authoring rule. The prescan it serves is weaker than the rule suggests: the standard
217
- makes it optional, and only *encourages* scanning the first 1024 bytes. Treat the
218
- number as a ceiling to write under, never as a budget a parser promises to read. The
219
- whole `<meta>` tag has to fit: one that straddles the boundary is not seen, and the
220
- parser falls back to its default encoding. Meeting the declaration later, during
221
- tokenization, does not rescue the file. The parser does not resume the prescan. It
222
- re-navigates the document under the new encoding instead, and a writer must not rely
223
- on that.
224
-
225
- The declaration decides the decoding only when nothing outranks it. Three things do,
226
- each returning an encoding with the standard's *certain* confidence, all of them ahead
227
- of the prescan: a byte order mark, a user's explicit encoding override, and a charset
228
- stated by the transport layer, which over HTTP means a `Content-Type` header carrying
229
- its own `charset`. Any of the three replaces the declared charset, the parsed text is
230
- then not what the writer encoded, and the region cannot be recovered from it. This is
231
- the one precondition universal mode has that the file cannot satisfy from within
232
- itself.
233
-
234
- Where it bites is narrower than that makes it sound, because the parsed text is the
235
- last rung, not the first. The bootstrap reads the file's raw bytes whenever it can
236
- (§4.1), and raw bytes carry no encoding; universal extraction is the fallback for when
237
- they are out of reach. Taking the three in turn:
238
-
239
- - A **transport charset** exists only over HTTP, and over HTTP the raw read is what
240
- runs — the bootstrap requests its own URL and takes the response as bytes, which no
241
- `Content-Type` can reinterpret. It reaches universal extraction only in a double
242
- failure: the response has to defeat the raw read, through a network or CORS failure
243
- or a non-200 status, *and* state a charset of its own.
244
- - A **BOM** is the writer's own doing. It is why universal and PNG variants never carry
245
- one (§3.1): the reference writer emits a BOM for the plain variant only
246
- (`includeBOM`), where nothing depends on the declared charset.
247
- - A **user override** is the one no software can prevent, and the rarest.
248
-
249
- On `file:` URLs the bootstrap goes straight to page-text extraction, since it attempts
250
- no raw read there (§4.1) — but there is also no transport layer, so the first of the
251
- three cannot arise on the very path that depends on the charset most.
252
-
253
- The failure is safe rather than silent, which is why the precondition is worth stating
254
- at all. Decoded under the wrong charset the reconstructed bytes are wrong, the payload
255
- checksum does not match, and the extractor MUST fail to the error message (§4.5)
256
- instead of displaying a corrupt page. A reader MAY tell the case apart from ordinary
257
- corruption by comparing the encoding the document was actually decoded with —
258
- `document.characterSet` in a browser — against the declared one, and say so in the
259
- error message. Nothing requires it, and the MUST is unaffected either way.
260
-
261
- Universal mode works in two parts, and the charset carries the first. The archive
262
- bytes themselves are recovered *from the parsed page text*: the browser decoded
263
- them as characters when it parsed the file, and the bootstrap re-encodes those
264
- characters back into bytes. The requirement this places on the charset is
265
- **injectivity**: under the encoding's index in the
266
- [WHATWG Encoding Standard](https://encoding.spec.whatwg.org/) — the mapping every
267
- browser implements — each of the 256 byte values MUST decode to a distinct code
268
- point, and no byte may decode to U+FFFD. Any encoding with that property carries
269
- arbitrary bytes through the parse, and 20 of the standard's encodings qualify (§8.4).
270
- Multi-byte encodings, `utf-8` included, do not: invalid sequences collapse to U+FFFD
271
- and the bytes cannot be recovered.
272
-
273
- The reference writer uses `windows-1252`, for reasons beyond injectivity. It is the
274
- best-supported single-byte encoding there is: the standard resolves the `iso-8859-1`
275
- and `ascii` label families to it, and it is the fallback the HTML standard prescribes
276
- for unlabelled content in most locales — so a file whose `<meta charset>` is stripped
277
- or overridden by a server still tends to be decoded the way the extractor expects. It
278
- also keeps the reverse table small, at 27 entries (§5.5).
279
-
280
- The second part is the extra-data payload, which does **not**
281
- contain the archive: it carries only what the round trip destroys or leaves
282
- undetermined, namely a checksum, the recovered range's length, and the information
283
- needed to restore newline bytes, which the parser normalizes.
284
- The parser also replaces NUL bytes with U+FFFD; since no byte decodes to U+FFFD under
285
- a qualifying encoding, the extractor maps U+FFFD back to NUL unambiguously and the
286
- payload needs nothing for it (§5.5).
287
-
288
- The declared charset governs the **whole document**, not only the regions the format
289
- reasons about. Everything the parser reads is decoded with it, the bootstrap script
290
- included, and the writer's own code is therefore subject to the same single-byte
291
- decoding as the page it carries. In universal mode the bootstrap MUST contain no
292
- character outside ASCII.
293
-
294
- Unlike the `<title>` (§4.6), it cannot be rescued by
295
- character references. A `<script>` element's content is script data, a tokenizer state
296
- that does not resolve them: `&#9786;` written there stays seven literal characters and
297
- reaches the program as seven characters. The escape has to happen one level down, in
298
- the JavaScript source — `\u263A` rather than `☺`, an escape the language resolves when
299
- the script is compiled, not one the HTML parser resolves when the file is read. A
300
- minifier will undo this if allowed to, since printing the shortest form is its default
301
- and the shortest form of `\u263A` is the literal character. A writer that assembles
302
- the bootstrap through a minifier MUST configure it to emit ASCII only.
303
-
304
- The consequence of getting this wrong is worse than the mojibake a raw title produces,
305
- and that is the reason for the MUST. A garbled title is visible; a garbled string
306
- inside the extractor is not. A lookup table is the sharpest case. Emitted as literal
307
- characters, a CP437 table is re-decoded as windows-1252 and grows from 256 entries to
308
- 508, shifting every lookup past the first 32 by 60 positions — and nothing about the
309
- page looks wrong, because the damage is confined to names the table decodes. Under
310
- §5.8 that can be a single entry, and one is enough when it is the entry the extractor
311
- matches by name. The requirement is on the whole bootstrap rather than on any table
312
- inside it, because a minifier does not know which strings are load-bearing.
313
-
314
- ### 2.2 File name conventions
315
-
316
- The reference implementation names files by variant: `.zip` (no HTML face),
317
- `.zip.html` (self-extracting), `.u.zip.html` (self-extracting, universal). These are
318
- conventions for humans and pickers; **readers MUST NOT rely on the file name**. Every
319
- face is discoverable from the bytes alone: PNG and PDF by their signatures, the ZIP
320
- face by its End Of Central Directory record, and the HTML face by an `<html` start tag
321
- occurring before the first local file header — inside the first `tEXt` chunk's data in
322
- the PNG variants, where the markup begins after the chunk's keyword and its NUL
323
- separator. Inside the
324
- archive, the `index.html` and `manifest.json` entries mark it as a saved page; a
325
- reader should identify it that way (§7.1). The self-extracting variants are told apart
326
- the same way: only a universal file carries an `<sfz-extra-data>` element.
327
-
328
- ## 3. The byte map
329
-
330
- Unless a row states otherwise, the layouts below are measured from specimen files
331
- saved from `example.com` (the generation commands are in §8). The relocated row covers
332
- two cases with one layout, `preventAppendedData` and a payload over the appended-data
333
- budget (§5.2): the first is
334
- measured on the relocated specimen, the second derived from the writer rules, because
335
- such a payload requires an archive too large for a readable specimen. The figure below shows
336
- the regions and their order; the glossary of §3.1 is the normative list, and it states
337
- in text everything the figure conveys.
338
-
339
- ![SingleFile archive byte map](assets/singlefile-archive-byte-map.svg)
340
-
341
- ### 3.1 Region glossary
342
-
343
- The names below are the block labels of the figure (where space is tight the figure
344
- merges adjacent blocks into one label, such as `--></body></html>`). In this table,
345
- *Producer* names
346
- the syntax the bytes belong to (*extractor* marks the recovery machinery of
347
- universal mode), and *Present* names the faces, variants or modes in which the
348
- region exists. Rows are grouped: the HTML/ZIP core first, then the regions the PDF
349
- face adds, then the regions the PNG face adds.
350
-
351
- | Region | Producer | Present | Contents |
352
- |---|---|---|---|
353
- | `html-prologue` | HTML | HTML face | Doctype, the root element start tag, `<meta charset>`, an optional implementation-defined comment, title, optional head elements (canonical link, `robots` meta, viewport, Content-Security-Policy), minimal CSS, `<body hidden>`, wait/error messages, optional text body (§4.6). The leading comment, the title, the canonical link and the text body are withheld when a password is set (§5.6). In the plain variant an optional UTF-8 BOM MAY precede the doctype (`includeBOM`); universal and PNG variants never carry one, and the reference writer ignores the option there. In the PNG variants the region is split: everything through `<body hidden>` is the data of the `tEXt "PNG"` chunk, while the messages and the optional text body follow the `tEXt "ZIP"` chunk header; the doctype and the leading comment are dropped. |
354
- | `bootstrap` | HTML | HTML face | One inline `<script>`: the embedded ZIP reader, the extractor, the display routine, and the content-acquisition logic (§4.1). In universal mode its bytes MUST be pure ASCII, since the declared charset decodes this region like any other and character references do not apply inside script data (§2.1). The wrapper start tag that opens the ZIP region follows it, directly or after a relocated `extra-data`. |
355
- | `<!--` / `-->` | HTML | HTML face | The wrapper tag pair hiding a binary region from the HTML parser — comment tags by default, another pair when the hidden bytes defeat them — which `-->` is only the commonest way to do, the full test being `<!--`, `--!>`, a trailing `<!-` and, for the PNG payload, a leading `>` or `->` (§5.1). Drawn at each opening and closing position. The close tag is absent whenever the recovery payload is relocated (§5.2): under `preventAppendedData`, when the payload outgrows the appended-data budget, or on the `<plaintext>` wrapper which cannot close. No markup then follows the archive and the wrapper runs to end-of-file. That does not mean the file ends at the EOCD — the PNG face's tail still follows, inside the wrapper, where it parses as text (§5.1). |
356
- | `zip-entries` | ZIP | always | The archive's local file headers and entry data, written by the ZIP writer. The central directory of an archive written by the reference writer lists `index.html` (the page) first, then `manifest.json` (a JSON description of the archive: original URL, title, save time, resource-to-URL map — informative; the page displays without it), then the resources; the *physical* order of the local headers inside the region is not guaranteed to match, and readers MUST NOT rely on either order — entries are addressed by name (§7.1). |
357
- | `central-directory · eocd` | ZIP | always | The central-directory records followed by the End Of Central Directory record. All offsets are absolute file positions (§5.3). In the HTML+PDF variants the EOCD accounts for the injected `pdf-central-record` (how the writer achieves that is §6). |
358
- | `extra-data` | extractor | universal | `<sfz-extra-data>` element holding the base64, deflate-compressed recovery payload (§5.5). It always sits outside the wrapper, so it parses as a real element the extractor can address. Normal placement: after the EOCD, between the wrapper close tag and the end tags. Relocated placement, used when the payload exceeds the appended-data budget (§5.2) or `preventAppendedData` is set: immediately before the wrapper start tag. In the relocated form the element is followed by space padding: its room is reserved before the archive is written, because the region precedes the ZIP data and resizing it would shift every central-directory offset (§6). Neither placement carries positional meaning — the extractor finds the ZIP region by identifier, not relative to this element (§4.5). |
359
- | `</body></html>` | HTML | HTML face | The end tags closing the document after the wrapper close tag. Omitted whenever the recovery payload is relocated (§5.2), and in the PNG variants so the file can end with the PNG tail. |
360
- | `pdf-local-header` | ZIP | PDF face with HTML | The hand-built local file header for `page.pdf` (STORE, checksum precomputed, language encoding flag left clear as on every other ASCII name — §5.8), written immediately before the PDF document so ZIP readers see an ordinary entry whose data is the PDF (§6). |
361
- | `pdf-document` | PDF | PDF face | The raw PDF bytes. With the HTML face, wrapped together with `pdf-local-header` in a wrapper tag pair inside `html-prologue`, placed so `%PDF-` starts at offset 1024 or lower — the range PDF readers search for the header, which is what lets a PDF document start after other bytes at all (§4.3). Without the HTML face and without the PNG face, the file simply *starts* with the PDF document, as prepended data the ZIP face tolerates; `page.pdf` is then not an archive entry at all — no local header, no central record. |
362
- | `pdf-central-record` | ZIP | PDF face with HTML | The central-directory record for `page.pdf`, injected *before* the writer's own central directory. The start of the central directory is the one place a record can be added without moving any offset the writer already committed, and it makes `page.pdf` the first entry ZIP tools list (§6). |
363
- | `png-signature · IHDR` | PNG | PNG face | The 8-byte PNG signature and the `IHDR` chunk declaring the source image's dimensions — the first 33 bytes of the file. |
364
- | `tEXt "PNG"` | PNG | PNG face with HTML | The 12 header bytes of the first `tEXt` chunk: the 4-byte big-endian length, the type, the keyword and its NUL separator. Its data is `html-prologue` (with the PDF face, the embedded PDF document rides inside it too), ending with the wrapper start tag. |
365
- | `tEXt "PDF"` | PNG | PNG + PDF faces without HTML | The 12 header bytes (length, type, keyword, NUL separator) of a `tEXt` chunk whose data is the raw PDF document. Written only when the PNG and PDF faces combine without HTML — with the HTML face the PDF rides inside `tEXt "PNG"` instead — and placed right after `IHDR` so `%PDF-` stays within the header scan window (§4.3). |
366
- | `pixel-data chunks` | PNG | PNG face | Every chunk of the source image between `IHDR` and `IEND`, ancillary chunks included, copied unmodified. The reference writer takes `IHDR` as the 25 bytes after the signature and `IEND` as the last 12 bytes of the source, so a source image with bytes after `IEND` is not supported. With the HTML face the chunks sit inside the wrapper so the HTML parser skips them. |
367
- | `tEXt "ZIP"` | PNG | PNG face | The 12 header bytes (length, type, keyword, NUL separator) of the archive's own `tEXt` chunk — the second one when the HTML face or the PDF face put a chunk ahead of it, the only one otherwise. Its declared length covers everything from there up to but not including the trailing chunk CRC, as a PNG chunk length always does, so the PNG decoder skips the archive — and, with the HTML face, the bootstrap and the appended data — as the data of one chunk. With the HTML face, the wrapper opened at the end of `tEXt "PNG"` closes immediately after these bytes: its content is the first chunk's CRC, the pixel-data chunks and this chunk's own header, and the prologue resumes as markup directly after the close tag. |
368
- | `crc · IEND` | PNG | PNG face | The `tEXt "ZIP"` chunk's CRC, computed once the archive bytes are final (§6), followed by the empty `IEND` chunk — the last bytes of the file (PNG requires `IEND` to end the stream, which is why the PNG variants drop the end tags). |
369
-
370
- The reader-by-reader interpretation of these regions is §4; the mechanics that keep
371
- them from colliding (wrapper-tag selection, checksums, offsets, the appended-data budget) are
372
- §5.
373
-
374
- ## 4. Reader lenses
375
-
376
- Each consumer of the file has a defined way of locating its own bytes and a defined
377
- reason to ignore the rest. This section walks the same file through each reader. The
378
- figure shows the all-four-faces variant of the byte map once per reader, fading the
379
- regions that reader ignores; the subsections explain each row.
380
-
381
- ![SingleFile archive reader lenses](assets/singlefile-archive-lenses.svg)
382
-
383
- This section describes what stock readers do with the file. The convention stated at
384
- the head of this document applies throughout it: a MUST about a face constrains the
385
- bytes a writer produces, never the stock reader whose behavior the format cannot
386
- change.
387
-
388
- ### 4.1 The browser
389
-
390
- The HTML parser consumes the whole file as one document. Its encoding prescan finds
391
- the `<meta charset>` declaration within the first 1024 bytes (§2.1) and the file is
392
- decoded as a single text; every binary region therefore also exists as characters in
393
- the parsed document, which is what universal mode exploits (§4.5). This holds only
394
- while the declaration is what decides the decoding: a BOM, a user override or a
395
- transport-layer charset outranks it, and universal extraction then fails its checksum
396
- rather than recovering anything (§2.1). The acquisition order below keeps that off the
397
- common path — the raw bytes are read in preference to the parsed text wherever they
398
- can be, and no encoding applies to them.
399
-
400
- The binary regions are kept out of the rendered page by the wrapper tags. The
401
- default wrapper is an HTML comment, and the HTML standard defines exactly which
402
- character sequences terminate one (`-->`, and the recovery form `--!>`); the writer
403
- MUST select a wrapper only after checking the bytes it must hide against that
404
- wrapper's patterns (the same test for every payload, with a shorter ladder for the
405
- PDF and PNG payloads and one extra check for the PNG one, §5.1), so hiding relies on
406
- normative parsing behavior. When no
407
- wrapper fits a PDF or PNG payload, the face is dropped rather than emitted bare
408
- (§5.1). Some binary content always sits *outside* a
409
- wrapper: in the PNG variants, the signature, IHDR and chunk framing bytes that
410
- precede the root element start tag decode to a short run of text that HTML error recovery
411
- places in the (hidden) body. The backstop for all these cases is the prologue: it
412
- declares `<body hidden>`, which only the bootstrap clears, and a stylesheet that
413
- suppresses everything except the wait and error messages once the body is shown, so
414
- the page comes up blank rather than showing raw bytes, with or without scripting.
415
-
416
- The bootstrap script runs at parse time and proceeds in three stages:
417
-
418
- 1. **Acquire the archive bytes.** On `file:` URLs it goes straight to page-text
419
- extraction (§4.5): whether a `file:` page may read its own bytes varies by
420
- browser and configuration (§2), so the bootstrap uses the rung that depends on
421
- neither. On other protocols it requests its own URL, aborting at the response
422
- headers: when the server advertises `Accept-Ranges: bytes` it switches to HTTP
423
- range reading, fetching only the central directory and the entries it needs (a
424
- large archive displays without downloading the ZIP region in full); otherwise it
425
- downloads the whole file. When the header probe fails it falls back to page-text
426
- extraction; so does a failure of the full download itself, past the probe, which
427
- is why the probe leaves the document in place. A failure inside range reading,
428
- past the probe, is not caught the same way: it goes to the error message. Only
429
- when every applicable rung fails does the error
430
- message appear, with recovery instructions that differ by variant (§2).
431
- 2. **Extract.** The embedded ZIP reader reads the archive through the ZIP lens
432
- (§4.2) and rebuilds the page: text entries are decoded, binary entries become
433
- in-memory URLs, and references between entries are rewritten deepest-first.
434
- 3. **Display.** The rebuilt page replaces the bootstrap document. Around this stage
435
- the saved page's `<noscript>` elements are neutralized — rewritten to inert
436
- placeholders before parsing, restored afterwards — because the page was captured
437
- with scripting available, so its noscript fallbacks must not activate in the
438
- viewer.
439
-
440
- The entry point is exposed as a run-once `globalThis.bootstrap(content)` function,
441
- so cooperating software MAY hand the bootstrap bytes it acquired itself; `content` is
442
- the file's bytes as a `Blob` or an array of byte values, or a reader object exposing
443
- the embedded ZIP reader's `readUint8Array` interface, and the call returns a promise
444
- that settles when the page has been displayed. This is how
445
- the SingleFile extension assists a *plain* (non-universal) file on `file:`: granted
446
- file-URL access, it reads the file and invokes the bootstrap. That is the recovery
447
- path the plain variant's error message describes.
448
-
449
- ### 4.2 The ZIP reader
450
-
451
- The ZIP face is read from the end. A reader locates the End Of Central Directory
452
- record by scanning backward from end-of-file, and how far back it scans is the one
453
- reader property the format cannot assume (§1.3, *appended data*). Everything the
454
- writer emits after the record — wrapper close tag, extra-data, end tags, PNG tail —
455
- fits the appended-data budget of §5.2, and the reference writer sizes that budget to
456
- the narrowest scan measured in §8.1, so the record stays reachable for every reader
457
- listed there. Accepting *undeclared* bytes
458
- in that window is itself a customary tolerance (§1.1): the ZIP specification
459
- documents the comment, not trailing junk. From the EOCD
460
- the reader jumps to the central directory and reads only what it references;
461
- central-directory-driven reading is a requirement of the format (§1.2, §7).
462
-
463
- Two properties keep ordinary ZIP tools comfortable:
464
-
465
- - **Offsets MUST be absolute file positions** (§5.3). The stored central-directory offset
466
- equals the record's actual position, and each entry's local-header offset points at
467
- a real local header — tools that cross-check offset arithmetic (rather than
468
- tolerating a uniform shift from prepended data) accept the file as-is.
469
- - **The non-ZIP regions are invisible to the reader.** Bytes before the first local
470
- header and after the EOCD are simply never referenced. The one deliberate
471
- exception: in the PDF-with-HTML variants the first central-directory record points
472
- *back into the HTML head*, where the hand-built `page.pdf` local header and the PDF
473
- document sit (`pdf-local-header`, §3.1) — an ordinary STORE entry that happens to
474
- live inside the prepended region.
475
-
476
- A listing shows `page.pdf` first (when the PDF face is present with HTML), then
477
- `index.html`, `manifest.json` and the page's resources. That order and the two
478
- conventions below describe the reference writer rather than constraining the format —
479
- readers address entries by name (§7.1):
480
-
481
- - Entries for resources fetched from a URL carry that URL in their *comment* field,
482
- and so does each `index.html`, whose comment is the URL of the page or frame it
483
- holds; a resource that came from a `data:` URL carries the literal marker `data:`
484
- instead, and `manifest.json` and `page.pdf` have no comment. Comments are omitted entirely
485
- from a password-protected archive, because the central directory is not encrypted
486
- (§5.6).
487
- - Entries whose content is already compressed (images, fonts, media, PDF) are STOREd
488
- and the rest are deflated, a size optimization no reader depends on.
489
-
490
- One rule here is a requirement. With a password, entry contents are AES-encrypted —
491
- except `page.pdf`, which MUST stay unencrypted and MUST be STOREd, because its bytes
492
- double as the PDF face (§5.6). The encryption is WinZip's AES scheme, the one ZIP
493
- tools implement under compression method 99 with the `0x9901` extra field: AE-2,
494
- AES-256, PBKDF2-HMAC-SHA1 key derivation and an HMAC-SHA1 authentication code. A
495
- reader that already supports encrypted ZIP entries needs nothing specific to this
496
- format, and the [WinZip AES specification](https://www.winzip.com/en/support/aes-encryption/)
497
- is normative for it. A reader implementing the scheme from primitives rather than from
498
- a ZIP library needs six parameters that specification supplies and this paragraph's
499
- names do not: 1000 PBKDF2 iterations; a 16-byte salt at AES-256 strength, stored
500
- before the data; a derived key of 32 + 32 + 2 bytes, read as encryption key,
501
- authentication key, then a password verifier the reader MUST check before decrypting;
502
- the HMAC-SHA1 code truncated to its first 10 bytes and stored after the data; and a
503
- CTR counter that increments **little-endian**, starting at 1, which general-purpose
504
- CTR interfaces do not do.
505
-
506
- Appended data comes in two forms and both are valid ZIP, so readers MUST accept
507
- both. **Raw** — the EOCD declares a zero-length comment and the trailing bytes are
508
- simply outside the archive — is the default, because tools print a declared archive
509
- comment on ordinary operations (§8.1), and in universal mode that comment is the
510
- whole base64 recovery payload. **Declared** — the EOCD's comment length covers every
511
- byte after the record, `declareAppendedData` in the reference writer — is the later
512
- addition, and it is the only form some
513
- readers accept at all: `java.util.zip`, and therefore Android and most JVM tooling,
514
- rejects an archive with undeclared trailing bytes outright (§8.1). A writer SHOULD
515
- offer both and default to raw.
516
-
517
- The declared form carries a ceiling the raw form does not. The comment length is a
518
- 16-bit field, so a run longer than 65535 bytes cannot be declared at all. A writer
519
- whose appended-data budget (§5.2) is raised past that ceiling MUST leave such a run
520
- undeclared rather than write its length back modulo 65536, and readers that accept
521
- only the declared form then reject the file with no diagnostic. The budget and the
522
- ceiling are two separate limits, and a writer that exposes the first as an option
523
- SHOULD say so where it documents it.
524
-
525
- Neither form constrains the other faces, and universal mode supports both, because
526
- the recovery payload describes the recovered range rather than the whole region: the
527
- comment-length field is excluded (§1.3), so its value can be decided after the
528
- payload is final. A writer that instead covered the field would have to solve a
529
- circular dependency — the field's value depends on the size of the appended run,
530
- which contains the payload, whose content depends on the field.
531
-
532
- ### 4.3 The PDF viewer
533
-
534
- The PDF face relies on two customary reader behaviors — conventions PDF
535
- implementations follow rather than guarantees of the PDF specification, so the
536
- compatibility appendix records real-world support (§8):
537
-
538
- - **The header scan.** Viewers accept a file whose `%PDF-` header starts at offset
539
- 1024 or lower — the bound is on the header's first byte, and 1024 itself passes —
540
- and treat the header's position as byte 0 of the document: every
541
- offset in the file (cross-reference entries, `startxref`) is interpreted relative
542
- to it. The writer MUST place the header inside that window; the document's own
543
- offsets then need no rewriting. Engines differ in how strictly they hold to the
544
- 1024-byte figure, and the strict ones are the common ones: PDFium — Chrome, Edge and
545
- everything else Chromium-based — accepts a header starting at offset 1024 and
546
- rejects one at 1025 outright, while poppler and macOS PDFKit impose no limit at all
547
- (§8.1).
548
- - **Tolerance of trailing data.** The archive continues after `%%EOF`, so the
549
- document's tail is not the file's tail. Viewers cope by searching backward for the
550
- trailer over a larger window or by reconstructing the cross-reference table from
551
- the objects themselves.
552
-
553
- With the HTML face, the PDF document sits inside the head, wrapped in its own
554
- wrapper-tag pair chosen against the local header and the PDF together (§5.1) — a PDF
555
- containing `-->` steps the ladder, and so can the header, whose CRC-32 and size fields
556
- hold arbitrary bytes — and preceded by the `page.pdf` local header so the same bytes
557
- are also a ZIP entry. That dual role is why the entry MUST be STOREd and MUST NOT be
558
- encrypted: a viewer reads the entry's data region directly, and any transformation
559
- of it would break the face. Without the HTML face the document needs no wrapper, and
560
- where it sits depends on the PNG face: alone with the ZIP face it simply starts the
561
- file, at offset 0, exercising only the trailing-data tolerance; with the PNG face it is
562
- the data of a `tEXt "PDF"` chunk placed right after `IHDR` (§3.1), which puts `%PDF-`
563
- at offset 45 exactly, inside the window but not at its start.
564
-
565
- ### 4.4 The PNG decoder
566
-
567
- PNG offers no header scan: the standard requires the signature to be the first
568
- 8 bytes of the stream and `IEND` to be its last chunk. The PNG face therefore owns
569
- both ends of the file — the HTML face gives up its doctype and leading comment
570
- at the front, and the closing `</body></html>` at the back (§3.1).
571
-
572
- After the signature, a decoder walks chunks — length, type, data, CRC — and skips
573
- ancillary chunks it does not use. The face hides all foreign bytes inside two `tEXt`
574
- chunks (ancillary by construction, their type starting lowercase):
575
-
576
- - **`tEXt` with keyword `PNG`** carries the head of the HTML prologue, through
577
- `<body hidden>` — and, in the all-faces variant, the embedded PDF document inside
578
- it — ending with the wrapper start tag. The rest of the prologue, the messages and
579
- the optional text body, comes later, inside the second chunk.
580
- - **`tEXt` with keyword `ZIP`** declares a length that covers everything from its
581
- keyword to the trailing chunk CRC: the rest of the HTML, the bootstrap, the whole
582
- ZIP region and the appended data. The decoder hops over all of it as the data of
583
- one chunk.
584
-
585
- The `tEXt "ZIP"` chunk carries text the PNG standard does not strictly permit: a
586
- `tEXt` text string is Latin-1 text, and its payload contains NUL bytes — 78 in the
587
- `png` specimen, 101 in the `png-pdf` one. The first chunk is pure printable ASCII in
588
- the plain `png` specimen, where the doctype and the provenance comment are suppressed
589
- and the title is escaped to character references; only the `-pdf` variants put NULs
590
- in it. Decoders skip
591
- ancillary chunks without inspecting their text, so this passes everywhere tested
592
- (§8.1). It exercises the PNG tolerance §1.1 lists at its limit: what decoders ignore
593
- in a `tEXt` chunk is text PNG does not permit.
594
-
595
- Both chunks MUST carry correct CRCs — decoders are entitled to verify them, and the
596
- second chunk's CRC can only be computed once the archive bytes are final (§6). The
597
- `pixel-data chunks` region is copied bit-identically from the source image, so the
598
- decoded image is exactly that image.
599
-
600
- ### 4.5 The universal-mode extractor
601
-
602
- The last reader is the format's own: the extraction path of universal mode, used
603
- when the raw bytes are unreachable (§4.1). Its input is not the file but the *parsed
604
- document* — the characters the HTML parser produced — and its output is the ZIP
605
- region reconstructed byte for byte, with one deliberate exception: the two bytes of
606
- the EOCD comment-length field, which the payload does not describe and the extractor
607
- always writes as zero (step 2 below, and the row in §7.4).
608
-
609
- It works in three steps:
610
-
611
- 1. **Locate.** The extractor finds the `<sfz-extra-data>` element for the payload, and
612
- the ZIP region's node by its identifier `sfz-data` (§1.3): the element returned by
613
- `getElementById`, or, when the wrapper is a comment, the first comment in the
614
- document whose data starts with those characters. An element bearing the identifier
615
- wins over a comment when both resolve; a reader that finds an id-bearing element
616
- which is not one of §5.1's wrapper rungs SHOULD fall back to the comment, since the
617
- `id` is then something else in the page; the reference extractor does not, and
618
- takes whatever element bears the identifier. The two placements (§3.1) need no
619
- telling apart, and neither the region's position in the tree nor its depth carries
620
- meaning — a document that moved the node before extraction resolves the same way,
621
- which matters because the reference extractor relocates `meta` and `style` elements
622
- into the head before this step and one wrapper rung is a `style` element. With a
623
- comment wrapper the identifier is part of the node's data, so the re-encoding in
624
- step 3 starts after it.
625
- 2. **Decode the payload.** The element's text is base64 of a raw-deflate stream, and
626
- the writer puts nothing else inside the element — the padding of the relocated
627
- placement sits outside it (§6.1) — though a reader SHOULD ignore whitespace there
628
- rather than reject the file. It
629
- inflates to four fields: a checksum of the recovered range, its byte length, the
630
- newline count, and the packed sequence of 2-bit codes recording each original
631
- newline (LF, CR or CR LF). §5.5 gives their wire format — little-endian 32-bit
632
- words, the codes packed 16 per word, least-significant pair first — which a reader
633
- needs before it can read any of what follows.
634
- Every one of the four describes the *recovered range*,
635
- not the whole ZIP region: a newline formed by the two excluded bytes is neither
636
- counted nor coded, and the checksum does not cover them. The declared length is the
637
- **only** bound on the re-encoding: the extractor MUST stop there and append two zero
638
- bytes to complete the EOCD record.
639
- Those two bytes are its comment-length field, and only that field: every other byte
640
- of the record is reconstructed from the parsed text like the rest of the region. The
641
- field is excluded because its value depends on what follows the region, so a payload
642
- covering it could not be computed until the appended data was final (§1.3); a
643
- recovered archive therefore always declares a zero-length comment, whatever the
644
- original declared. Never infer the bound from the node instead: the wrapper's close tag is
645
- absent whenever appended data is prevented, and under `<plaintext>` the node always
646
- runs to end of file. The final word is padded to 16 codes,
647
- so codes beyond the declared newline count carry no meaning and a reader MUST ignore
648
- them; within the count, the value 3 is unassigned and a reader MUST reject a payload
649
- that uses it.
650
- 3. **Re-encode the characters.** Apply the inverse of the declared charset, exactly
651
- as §5.5 defines it: a character whose code point is a byte value that decodes to
652
- itself becomes that byte, every other code point goes through the reverse table,
653
- and U+FFFD becomes NUL. Under windows-1252 this is nearly an identity — 229 of the
654
- 256 values map to themselves — but the shortcut "any code point ≤ 255 is that
655
- byte" is **not** a valid substitute: under other qualifying charsets (§2.1) code
656
- points below 256 can belong to a different byte, 75 of them under `macintosh`, and
657
- the shortcut would silently corrupt the region. The reference extractor takes it,
658
- and is correct only because it supports windows-1252 alone. The two things parsing destroyed
659
- are restored from the payload: each parsed newline consumes the next 2-bit code to
660
- reproduce the original byte sequence.
661
-
662
- In the PDF-with-HTML variants the recovered region is a complete archive except for
663
- one entry: its central directory still lists `page.pdf`, but that entry's local
664
- header and data sit in the HTML head, outside the ZIP region (§1.3). The extractor
665
- skips the entry instead of failing — the displayed page never references it, and
666
- the PDF stays reachable through every raw-bytes path (§4.2).
667
-
668
- The entry's offset is unusable *within the recovered region* — it is a true file
669
- position like every other (§5.3), and a reader of the whole file follows it normally;
670
- what fails is only the shifted arithmetic of the paragraph below. Its bytes are not out
671
- of reach either: the local header
672
- and the document sit in a wrapper in the head, so they are in the parsed page like any
673
- other region, and a reader MAY recover them by the same round trip — find a local file
674
- header naming `page.pdf` among the other parsed nodes, take the declared number of
675
- bytes after it, and check them against the CRC-32 the central directory holds. The CRC
676
- check is mandatory. The recovery payload's newline codes cover the ZIP region only, so
677
- a newline in the PDF block has no code and its original bytes must be guessed: assume
678
- LF, the byte the parser normalized *to*. A PDF routinely contains CR, so the guess
679
- often fails, and nothing may be written unless the CRC-32 agrees. Take the length from
680
- the central-directory record, which is the entry's authority for it; the hand-built
681
- local header of §6.1 carries the same value, but a reader cannot tell that from a
682
- length deferred to a data descriptor. A matching CRC-32 is a 32-bit non-cryptographic
683
- check over a reconstruction that differs from the original in at most a few newline
684
- bytes, so an undetected error is improbable but cannot be ruled out. A reader MUST NOT
685
- present a reconstructed `page.pdf` as verified, and MUST NOT let one displace bytes
686
- obtained from a raw-bytes read.
687
-
688
- Recovering the entry is therefore optional; the reference extractor skips it, which
689
- conforms. A reader that *presents the archive's contents* — a listing, an
690
- extract-to-disk, an entry enumeration offered to a caller (§7.1) — MUST report
691
- `page.pdf` as present and unretrieved rather than omit it, whether it skipped the
692
- recovery or tried and failed the CRC check. Its listing is then the same as a raw-bytes
693
- reader's, with only the bytes missing. A reader that has no such surface is outside the
694
- rule: the display path of §4.1 rebuilds a page from the entries it needs, `page.pdf` is
695
- referenced by nothing in that page, and the reference extractor accordingly filters the
696
- entry out on every acquisition path, including the ones that read raw bytes and could
697
- return it.
698
-
699
- The extractor MUST verify the three checkable payload fields — byte length, newline
700
- count and checksum — and fail to the error message on any mismatch. It MUST also fail
701
- on the unassigned newline code of step 2 rather than decode it, so that a payload
702
- written against a later revision of the format is named as unsupported instead of
703
- silently reconstructing the wrong bytes.
704
-
705
- The recovered region is a complete archive but **not an offset-self-contained one**.
706
- Its offsets are still absolute positions in the original file (§5.3), so every
707
- local-header offset in its central directory, and the central-directory offset in its
708
- EOCD record, overshoot by exactly the region's start position in the file. A reader of
709
- the recovered region MUST therefore apply a uniform negative shift of that amount, the
710
- prepended-data compensation ZIP readers already implement: from the region's point of
711
- view the missing bytes look like a prefix that was stripped. In the
712
- specimen of §8.2 the shift is 122005 bytes: Info-ZIP reports `missing 122005 bytes in
713
- zipfile`, adds `(attempting to process anyway)` and lists both entries, while readers
714
- that compensate silently, such as Python's `zipfile`, show no diagnostic at all. The
715
- shift also puts `page.pdf` out of the
716
- offset-following path: its local header lies *before* the region, so its compensated
717
- offset is negative and no reader can seek to it. That offset is the header's own
718
- position — which §4.3 keeps inside the first 1024 bytes — minus the region's start,
719
- which lies past the whole bootstrap, so it is negative for every archive and its
720
- magnitude is essentially the region's start, so it moves with the size of the inlined
721
- ZIP library and no particular value should be read into it. On
722
- success the shifted bytes enter the normal extraction path (§4.2).
723
-
724
- The shift is a file offset, and a universal-mode reader has no file. It does not need
725
- one: the region carries the shift within itself, since its EOCD declares both the size
726
- of the central directory and its absolute offset, while the directory's position inside
727
- the region is known. With `region` the recovered bytes,
728
-
729
- ```
730
- shift = eocd.centralDirectoryOffset - (eocdPosition - eocd.centralDirectorySize)
731
- ```
732
-
733
- where `eocdPosition` is the EOCD record's own offset within `region`, found by scanning
734
- backward for its signature the way any ZIP reader finds it. A reader arrives at the
735
- same number as a ZIP library's prepended-data compensation, which derives it from the
736
- record's position rather than from the buffer's end. Do not substitute
737
- `region.length - 22` for `eocdPosition`. The two are in fact equal for every recovered
738
- region, which always ends at the EOCD's last byte and declares a zero-length comment
739
- (§1.3), but the habit fails the moment the same code is pointed at a file rather than a
740
- recovered region: there a non-empty archive comment puts bytes after the record. Zip64
741
- does not — its records precede the EOCD, which stays last.
742
-
743
- Under zip64 (§5.7) both of those EOCD fields are the `0xFFFFFFFF` sentinel, and the
744
- zip64 end of central directory record carries the real values. Take them from there,
745
- using the same `eocdPosition` arithmetic against that record's own position — the
746
- zip64 locator states an absolute offset in the original file, so it needs the shift
747
- this formula produces and cannot be used to find it. A reader that instead uses the
748
- sentinels arithmetically gets a shift in the billions, with no diagnostic.
749
-
750
- Which leaves the record itself to be located without the offset that normally points at
751
- it. Scan backward from the locator for the `PK\x06\x06` signature and confirm each
752
- candidate against the record's own size field, the 8 bytes at `p + 4`, which by
753
- definition excludes the leading 12:
754
-
755
- ```
756
- p + 12 + size == locatorPosition
757
- ```
758
-
759
- A well-formed archive puts the record immediately before the locator, where it is 56
760
- bytes long if it carries no extensible data sector, so `locatorPosition - 56` is worth
761
- testing before scanning at all. The confirmation matters on the archives that miss:
762
- past the record the scan walks back through the central directory, whose file names and
763
- extra fields are arbitrary bytes, and past that through entry data, and a four-byte
764
- signature turns up in bytes nothing constrains. The test settles each candidate against
765
- the record's own field, so it needs no offset it does not already have.
766
-
767
- ### 4.6 Text tools
768
-
769
- The optional text body (`insertTextBody`) addresses one more consumer: software that
770
- reads the file as plain text — `grep`, desktop search, indexers — and will never run
771
- the bootstrap or unzip anything. It is a `<main hidden>` element at the end of the
772
- visible prologue holding the page's text content, so the page stays searchable
773
- without any extraction. It is searchable by everyone, so it is not written at all
774
- when a password is set (§5.6): its text would be readable without the password.
775
-
776
- The text body is always written in UTF-8, regardless of the declared charset. In
777
- universal mode this cuts its audience in two: a charset-oblivious tool that reads
778
- raw bytes — `grep`, plain-text search — sees intact UTF-8, while any consumer that
779
- honors the declared `<meta charset>` decodes it as windows-1252 and garbles
780
- non-ASCII text. That includes the HTML parser itself — harmless there, because the
781
- region is hidden and replaced (§4.1) — but also HTML-aware indexers. In universal mode the text body
782
- opens with the page title, for the same raw-byte
783
- audience — it is the first *text* in the element, which is not necessarily the
784
- element's first line: the reference writer's serialization puts a newline before it.
785
- Outside universal mode the title is not repeated there, since the prologue's own
786
- `<title>` is already readable as bytes.
787
-
788
- The `<title>` element takes the opposite route, and so does every other piece of
789
- prologue text the writer assembles itself. Character references are resolved against
790
- Unicode independently of the declared encoding — in RCDATA, where the title's content
791
- sits, in ordinary element text, and in attribute values alike — so the writer emits
792
- every character outside printable ASCII, along with `&`, `<`, `>` and `"`, as a
793
- numeric reference. Those bytes are therefore pure ASCII and the text survives the
794
- single-byte declaration intact: a page titled 日本語 shows as 日本語 in the browser tab
795
- and to any conforming parser. The reference writer passes the title, the canonical
796
- link's `href` and the viewport value — attribute values, which is what the `"` is for —
797
- through one shared escaper. Writers that emit such text raw MUST NOT do so in
798
- universal mode, where the same bytes decode as mojibake.
799
-
800
- The text body above is the deliberate exception, not an oversight: it is left as raw
801
- UTF-8 because its audience reads bytes rather than parsed text. The bootstrap script
802
- is the other region outside this rule, and it is outside for a harder reason — script
803
- data does not resolve character references at all, so the escape must happen in the
804
- JavaScript source instead (§2.1).
805
-
806
- An implementation-defined comment (§3.1) is the third region outside the rule, and the
807
- only one with no escape available at all. Comment data does not resolve character
808
- references either, and unlike script data it has no second language of its own to
809
- escape in: `&#233;` written in a comment stays `&#233;` in every reader, so the escaper
810
- does not restore the character, it replaces one unreadable form with another. The
811
- reference writer therefore leaves the comment's characters alone and serializes it as
812
- UTF-8 with the rest of the prologue, deliberately. What it does rewrite is the
813
- comment's own terminators: a space goes before the `>` of `-->` and `--!>`, before a
814
- leading `>` or `->`, and after a trailing `<!-`, so the comment cannot close itself
815
- (§5.1). Its audience is whoever opens the raw file in an
816
- editor or runs a text tool over it, and those decode the bytes as UTF-8 whatever the
817
- declaration says; only a browser's raw view, which honors the declared charset, shows
818
- the text as mojibake in universal mode. The bytes are harmless to extraction, since
819
- the recovery payload covers the ZIP region alone, far past the prologue. A reader MUST
820
- NOT rely on decoding this comment through the declared charset, and a writer that
821
- wants it readable everywhere restricts it to printable ASCII. The page's own copy of
822
- the comment, inside `index.html`, is UTF-8 in a UTF-8 document and is the one the
823
- displayed page and the infobar carry.
824
-
825
- ## 5. Cross-cutting mechanics
826
-
827
- Section 3 named the regions and §4 read them one reader at a time. What remains are
828
- the rules that span readers: how a payload is hidden, how much room is left at the
829
- end of the file, which numbers are offsets into what, what each checksum covers, how
830
- the character round trip is inverted, what a password protects, and what changes when
831
- the archive is large enough to need zip64.
832
-
833
- ### 5.1 Wrapper-tag selection
834
-
835
- Binary payloads inside the HTML face are hidden by a wrapper tag pair. What the format
836
- requires of a wrapper is that the HTML parser not treat its content as markup, so the
837
- payload survives parsing as text, and that the payload not contain the construct's
838
- terminator. Any construct with those properties works; the reference writer picks from
839
- this ladder, in order:
840
-
841
- | Order | Wrapper | Parser treatment of the content | Terminated by |
842
- |---|---|---|---|
843
- | 1 | `<!--` … `-->` | comment | `-->`, or the recovery form `--!>` |
844
- | 2 | `<script type=sfz-data>` | script data, not executed (the type is not a JavaScript MIME type) | `</script` followed by whitespace, `/` or `>` |
845
- | 3 | `<style type=sfz-data>` | raw text, no style sheet built (the type is not a CSS MIME type) | `</style` + delimiter |
846
- | 4 | `<noframes>` | raw text | `</noframes` + delimiter |
847
- | 5 | `<noembed>` | raw text | `</noembed` + delimiter |
848
- | 6 | `<iframe>` | raw text | `</iframe` + delimiter |
849
- | 7 | `<xmp>` | raw text | `</xmp` + delimiter |
850
- | 8 | `<svg><![CDATA[` … `]]></svg>` | CDATA section | `]]>` |
851
- | 9 | `<plaintext>` | everything to end of file | nothing — the element cannot be closed |
852
-
853
- The terminators are written lower case above, but HTML matches end tag names ASCII
854
- case-insensitively: `</XMP>` and `</Script ` close their elements just as `</xmp>` and
855
- `</script>` do. A writer's test MUST be case-insensitive, on the start patterns as well
856
- as the end ones. A stored, uncompressed resource is the realistic source of an
857
- upper-case one.
858
-
859
- Every rung hides its content unconditionally, which is why `<noscript>` is not one. It
860
- has the right terminator, but it is the one construct whose content is raw text only
861
- while scripting is enabled and markup when it is not, so on a page opened without
862
- scripting the archive bytes would reach the tree builder as tags. The rungs below it
863
- hide the same payloads at no extra cost, so there is nothing to weigh against that.
864
-
865
- The order under the comment is not arbitrary. Every rung hides its content from an HTML
866
- parser, but text extractors differ, and the ZIP region is large enough that the
867
- difference is a user-visible one. Measured on macOS: Spotlight's HTML importer indexes
868
- the content of `<noframes>`, `<noembed>`, `<iframe>`, `<xmp>` and `<plaintext>`, and
869
- `textutil` reads the last two of those, `<xmp>` and `<plaintext>`, while the comment
870
- and the `script` and `style` rungs are dropped by both. Those two therefore sit
871
- directly under the comment, so that an escalating writer keeps the archive out of the
872
- reader's local search index
873
- for as long as the payload allows. Both are inert at those types: the script is not
874
- executed and no style sheet is built.
875
-
876
- That ordering is the one place the format optimizes against measured third-party
877
- behavior rather than against a rule, and unlike the tolerances of §1.1 nothing depends
878
- on the measurement holding. An extractor that starts reading `<script type=sfz-data>`,
879
- or stops reading `<xmp>`, changes only which archives end up in a local search index;
880
- every rung still hides its content from the HTML parser, and a writer whose ladder is
881
- ordered differently produces files that are just as correct.
882
-
883
- The CDATA rung sits where it does for the same reason, and it is the only rung whose
884
- placement understates it. A CDATA section is a CDATA section only in foreign content,
885
- which is what the `<svg>` element is there for — in HTML content `<![CDATA[` is a
886
- bogus comment, and the payload would be markup. Given the `<svg>`, the construct is the
887
- strongest on the ladder: `]]>` is the whole of its terminator, and it is a sequence real
888
- payloads carry far less often than `-->` or `</script>`. That matters most for the one
889
- thing the ladder cannot otherwise avoid — an archive nested inside another as a face
890
- carries the terminator of every rung it climbed, so each rung is spent once and only
891
- once (§5.1, ladder depth). A writer MUST place the identifier on the `<svg>` element and
892
- not on the markup declaration, which takes no attributes: `<svg id=sfz-data><![CDATA[`.
893
-
894
- Two properties of the CDATA section state are worth stating because a writer is tempted
895
- to guard against both and needs neither. Sections do not nest, so a `<![CDATA[` inside
896
- the payload is text like any other — the start-pattern test on this rung is the same
897
- conservatism the raw-text rungs get, not a necessity. And trailing brackets are safe: a
898
- payload ending `]]` against the writer's `]]>` produces `]]]]>`, and the tokenizer's
899
- CDATA section end state emits the payload's own two brackets before closing, so the
900
- recovered bytes are exact.
901
-
902
- Verified in Blink, Gecko and WebKit, and against html5lib: a universal-mode archive on
903
- this rung recovers from the parsed document byte for byte, with the checksum of §4.5
904
- matching, indistinguishably from the same archive on the comment rung. The same probe
905
- covered a payload holding every byte value, every rung's patterns, and the near-misses
906
- `]]x>`, `] ]>`, `]>` and `]]`, in both the prologue position and mid-document.
907
-
908
- Two things about the ladder *are* required. Whatever order a writer gives the eight
909
- closable rungs, it MUST apply the selection test below to every rung it considers, and
910
- MUST keep `<plaintext>` available as the rung of last resort: §6.2's termination
911
- argument needs one rung no payload can defeat.
912
-
913
- The wrapper of the ZIP region also carries the identifier the extractor addresses it
914
- with (§4.5): an element rung takes it as an `id` attribute — `<script type=sfz-data
915
- id=sfz-data>`, `<noframes id=sfz-data>` — and the comment rung as the first characters of
916
- its data, `<!--sfz-data`. The wrappers hiding the PDF and PNG faces MUST NOT carry it:
917
- those payloads are found by byte structure, and a second node bearing the identifier
918
- would shadow the archive. For the same reason no comment ahead of the wrapper may
919
- begin with those characters, the implementation-defined comment of §3.1 included,
920
- since the lookup takes the first that does.
921
-
922
- The reference writer walks the ladder from the top and takes the first rung the payload
923
- does not defeat. The test it applies is the format's, and is the same for every payload;
924
- what differs is how far the ladder goes:
925
-
926
- - **The ZIP region** rejects a rung whose *end* pattern the payload contains, and also
927
- a rung whose *start* pattern it contains. A rung's start pattern is the tag's opening
928
- delimiter and name, without attributes: `<!--` for the comment, then `<script`,
929
- `<style`, `<noframes`, `<noembed`, `<iframe`, `<xmp` for the elements, and
930
- `<![CDATA[` for the CDATA rung.
931
-
932
- `<plaintext>` is exempt from **both** tests. It has no terminator to occur and no
933
- tokenizer states to escape into — a `<plaintext` inside a `<plaintext>` is inert
934
- text like everything else — so no payload can defeat it. §6.2's termination argument
935
- rests on that exemption.
936
-
937
- The other eight are all tested, and a writer MUST test all eight rather than the one
938
- that needs it. The `<script>` rung needs it
939
- to be correct at all, because script data has escape states no other rung has: `<!--`
940
- in script data enters *script data escaped*, and a `<script` after that enters *script
941
- data double escaped*, where `</script>` does **not** close the element. So a payload
942
- can hold `<!--` and then `<script`, contain no `</script` anywhere, pass the end test —
943
- and the wrapper then swallows its own end tag, the extra-data element and the rest of
944
- the document. On the other seven the start test is genuine conservatism: a nested `<!--`
945
- is a parse error inside a comment but does not close it, and the raw-text rungs hold a
946
- flat run of characters with no states at all, while CDATA sections do not nest. The
947
- rule is uniform deliberately: the
948
- exemption would save one pattern match per rung on bytes already in memory, at the
949
- cost of a special case an implementer has to remember correctly about the single rung
950
- where forgetting it destroys the document.
951
- - **The PDF and PNG payloads** apply the same two tests, for the same reason — a face
952
- that took the `<script>` rung on a payload holding `<!--` and `<script` would swallow
953
- the rest of the document, title, bootstrap and extra-data element included — but the
954
- `<plaintext>` rung is excluded from their ladder: those payloads sit in the middle
955
- of the file, so a wrapper that can never close is not an option. When no rung fits,
956
- the writer MUST omit the face and emit the archive without it. It MUST NOT write the
957
- payload bare. Bare renders acceptably — the blank-page backstop of §4.1 keeps it
958
- invisible — but the payload's markup joins the document, and a payload that is itself
959
- a SingleFile archive then contributes an `sfz-data` node ahead of the file's own. A
960
- reader looking for one node finds two, takes the first, and returns an archive that
961
- passes every check it has (§7.4). A face is a convenience; the archive is not.
962
-
963
- The comment rung has one restriction more than a terminator. HTML forbids comment text
964
- that *starts* with `>` or `->`, and the tokenizer enforces it: it closes the comment
965
- right there, spilling the payload into the parser. What starts the comment differs by
966
- payload — the ZIP region begins with the identifier, the PDF face with a local file
967
- header or `%PDF-` — but the PNG face begins with the CRC of the chunk carrying the
968
- start tag, four bytes that are only settled once the tag is chosen, and one in 256 of
969
- them is `>`. A writer using a comment there MUST compute that checksum and leave the
970
- comment rung when it opens with `>` or `->`. It MUST leave it by *resuming the rung
971
- search* below it, not by taking the rung that follows: the checksum says only that the
972
- comment is unusable, and which rung is usable remains the payload's to say. The test
973
- itself cannot cascade — only the comment rung carries the restriction, and every rung
974
- below it is an element — but the payload's terminators still apply, and a payload
975
- holding `</script>` sends a writer that stepped rather than searched onto the one rung
976
- it is guaranteed to close. HTML also forbids comment text ending
977
- with `<!-`, which the terminator check covers by testing that pattern anchored at the
978
- payload's end.
979
-
980
- Choosing the last rung has consequences that reach the rest of the file: because
981
- `<plaintext>` cannot be closed, selecting it sets `preventAppendedData` — no *markup*
982
- may follow the ZIP region, which forces the relocated placement of the extra-data
983
- element (§5.2) and drops the closing `</body></html>`.
984
-
985
- That constraint is about markup, not about the last byte of the file, so the PNG face
986
- composes with this rung: the `tEXt` chunk's checksum and the `IEND` chunk still follow
987
- the region, as the PNG face requires, and `<plaintext>` reads them as the text they are.
988
- Verified on a build forced onto this rung with a screenshot embedded: the file ends
989
- `49 45 4e 44 ae 42 60 82`, decodes as a PNG, and its archive extracts from the parsed
990
- page. The termination argument of §6.2 therefore holds for the faced variants too.
991
-
992
- The ladder has a depth, and nesting reaches it. An archive used as the PDF or PNG
993
- payload of another one carries the terminator of every rung its own faces climbed
994
- through, and `-->`, `</script>` and `</style>` besides, which every prologue emits.
995
- Each level of nesting therefore burns exactly one rung, and the eight a face may use
996
- run out at the sixth: the reference writer selects `<!--`, then `<noframes>`,
997
- `<noembed>`, `<iframe>`, `<xmp>`, the CDATA rung, and then has nothing left. No ladder
998
- of fixed length avoids this; adding a rung moves the limit by one level, which is the
999
- limit of what the CDATA rung buys here — its value is that real payloads rarely hold
1000
- `]]>`, not that it makes nesting unbounded. That is why the paragraph
1001
- above states a MUST rather than a quality-of-implementation preference — exhaustion is
1002
- reachable by construction, not only by a payload built to provoke it.
1003
-
1004
- The selection test above — both patterns, on every rung the writer considers — MUST
1005
- be applied to the payload's bytes in their final form, for every payload the writer
1006
- hides and in every variant that hides one. *Final* is the whole of the requirement:
1007
- bytes the writer has yet to settle have not been tested. Every rejection restarts the build (§6):
1008
- the wrapper choice changes the bytes preceding the archive, so the archive must be
1009
- rewritten at its new position.
1010
-
1011
- Two fields are patched after that test, and each needs one of its own. The EOCD
1012
- comment-length field sits at the end of the ZIP region and is patched under the
1013
- declared form (§6.1, step 11); the writer tests the bytes around it again with the
1014
- final value in place and keeps the raw form when that value would complete a pattern,
1015
- since the raw form is always valid. The `tEXt "ZIP"` length field sits inside the pixel-data wrapper,
1016
- with the fixed `tEXt` type and `ZIP` keyword after it, and is written last (step 12).
1017
- The header is tested with the rest of the payload, the length as zeros, which cannot
1018
- join a pattern; the real length is big-endian, so a pattern byte in it would have to be
1019
- the most significant byte of the chunk's size, and the smallest byte any pattern
1020
- contains, `-` at 0x2D, puts that size at 0x2D000000 bytes, about 755 MB. The writer
1021
- refuses to build a self-extracting PNG variant whose chunk reaches that size rather than
1022
- re-check the field.
1023
-
1024
- ### 5.2 The appended-data budget
1025
-
1026
- The run the writer emits after the EOCD record has two limits, and only one of them
1027
- comes from the format. A run *declared* as the archive comment MUST fit in 65535
1028
- bytes, the largest value a comment-length field can hold (§4.2). A run left *raw* has
1029
- no format limit at all: the bytes are outside the archive, and nothing in ZIP bounds
1030
- them. What bounds both in practice is the reader. Locating the EOCD record means
1031
- scanning backward from end-of-file, and the searches measured in §8.1 stop at 16383
1032
- bytes for libarchive, 32768 for perl `Archive::Zip` and 65557 for Python `zipfile`, so
1033
- a run sized to the comment ceiling is already invisible to the narrowest of them. A
1034
- writer therefore keeps a *budget*, sized to the readers it means to satisfy rather
1035
- than to the format. The appended run is:
1036
-
1037
- ```
1038
- wrapper close tag + extra-data element + end tags + (PNG face: 4-byte chunk CRC + 12-byte IEND)
1039
- ```
1040
-
1041
- and the writer compares its total against that budget before committing to it. The
1042
- EOCD record's own 22 bytes sit inside a reader's window as well, which is what turns a
1043
- 65535-byte run into the 65557 bytes of §1.3 and the reference writer's budget into
1044
- libarchive's 16383.
1045
-
1046
- The reference writer exposes the budget as `maxAppendedDataLength` and defaults it to
1047
- 16361 bytes: libarchive's window less the 22 bytes of the record, which is the largest
1048
- run behind which every reader of §8.1 still finds the record. A writer MAY choose
1049
- another value. Raising it above 65535 leaves the run undeclarable: it is emitted, and
1050
- it is still valid ZIP, but no comment length can cover it, and §4.2 says what that
1051
- costs.
1052
-
1053
- Only the extra-data element can outgrow the budget: it carries one 2-bit code per
1054
- newline sequence in the recovered range — the ZIP region without its comment-length
1055
- field (§4.5), CR LF counting once, for two bytes (§5.5) — so it
1056
- grows with the archive. Newline bytes
1057
- occur at their natural density in compressed and STOREd binary data — about two in
1058
- every 256 bytes — and the codes are compressed and base64-encoded, which measures at
1059
- one byte of element per 650 bytes of archive at scale (§8). The default budget is
1060
- therefore exhausted at an archive of roughly 10 MB, and the 65535-byte ceiling at
1061
- roughly 40 MB, so the relocated placement is uncommon in practice. That ratio is the large-archive limit and must not be used to size a
1062
- particular file: deflate's overhead is a fixed cost spread over a growing payload, so
1063
- small archives are far less efficient. Measured on exact byte counts, a 6099-byte region
1064
- needs 69 bytes of element — a ratio of 88 — and a 74057-byte region needs 189, a ratio
1065
- of 392; §8.3's series then runs 475, 609, 650 and 646 as the archive grows from 86 KB to
1066
- 4.2 MB, so the ratio approaches the headline figure from below and levels off rather
1067
- than climbing past it. A writer sizes its reservation from the payload it actually
1068
- produced (§6.2), never from this figure. When the payload does not fit, or when
1069
- `preventAppendedData` is set, the writer switches to the **relocated placement**: the
1070
- element moves in front of the wrapper start tag, ahead of the archive (§3.1). Room
1071
- for it MUST be reserved before the ZIP region is written, because inserting bytes
1072
- ahead of the archive would shift every offset the ZIP writer has already committed;
1073
- the reservation is padded with spaces and the real payload is written into it once
1074
- its final size is known (§6). Relocation is final for the build, and a relocated
1075
- archive carries no appended run at all: the writer emits neither the wrapper's
1076
- terminator nor the end tags, so outside the PNG face, whose tail still follows (§5.1),
1077
- the file ends at the EOCD record like a plain ZIP file and the readers that reject
1078
- trailing bytes open it (§8.1). The parser closes the open
1079
- comment or element at end of file, and `</body></html>` are implied, so the page
1080
- renders the same.
1081
-
1082
- Relocation moves the element rather than copying it, but it is not a move at constant
1083
- size, and wherever there is an element to move it costs bytes. The appended placement
1084
- emits the wrapper terminator, the element and the end tags, the element plus 17; the
1085
- relocated placement emits none of those and reserves room ahead of the archive instead,
1086
- `Math.ceil(length * 1.01) + 32` bytes in the reference writer, where *length* is the
1087
- element with its tags. The net is that reservation less the element and less the 17
1088
- bytes, so about one percent of the element plus fifteen: what relocation costs is the
1089
- margin, not a second copy. Measured on elements from 61 to 17577 bytes the formula holds
1090
- to within a few bytes, the residual being the element itself changing length between the
1091
- two passes, since the reservation lengthens the prologue and moves every
1092
- central-directory offset with it. The wrapper rung sets the constant: fifteen bytes
1093
- behind a comment, nine behind `</script>` or `]]></svg>`, six behind `</plaintext>`.
1094
- With extraction disabled there is no element and nothing is reserved, so suppressing the
1095
- appended run drops those 17 bytes and nothing else.
1096
-
1097
- The two cases a writer meets differ by an order of magnitude, and the budget is what
1098
- separates them. A relocation forced by `preventAppendedData` acts on whatever element
1099
- exists, which on a small archive is small: 16 bytes on a 2848-byte ZIP region, 35 bytes
1100
- on a 1.3 MB one. A relocation the budget triggers cannot be cheap, because it happens
1101
- only once the element no longer fits: at the default 16361 that means an element past
1102
- 16344 bytes, and 185 bytes measured on a 12.7 MB region is near the least it can cost.
1103
- It keeps rising from there, since a relocated element sits in the prologue and no comment
1104
- ceiling bounds it — at the ratio above, a 40 MB archive carries roughly 57 KB of element
1105
- and costs roughly 590 bytes. A writer sizing a file should compute the cost from the
1106
- element it produced rather than quote any of these figures.
1107
-
1108
- ### 5.3 Offset bookkeeping
1109
-
1110
- Three coordinate systems coexist in one file, and the format's job is to keep each
1111
- self-consistent:
1112
-
1113
- - **ZIP offsets are absolute file positions.** The writer is told the size of
1114
- everything already emitted before the first local header, so the central directory
1115
- offset in the EOCD and every local-header offset in the central directory are true
1116
- file positions (§4.2), so a reader of the *whole file* never needs prepended-data
1117
- compensation — the repair by which a reader recomputes offsets that disagree with
1118
- the file size. A reader of the recovered ZIP region alone does need it (§4.5).
1119
-
1120
- The alternative, offsets relative to the start of the region, is not a compatibility
1121
- problem in itself: a reader that compensates arrives at the same entries, and 7-Zip
1122
- opens such a file when told the type. What absolute offsets buy is the step before
1123
- that. The file is a valid archive read as it stands, so it survives format
1124
- auto-detection — 7-Zip reports a base of 0 and a physical size covering the whole
1125
- file — and the compensation is confined to the one path that cannot avoid it,
1126
- universal-mode recovery. Nothing in the format depends on the choice; a writer using
1127
- the other form produces files this document's readers still open.
1128
- - **PDF offsets are header-relative.** The document's own cross-reference offsets are
1129
- interpreted from the `%PDF-` header, so embedding it needs no rewriting; the writer
1130
- only MUST keep the header inside the scan window (§4.3).
1131
- - **PNG has no offsets, only lengths.** Each chunk declares its data length. The
1132
- `tEXt "ZIP"` chunk's length covers the whole archive, and the appended data too in
1133
- the variants that have it, so it can only be written once the file's final size is
1134
- known, and the writer patches it in place at the end (§6). A `tEXt` chunk precedes it
1135
- whenever the HTML face or the PDF face is present — carrying the prologue or the PDF
1136
- document respectively — and under the PNG face alone it is the only one. Appended
1137
- data follows it only under the HTML face.
1138
-
1139
- The injected `page.pdf` central record exploits a fourth, deliberate discrepancy. It
1140
- is written directly to the output stream, bypassing the ZIP writer's own byte
1141
- counter, at exactly the position where the central directory is about to start.
1142
- The writer's counter is therefore left *behind* the true stream position by exactly
1143
- the record's length, so the central-directory offset it stores lands on the injected
1144
- record rather than after it: the stored offset needs no correction and the record
1145
- becomes the first entry of the directory. The accounting does need correcting: after
1146
- the archive is closed the writer increments the entry counts and adds the record's
1147
- length to the directory size (§6).
1148
-
1149
- ### 5.4 Checksum inventory
1150
-
1151
- Four independent integrity mechanisms cover overlapping byte ranges. All three CRC-32
1152
- variants are the standard ZIP and PNG CRC-32: reflected polynomial `0xEDB88320`,
1153
- initial value `0xFFFFFFFF`, final complement, processing each byte
1154
- least-significant-bit first — the function `zlib.crc32` and its equivalents compute.
1155
- One table therefore serves all three, but they cover different ranges and live in
1156
- different structures:
1157
-
1158
- | Checksum | Covers | Stored in |
1159
- |---|---|---|
1160
- | ZIP entry CRC-32 | one entry's *uncompressed* content | local file header and central-directory record of that entry, including the hand-built `page.pdf` records. Zero for AES-encrypted entries, whose integrity comes from their authentication code instead |
1161
- | PNG chunk CRC-32 | one chunk's type and data bytes | the 4 bytes following each chunk's data. For `tEXt "ZIP"` this spans the whole ZIP region and the appended data |
1162
- | Universal payload CRC-32 | the recovered range as the extractor re-encodes it — the ZIP region without its comment-length field (§1.3) | the recovery payload, with the range's length and newline count (§4.5) |
1163
- | AES authentication code | one encrypted entry's stored bytes | that entry's data, when a password is set |
1164
-
1165
- The PDF face contributes none: PDF has no whole-file checksum, so the document can sit
1166
- inside a larger file unchanged.
1167
-
1168
- Two of these — the `tEXt "ZIP"` chunk CRC and the universal payload — can only be
1169
- computed when the file is otherwise final, which fixes the last steps of the writer's
1170
- order (§6).
1171
-
1172
- ### 5.5 The character round trip
1173
-
1174
- Universal mode recovers the ZIP region from characters rather than bytes (§2.1). The
1175
- inverse mapping the extractor applies is, for the declared charset:
1176
-
1177
- 1. **A code point equal to a byte value that decodes to itself → that byte.** Under
1178
- windows-1252 this covers 229 of the 256 values.
1179
- 2. **Every other code point → a fixed reverse table.** The table is the inverse of the
1180
- encoding's index in the WHATWG standard, restricted to the byte values it does not
1181
- map to themselves: 27 entries for windows-1252 — the printable characters it places
1182
- in the 0x80–0x9F range (typographic quotes, dashes, the euro sign and so on). That
1183
- count is this rule's table alone; an implementation that folds rule 3 into the same
1184
- lookup, as the reference extractor does, has 28.
1185
- Deriving this table is mechanical, so a reader supports any qualifying charset the
1186
- same way — but it MUST be derived from the WHATWG index and not from the platform's
1187
- codec of the same name, which is usually not the same mapping. The WHATWG index
1188
- assigns every byte a code point; most platform codecs leave five positions of
1189
- windows-1252 undefined:
1190
-
1191
- | Byte | 0x81 | 0x8D | 0x8F | 0x90 | 0x9D |
1192
- |---|---|---|---|---|---|
1193
- | WHATWG | U+0081 | U+008D | U+008F | U+0090 | U+009D |
1194
- | Python `cp1252`, Java `windows-1252` | undefined | undefined | undefined | undefined | undefined |
1195
-
1196
- Those five bytes occur in ordinary compressed data, so a strict platform decode
1197
- raises on essentially every archive. Configuring the decoder to replace what it
1198
- cannot map is worse: it emits U+FFFD, which rule 3 below turns into NUL, corrupting
1199
- one byte per occurrence. Measured over ten specimen archives, 42 to 1468 bytes per
1200
- file would be lost this way. The payload checksum catches it.
1201
- 3. **U+FFFD → 0x00.** No byte decodes to U+FFFD under a qualifying encoding (§2.1), so
1202
- the replacement character can only have come from a NUL byte. This holds because
1203
- the payload is inside a wrapper: in every tokenizer state the ladder of §5.1
1204
- produces — comment, raw text, script data, CDATA section, plaintext — the parser
1205
- replaces NUL with U+FFFD.
1206
- 4. **Newlines from the payload.** The parser normalizes CR and CR LF to LF, so the
1207
- original byte sequence is unrecoverable from the text alone; each newline consumes
1208
- the next 2-bit code (0 = LF, 1 = CR, 2 = CR LF).
1209
-
1210
- The payload itself is a sequence of little-endian 32-bit words — checksum, recovered
1211
- range length, newline count, then the codes packed 16 per word, least-significant pair
1212
- first — raw-deflated and base64-encoded with the standard alphabet and padding.
1213
-
1214
- Those word widths cap what the payload can describe. A writer MUST NOT use universal
1215
- mode for a ZIP region of 2^32 bytes or more, since the length field cannot express it.
1216
- The cap is not enforced by the wire format itself: a writer that ignores it stores the
1217
- length modulo 2^32 and produces a file that looks well-formed, and the mismatch
1218
- surfaces only when a reader verifies the field (§4.5). The reference writer is in that
1219
- position — it assigns the length into a `Uint32Array`, where the truncation is silent
1220
- — and reaches the cap in no saved page. This bound and zip64 (§5.7) are separate
1221
- things: zip64 is reachable at any archive size through the 65535-entry trigger and
1222
- stays compatible with universal mode, and it is only a region large enough to need
1223
- zip64's 64-bit *offsets* that runs past what the payload can describe.
1224
-
1225
- An engine limit binds long before the format's. The extractor holds the region as one
1226
- JavaScript string, and the maximum string length is engine-specific: V8 caps it at
1227
- 2^29 − 24 characters, 536870888, measured on V8 15.0.245. A universal-mode archive
1228
- whose ZIP region approaches half a gigabyte is therefore already unreadable in Chrome,
1229
- Edge and Node, whatever the payload declares. Other engines set the limit elsewhere.
1230
- The practical ceiling on universal mode is this one, not the 4 GiB above.
1231
-
1232
- ### 5.6 Password scope
1233
-
1234
- A password encrypts the *contents* of ZIP entries with AES, and nothing else. A reader
1235
- gets no protection beyond that. Four consequences follow:
1236
-
1237
- - **`page.pdf` is never encrypted** and never compressed: its bytes double as the PDF
1238
- face, which a viewer reads directly from the entry's data region (§4.3).
1239
- - **The PNG and PDF faces stay in the clear.** They render the page, and a viewer
1240
- reads their bytes directly (§4.3, §4.4), so they cannot be encrypted without
1241
- destroying the face. A password on an archive that also has one of them protects the
1242
- archived resources, not the page's visible content.
1243
- - **Entry metadata is never encrypted.** Names, uncompressed sizes and dates remain
1244
- readable in the central directory, so the resource list of an encrypted archive is
1245
- public. This is standard ZIP behavior, not a property of this format; §7 restates it.
1246
- - **What the writer withholds instead.** Five things are not forced into the clear by
1247
- the format, and so are withheld when a password is set. Three of them state a URL:
1248
- the entry comments, which publish every resource's source URL (§4.2), and two
1249
- prologue fields carrying the address the page was saved from — the provenance
1250
- comment an implementation may write there, and the canonical `<link>` among the head
1251
- elements (§3.1). The other two are the `<title>` element's text, leaving an empty
1252
- `<title></title>` in the prologue, and the optional text body, which repeats the
1253
- whole page text outside the archive (§4.6). Nothing is lost by leaving any of them
1254
- out: `manifest.json` holds the page URL, the title and the resource-URL map, and it
1255
- is an encrypted entry like the rest. Unlike the PNG and PDF faces, none of the five
1256
- is load-bearing for a reader, so a writer that emits them in a password-protected
1257
- archive publishes what the password is meant to cover for no gain.
1258
-
1259
- Encrypted entries are stamped AE-2, so their CRC-32 field is zero (§5.4). `page.pdf`
1260
- stays unencrypted, so in a password-protected archive its checksum is the only one a
1261
- ZIP tool can verify.
1262
-
1263
- ### 5.7 zip64
1264
-
1265
- The archive uses the zip64 end of central directory structures whenever the ordinary
1266
- records cannot express it: a central directory starting beyond 4 GiB (the prefix counts
1267
- toward the offset, §5.3), a directory 4 GiB or longer, or 65535 entries or more. A
1268
- single entry of 4 GiB or more also produces zip64 extra fields, in that entry's local
1269
- and central headers, without any zip64 end of central directory record. The reference
1270
- writer never requests zip64 explicitly, so it appears only when reached, and given how
1271
- large that is, effectively never in a saved page.
1272
-
1273
- When it is reached, the EOCD record carries the sentinel values `0xFFFF` and
1274
- `0xFFFFFFFF`, preceded by a zip64 end of central directory record and its locator.
1275
- The sentinels are not selective: the writer saturates the entry counts, the directory
1276
- size and the directory offset together once zip64 is emitted, whichever one of them
1277
- overflowed. The `page.pdf` record injection then applies its accounting to the zip64
1278
- record instead — entry counts and directory size there, and
1279
- the locator's pointer moved by the record's length — while leaving each saturated
1280
- field at its sentinel. A writer MUST NOT let the injection push a 16-bit or 32-bit
1281
- field to its sentinel value without emitting the corresponding zip64 record: a count
1282
- of `0xFFFF` sends readers looking for a zip64 record that does not exist.
1283
-
1284
- `test/sfz-harness/zip64.js` covers this: the sentinels stay, the counts and the
1285
- directory size land in the zip64 record, its directory offset points at the injected
1286
- record, `page.pdf` is the first record in the directory, and a reader lists every entry.
1287
-
1288
- zip64 does not conflict with universal mode. Its commonest trigger, 65535 entries or
1289
- more, is reached at any archive size, and §4.5 gives the offset arithmetic for a
1290
- recovered region whose EOCD fields are sentinels. What universal mode cannot carry is
1291
- a ZIP region of 2^32 bytes or more, which the recovery payload's 32-bit length field
1292
- cannot express (§5.5) — a size bound, not a zip64 one.
1293
-
1294
- ### 5.8 Entry name encoding
1295
-
1296
- Entry names in this format are arbitrary Unicode, and how a name is decoded is an
1297
- interoperability question rather than a detail.
1298
-
1299
- The reference writer never exercises that range. Its names are a fixed prefix, an
1300
- index and an extension — `index.html`, `manifest.json`, `stylesheet_0.css`,
1301
- `images/1.png`, `fonts/2.woff2`, `scripts/3.js`, `frames/4/`, `page.pdf` — and the
1302
- extension comes either from a table of content types or from a URL pathname, which is
1303
- percent-encoded. Every name it writes is therefore ASCII, whatever the language of the
1304
- captured page. That is a property of this writer, not a guarantee of the format: a
1305
- conforming writer may name entries after the resources themselves, and §7.3's rule
1306
- that entry names are untrusted assumes one does.
1307
-
1308
- How a name is encoded is ZIP's own business, not this format's: bit 11 of the general
1309
- purpose bit flag selects UTF-8, and its absence selects the legacy code page. This
1310
- document adds two requirements to that and specifies nothing else about it.
1311
-
1312
- **A writer MUST set bit 11 whenever a name or a comment needs it**, and the rule for
1313
- when it does is ZIP's, not this format's: an encoded name or comment holding a byte
1314
- outside printable ASCII needs it, one holding only printable ASCII does not, since the
1315
- two encodings agree there. Control characters count as needing it, the legacy code page
1316
- mapping them to graphic characters rather than to themselves. Setting it on names that
1317
- do not need it is allowed and used to be required here; it was dropped because readers
1318
- disagree about the flag more than they disagree about ASCII, so the safest name is the
1319
- one that does not exercise the question. A writer MUST NOT set it on a name it then
1320
- encodes in the legacy code page, which is the one combination that is simply wrong.
1321
-
1322
- **A reader MUST honor the flag** rather than assume one encoding, and MUST expect to
1323
- meet a clear one — which, in an archive from the reference writer, is most of them:
1324
- that writer percent-encodes every name it produces, so every name is printable ASCII
1325
- and carries no flag, while an entry comment holding the original URL of a resource can
1326
- carry one when that URL is not ASCII. The hand-built `page.pdf` records (§3.1, §6) are
1327
- the only ones the reference writer does not produce through its ZIP writer, and they
1328
- follow the same rule: `page.pdf` is ASCII, so they carry no flag either, and no entry
1329
- in the archive is decoded differently from the rest. Archives written before this rule
1330
- was relaxed carry the flag on every entry instead. Both decode identically, which is
1331
- the point, but a reader that hardcodes either answer meets the other one eventually.
1332
-
1333
- A name is not a path. §7.3's rule that entry names are untrusted applies to the decoded
1334
- name, and decoding is the step before that check, not a substitute for it.
1335
-
1336
- ## 6. Writer algorithm
1337
-
1338
- This section specifies the reference writer's build order. It is normative in the
1339
- sense that a file produced differently but satisfying every rule above is a valid
1340
- SingleFile archive; the order matters because several values can only be computed
1341
- once later bytes exist.
1342
-
1343
- Most of that difficulty is optional, and a writer should know how much of it each face
1344
- buys. The cost is not evenly spread:
1345
-
1346
- | To produce | The writer needs |
1347
- |---|---|
1348
- | The ZIP face alone | Nothing from this section. Write an ordinary archive with `index.html` first and a `manifest.json`; no wrapper, no retry, no patching |
1349
- | Plus the HTML face | The prologue and bootstrap, and the wrapper ladder of §5.1 — one scan of the finished archive, and a rebuild if the rung changes |
1350
- | Plus universal mode | The recovery payload, the character round trip of §5.5, the appended-data budget of §5.2, and the retry loops of §6.2. This is where the real complexity lives, and it buys opening the file from `file:` with no cooperation |
1351
- | Plus the PDF or PNG face | The header window of §4.3 or the chunk patching of §5.3, plus a second wrapper choice for the embedded payload |
1352
-
1353
- The second row already needs a rebuild when the rung changes; the third adds the rest of
1354
- the retry loops and the first value computed only once the archive is final, the
1355
- recovery payload; the fourth adds the second such value, the PNG chunk length and CRC
1356
- (§5.4). A writer that only wants durable saved
1357
- pages can stop at the first row; the files it produces are accepted by every reader in
1358
- §8.1.
1359
-
1360
- ### 6.1 Build order
1361
-
1362
- 1. **PNG head.** With the PNG face, copy the signature and `IHDR` from the source
1363
- image unchanged. With the HTML face, choose the wrapper for the pixel-data payload
1364
- (§5.1) and emit the `tEXt "PNG"` chunk: its 12 header bytes, the head of the
1365
- prologue through `<body hidden>` as built in steps 2 and 3, the wrapper start tag,
1366
- and the chunk CRC. Without the HTML face but with the PDF face, emit the
1367
- `tEXt "PDF"` chunk holding the PDF document here instead, so its header falls
1368
- inside the PDF scan window (§4.3). Then copy every source chunk between `IHDR` and
1369
- `IEND`, write the `tEXt "ZIP"` chunk header with a zero length that step 12 patches,
1370
- and, with the HTML face, the pixel-data wrapper end tag.
1371
- 2. **HTML prologue.** With the HTML face, emit the doctype (omitted under the PNG
1372
- face, which owns the start of the file), the root element start tag, the
1373
- `<meta charset>` required by §2.1, any comment the implementation adds — after the
1374
- charset declaration, since a comment carrying the page URL has no bound and would
1375
- otherwise push that declaration out of the first 1024 bytes. The doctype is the
1376
- other unbounded region ahead of the declaration, copied from the saved page with its
1377
- identifiers verbatim, so a writer MUST emit a minimal doctype in its place when
1378
- keeping it would push the declaration past 1024 bytes.
1379
-
1380
- **Replace it; do not truncate it, and do not drop it.** Truncation is unsafe:
1381
- a cut inside a quoted identifier leaves the tokenizer in the system-identifier
1382
- state, where it consumes the markup that follows until the next `>` — swallowing
1383
- the root element start tag and the `data-sfz` marker on it (§1.3), so the document
1384
- loses both. Dropping the doctype parses
1385
- cleanly but puts the document in quirks mode, which is the mode the blank-page
1386
- backstop, the wait message and the error message are then rendered under (§4.1) —
1387
- the error message most of all, since it is what a reader sees precisely when
1388
- nothing else has worked. A minimal doctype is 15 bytes, keeps standards mode, and
1389
- costs nothing else: the extracted page is written into the document with its own
1390
- doctype (§4.1), so the outer one never governs the restored page.
1391
-
1392
- The PNG face is the exception, and nothing is available to it either way. A PNG file
1393
- MUST begin with its 8-byte signature, so no doctype can precede it, and one written
1394
- after the PNG head is discarded: those bytes are character data, so the parser has
1395
- left its initial insertion mode and ignores a DOCTYPE token. The variant renders in
1396
- quirks mode until the extracted page replaces it, whatever the writer does, so this
1397
- section requires nothing about the doctype there. A writer MAY drop it, as the
1398
- reference writer does, or keep it — but a kept one is content like any other, and
1399
- both windows are measured from the start of the *file*, which under this face begins
1400
- 45 bytes before the HTML does: the signature, `IHDR`, and the chunk length, type,
1401
- keyword and NUL separator. That is 45 bytes less room than the arithmetic above
1402
- suggests.
1403
-
1404
- Then the head elements (the
1405
- `<title>` and the canonical link among them), the CSS and `<body hidden>`,
1406
- the wait and error messages, the optional text body, and the
1407
- bootstrap script. With a password, five of those are left out: the comment, the
1408
- title, the canonical link, the text body and the entry comments of step 6 (§5.6).
1409
- With the PNG face the head of this region,
1410
- through `<body hidden>`, is the data of the `tEXt "PNG"` chunk and the remainder is
1411
- emitted after the `tEXt "ZIP"` chunk header, which step 1 has already written and
1412
- step 12 only patches; with the PDF face the
1413
- region is interrupted by step 3 as well.
1414
-
1415
- Whatever a writer puts in the prologue, closing every element it opens before the
1416
- wrapper start tag is good practice but not a requirement: the extractor addresses
1417
- the ZIP region by identifier (§4.5), so an element left open only makes the archive
1418
- a descendant of it, and the lookup resolves the same way.
1419
- 3. **Embedded PDF.** With the PDF face and the HTML face, the prologue is *split*
1420
- around the PDF, which MUST come early enough for `%PDF-` to start at offset 1024
1421
- or lower (§4.3). Only what a parser needs first precedes it — the doctype, the
1422
- root element and the charset declaration — and everything
1423
- else in the head (title, link and meta elements, the stylesheet, `<body hidden>`,
1424
- the messages, the optional text body) follows it. Emit the
1425
- wrapper start tag chosen for the PDF payload (§5.1), the hand-built `page.pdf`
1426
- local file header, the PDF document, the wrapper end tag, and record the local
1427
- header's absolute position; then resume the prologue. The reference writer's
1428
- header declares version 2.0, no general purpose bit flag, method STORE, the
1429
- build's modification date in DOS form, the precomputed CRC-32, the document's
1430
- length as both sizes, and no extra field; its central record adds a Unix
1431
- "made by" version and external attributes of a regular file, mode 0644.
1432
-
1433
- The window is reachable but not structurally guaranteed, and it is the one place
1434
- where the format depends on the writer rather than on its own layout. The
1435
- irreducible part of the prefix is small: the root element start tag, the charset
1436
- declaration, the wrapper start tag and the 38-byte local file header for
1437
- `page.pdf`, plus a minimal doctype — 92 bytes in the reference layout with a
1438
- `utf-8` label, 99 with `windows-1252`, and a few more with a wrapper past the first
1439
- rung (§5.1). But two
1440
- regions ahead of the header have no length the format controls: the doctype, which
1441
- is copied from the saved page and carries its public and system identifiers
1442
- verbatim, and any comment the implementation chooses to write there. Real doctypes
1443
- are small; the longest in common use, XHTML 1.1 with MathML and SVG, is about 140
1444
- bytes, though a crafted one is bounded only by what the parser accepts. Step 2's
1445
- MUST already caps the doctype, but only far enough to keep the charset declaration
1446
- inside the window; this header sits further into the file, behind the wrapper tag
1447
- and a 38-byte local header, so it needs the tighter bound below and the comment
1448
- needs one too. A writer MUST cap them itself, keeping
1449
- everything before the local file header inside the remaining budget of roughly 930
1450
- bytes (about 900 with the PNG face, whose signature, `IHDR` and first chunk header
1451
- take the first 45 bytes of the same window while its variant drops the 15-byte
1452
- doctype in exchange), shortening, dropping or relocating that content instead of emitting a
1453
- header outside the window. A writer that places nothing of unbounded length before
1454
- the PDF block satisfies the rule by construction and needs no check at all.
1455
-
1456
- The reference writer does both. Its provenance comment is emitted after the PDF
1457
- block, so the page URL it carries cannot reach the window at all, and the prefix is
1458
- measured before the header is written: when the page's own doctype would push
1459
- `%PDF-` past 1024, `<!DOCTYPE html>` is emitted in its place, on step 2's rule. A
1460
- writer that must keep the page doctype has to find the room elsewhere.
1461
-
1462
- Without the HTML face, the PDF is simply the first thing in the file and the
1463
- question does not arise.
1464
- 4. **Reserved extra-data.** In universal mode, when a previous pass determined that
1465
- the payload must be relocated (§5.2), emit an empty `<sfz-extra-data>` element
1466
- followed by enough spaces to fill the reservation. The padding sits **outside** the
1467
- element, so the element's text stays exactly the payload. With
1468
- `preventAppendedData` set from the start there is still no reservation on the first
1469
- pass: that pass measures the payload, and the second reserves (§6.2).
1470
- 5. **Wrapper start tag** for the ZIP region, carrying the identifier (§5.1).
1471
- 6. **The archive.** Create the ZIP writer, telling it the number of bytes already
1472
- written so that its offsets are absolute (§5.3). Add `index.html` first, then
1473
- `manifest.json`, then the page's resources, preserving that order in the central
1474
- directory; STORE entries whose content is already compressed and deflate the rest;
1475
- put each resource's source URL in its entry comment; encrypt entry contents if a
1476
- password was given.
1477
- 7. **PDF central record.** With the PDF face and the HTML face, write the record for
1478
- `page.pdf`, with the local header offset from step 3, immediately before closing
1479
- the writer.
1480
- 8. **Close and patch.** Close the archive, then correct the end of central directory
1481
- record for the injected record: entry counts, directory size, and the zip64
1482
- record and locator when present (§5.7).
1483
- 9. **Wrapper check, then the universal payload.** With the HTML face — not only in
1484
- universal mode, since any self-extracting file needs it — read back the ZIP region
1485
- and check it against the current wrapper (§5.1); on a collision, restart (§6.2).
1486
- Then, in universal mode only, compute the region's CRC-32 and its newline codes,
1487
- build and compress the payload, and decide its placement against the budget
1488
- (§5.2), restarting when an appended payload turns out not to fit. Relocation is
1489
- never undone (§6.2).
1490
- 10. **Appended run.** Unless appended data is prevented or the payload is relocated
1491
- (§5.2), emit the wrapper end tag,
1492
- the extra-data element when it is appended, and `</body></html>` — the end tags
1493
- are omitted under the PNG face, which must end with `IEND`.
1494
- 11. **Fill the reservation.** In the relocated placement, write the payload into the
1495
- space reserved in step 4; if it no longer fits, restart (§6.2). Under
1496
- `declareAppendedData` (§4.2), the EOCD's comment-length field is patched here too,
1497
- the appended run's length now being final, unless the value would complete a
1498
- pattern of the current wrapper, in which case the raw form stays (§5.1).
1499
- 12. **PNG tail.** With the PNG face, patch the `tEXt "ZIP"` chunk's length field, now
1500
- that the total size is known, compute that chunk's CRC over everything from its
1501
- type to the last byte written, and append the CRC and the `IEND` chunk.
1502
-
1503
- ### 6.2 The retry loops
1504
-
1505
- Four conditions restart the build from step 1, and each restart carries forward what
1506
- the failed pass learned. Nothing a restart changes reaches the entries' bytes: a writer
1507
- may compress them once and copy them into every pass, rewriting only the central
1508
- directory's offsets, which is what the reference writer does. The first three
1509
- terminate because each of them advances a monotone quantity:
1510
-
1511
- - **Wrapper collision** (§5.1): the next pass starts at the next rung of the ladder.
1512
- The ladder is finite and its last rung, `<plaintext>`, is exempt from both selection
1513
- tests, so it always fits.
1514
- - **Payload does not fit the appended budget** (§5.2): the next pass reserves room
1515
- ahead of the archive, sized at the measured payload length plus a margin.
1516
- - **Reservation too small**: relocating the payload changes the file's layout, hence
1517
- its offsets, hence the payload, which can grow past the room reserved for it. The
1518
- next pass reserves the new length plus the same margin. This restart fires only
1519
- when the payload outgrew its reservation and it reserves at least that payload, so
1520
- every reservation is larger than the one before and the loop cannot revisit a size.
1521
- What keeps it short is the margin. Shifting the offsets changes a few of the
1522
- central directory's bytes, which changes the line-ending codes, the deflate output
1523
- and the base64 rounding, so the payload moves by a few quanta of 4 characters
1524
- between two layouts: measured between -16 and +20 characters over archives of 8 to
1525
- 2000 entries. A margin smaller than that shift buys a third pass in about one build
1526
- out of four. The writer reserves the measured length plus 1 % plus 32 characters,
1527
- which absorbed every shift measured.
1528
-
1529
- There is no converse of the second: a pass that reserved room never discards it,
1530
- even when the relocated payload would have fit the appended window. Relocation moves
1531
- the archive, which changes the offsets, which changes the payload that made the
1532
- relocation necessary, so a payload lying on the budget boundary can be too large
1533
- appended and small enough relocated, and a writer that dropped the reservation could
1534
- rebuild the two placements forever. Relocation is therefore final (§5.2), and the file
1535
- keeps at most the reservation's own margin of dead padding.
1536
-
1537
- The fourth stands apart from the other three, and terminates trivially because it can
1538
- fire only once: if the end of central directory record cannot be patched to account for
1539
- the injected `page.pdf` record — its signature not where the accounting expects it —
1540
- the writer rebuilds without that record rather than leave a central directory the EOCD
1541
- does not count. That is the restart enforcing §5.7's requirement that the injection
1542
- never leave the two disagreeing, and the rebuilt archive simply has no `page.pdf`
1543
- entry.
1544
-
1545
- Given identical inputs, modification date and archive time, the process is
1546
- deterministic: the same page produces the same bytes, retries included. `manifest.json`
1547
- records when the archive was made (§7.1), so two builds of one page at two moments
1548
- differ in that entry and in the entry sizes around it. A writer that retries MUST pin
1549
- the archive time across the passes of one build rather than read the clock again on
1550
- each. The reference writer reads the clock once, inside the callback that emits the
1551
- entries, which runs once per build; every retry reuses the entries that callback
1552
- produced. Two builds of one page still read the clock twice, so its own determinism
1553
- test freezes it. A consumer MUST NOT
1554
- treat the byte identity of two archives of the same page as meaningful.
1555
-
1556
- ## 7. Consuming SingleFile archives safely
1557
-
1558
- This section addresses software that reads SingleFile archives it did not produce.
1559
- The ZIP face is the interoperable one, and a reader that follows the rules below
1560
- handles every variant of §2 without knowing which one it has.
1561
-
1562
- ### 7.1 Reading
1563
-
1564
- - **Read through the central directory.** Locate the End Of Central Directory record
1565
- by scanning backward from the end of the file, then follow its offset. A reader that
1566
- streams local headers from offset 0 will not find an archive in any variant that has
1567
- a face, since the file then starts with the HTML, PDF or PNG face. The variant with
1568
- no face is an ordinary ZIP file and streams fine (§1.2, and §8.1 measures what such
1569
- readers actually do).
1570
- - **Tolerate bytes before and after the archive.** They are the other faces, not
1571
- corruption. Offsets are absolute, so no compensation is needed (§5.3).
1572
- - **Accept both forms of appended data.** The bytes after the EOCD record may be raw
1573
- or declared as the archive comment; both are valid (§4.2). A reader MUST NOT treat
1574
- undeclared trailing bytes as a defect.
1575
- - **Do not identify the format by file name.** Extensions are conventions (§2.2). A
1576
- SingleFile archive is identifiable from its content, and recognition and extraction
1577
- use different tests: an `index.html` entry accompanied by a `manifest.json` entry in
1578
- the same directory is the positive signal for recognizing the format, while
1579
- `index.html` alone is enough to *extract* from, since `manifest.json` is informative
1580
- and MUST NOT be required (below). Neither test distinguishes this format from an
1581
- arbitrary ZIP file laid out the same way, and none is offered, because nothing in the
1582
- format depends on recognizing it. A reader that treats any archive containing a page
1583
- entry as a saved page loses nothing.
1584
- - **Resolve the page entry in this order.** The archive's internal layout is
1585
- implementation-defined, and two properties of the reference layout matter to a
1586
- reader. Every entry MAY sit under a single root directory, which the reference
1587
- writer names `<milliseconds since the epoch>_<tab id>/` when asked to create one
1588
- (`createRootDirectory`); the page is then `<root>/index.html`. And a page's nested
1589
- frames are stored as complete pages of their own under `frames/<n>/`, recursively,
1590
- each with its own `index.html` and `manifest.json`, so an archive normally holds
1591
- several of both and only the outermost pair is the page. The recognition test above
1592
- therefore matches every frame directory too. Since neither property
1593
- is guaranteed, a reader resolves the entry point in three steps, stopping at the
1594
- first that succeeds:
1595
-
1596
- 1. The `indexFilename` of the `manifest.json` at the smallest directory depth,
1597
- resolved against that manifest's directory, when the entry exists. This is the
1598
- only authoritative answer, so a writer that departs from
1599
- the reference layout SHOULD emit the manifest even though a reader MUST NOT
1600
- require it.
1601
- 2. Otherwise the `index.html` entry at the smallest directory depth.
1602
- 3. If several `index.html` entries tie at that depth, the archive does not name its
1603
- page: a reader MUST NOT pick one arbitrarily. Report the ambiguity, or treat the
1604
- file as a plain ZIP archive.
1605
-
1606
- The reference writer never produces a tie, since it creates at most one root
1607
- directory and nests every other page under `frames/<n>/`; step 3 exists for archives
1608
- from other writers.
1609
- - **Treat `manifest.json` as informative.** The reference writer records the original
1610
- URL as `originalUrl`, the title as `title`, the save time as `archiveTime` (an ISO
1611
- 8601 string), the entry name of the page as `indexFilename` and the resource-to-URL
1612
- map as `resources`. The page displays without any of it, and a reader MUST NOT require
1613
- the entry or any field of it. `indexFilename` names the page relative to the
1614
- manifest's own directory, not as a full entry name. A frame's manifest carries the
1615
- same `archiveTime` as the page's. The set of fields is not closed: a reader
1616
- MUST ignore what it does not recognize.
1617
- - **Expect a `page.pdf` entry whose data lies outside the archive proper** (§4.2). It
1618
- is an ordinary STORE entry at an ordinary offset, so nothing special is needed to
1619
- read it, but a reader that assumes the entries are contiguous will reject or
1620
- mislocate it: `page.pdf`'s local header is the first in the file, and the whole
1621
- bootstrap lies between its data and the next one. It is never placed under the root
1622
- directory: the PDF face is one document per file, so an archive holds at most one
1623
- `page.pdf`, and it sits at the top level whatever `createRootDirectory` does to the
1624
- other entries.
1625
-
1626
- ### 7.2 Modifying
1627
-
1628
- Do not rewrite the file in place. Adding, removing or recompressing entries moves the
1629
- ZIP region and invalidates the other faces: the PNG chunk length and CRC that span the
1630
- archive, the recovery payload's checksum, and the wrapper choice that depends on the
1631
- archive's exact bytes (§1.2). Most ZIP rewriters also drop the prepended and appended
1632
- regions, which discards every face but ZIP.
1633
-
1634
- A tool that wants to produce a modified archive MUST rebuild it through the writer
1635
- rules of §6. A tool that only wants the page content SHOULD extract rather than
1636
- rewrite.
1637
-
1638
- The hazard is automated handling: a deduplicating store, a backup system that
1639
- recompresses, a mail or chat service that repacks attachments, or any pipeline that
1640
- round-trips the file through a ZIP library will return an archive whose entries are all
1641
- intact and whose other faces are gone, with no error at any step and no record of it in
1642
- the file. Software that stores these archives SHOULD treat them as opaque bytes, and a
1643
- preservation workflow that cannot guarantee that SHOULD keep a checksum of the original
1644
- alongside it.
1645
-
1646
- ### 7.3 Security considerations
1647
-
1648
- - **Entry names are untrusted.** The reference writer's names are a fixed prefix, an
1649
- index and an extension (§5.8), but nothing in the format requires that, and a writer
1650
- may name entries after the resources themselves. A reader MUST sanitize them before
1651
- writing to a filesystem: reject absolute paths and
1652
- `..` segments, and be aware that names may be long, may collide after case folding,
1653
- and may contain characters the local filesystem rejects.
1654
- - **Declared sizes are untrusted.** Do not pre-allocate from the declared uncompressed
1655
- size, and enforce a limit on the expansion ratio; the archive can be crafted like
1656
- any other ZIP file.
1657
- - **The archived page is untrusted web content**, and the HTML face contains a script.
1658
- Software that displays either MUST do so in a sandboxed context, and MUST NOT run
1659
- the bootstrap in a privileged one. The format's own display path replaces the
1660
- document with the extracted page, which is not an isolation boundary by itself.
1661
- - **A password protects entry contents only** (§5.6). Entry names, sizes and dates
1662
- stay readable in the central directory, and while the reference writer's names carry
1663
- no information about the resources (§5.8), another writer's may state their
1664
- filenames. The PNG and PDF faces render the page regardless. A conforming writer
1665
- withholds the five fields of §5.6, the source URLs among them, but a reader MUST NOT
1666
- read their absence as protection: nothing in the format stops a writer from emitting
1667
- any of them, so an archive of unknown provenance may state every URL in the clear.
1668
- Software MUST NOT present a password-protected archive as an encrypted document.
1669
- - **Sniffing disagrees with itself on these files.** `file(1)` reports HTML, PNG, PDF
1670
- or "data" depending on the variant (§8.1), so a server that guesses the media type
1671
- from content may serve a saved page as an image. Software that serves SingleFile
1672
- archives SHOULD set the media type explicitly — `text/html` for the self-extracting
1673
- variants, `application/zip` otherwise.
1674
-
1675
- ### 7.4 What to reject and what to tolerate
1676
-
1677
- A reader of a polyglot file meets conditions that look like corruption and are not, and
1678
- others that look harmless but leave the extracted page wrong. A condition matters if and
1679
- only if it affects the bytes the page is built from:
1680
-
1681
- | Condition | Reader behavior |
1682
- |---|---|
1683
- | A recovery payload field disagrees with the reconstruction — length, newline count or checksum | **MUST** fail (§4.5). The reconstruction is wrong and nothing built from it can be trusted |
1684
- | An entry's CRC-32 or AES authentication code does not match | **SHOULD** fail for that entry, and MUST NOT present a page rebuilt from it as intact |
1685
- | `page.pdf` was reconstructed from the parsed page and its CRC-32 does not match | **MUST** discard the reconstruction (§4.5). The bytes are a guess about newlines the recovery payload does not describe, and the checksum is the only thing that tests it — unlike the row above, there is no read to have gone wrong, only an inference |
1686
- | Bytes outside the archive proper — before the first local file header, after the EOCD record, or between an entry's data and the next header | **MUST** tolerate: they are the other faces (§7.1). The gap in the middle is not hypothetical: with the PDF face the bootstrap lies between `page.pdf`'s data and the ZIP region |
1687
- | The appended run exceeds the 65535-byte ceiling (§5.2) | Not a reader's problem: if the EOCD record was found, the archive is readable. Readers MAY warn |
1688
- | A `tEXt` chunk CRC does not match, or a chunk holds bytes PNG does not permit (§4.4) | Irrelevant to extraction; a reader of the archive MAY ignore both |
1689
- | `page.pdf` is present but its data does not begin with `%PDF-` | Not an error. The entry is data like any other |
1690
- | `index.html` is present without `manifest.json` | **MUST** still extract (§7.1) |
1691
- | More than one candidate carries the `sfz-data` identifier once §4.5's tie-break has been applied | **MUST NOT** extract either silently. The tie-break comes first and settles the ordinary pairing: an id-bearing element that is one of §5.1's wrapper rungs wins over a comment, and one that is not a rung loses to it, since the `id` is then something else in the page. What this row forbids is what the tie-break does not reach — two elements, or two comments, or an element and a comment that both survive it. A conforming writer emits one candidate (§5.1), so a second is a payload that escaped its wrapper, most often a nested archive written by a writer that emitted a face bare. Both extract cleanly and check out, and the checksums say nothing about which one the file was built around |
1692
- | The recovered region (universal mode) disagrees with the same bytes read directly, in the EOCD's two comment-length bytes only | Expected, not an error. A recovered region always declares a zero-length comment (§4.5), so it differs here from any archive written in the declared form (§4.2). Compare the two only up to those bytes |
1693
- | The recovered region (universal mode) disagrees with the same bytes read directly, anywhere else | The file is not well-formed, whichever side is at fault, and a reader that has both MUST NOT silently merge them or pick per entry. Prefer the direct read — it is the writer's own output, where the recovered region is a reconstruction of it — and surface the disagreement rather than displaying either as intact |
1694
-
1695
- Anything the format does not constrain, a reader MUST NOT reject: entries may carry any
1696
- extra fields, timestamps or data descriptors a ZIP writer would ordinarily emit. The few
1697
- this document does constrain — the `0x9901` field of an encrypted entry (§4.2), the
1698
- zip64 records (§5.7), the name-encoding flag (§5.8) — say how an entry is read, not
1699
- whether it is acceptable, so this row covers them too: each is something a reader meets
1700
- and reads.
1701
-
1702
- ## 8. Appendices
1703
-
1704
- ### 8.1 Tool compatibility
1705
-
1706
- Measured on macOS 26 with the specimens of §8.3 (Info-ZIP UnZip 6.00, libarchive
1707
- 3.7.4, Python 3.14, OpenJDK 21, 7-Zip 25.01, poppler `pdftotext`, macOS `ditto`,
1708
- `sips` and Quick Look). Every result is predicted by two structural properties, so
1709
- the variants are grouped by them:
1710
-
1711
- | Class | Bytes before the archive | Bytes after the EOCD | Variants |
1712
- |---|---|---|---|
1713
- | A | — | — | pure zip |
1714
- | B | yes | — | relocated (`preventAppendedData`), zip-pdf |
1715
- | C | yes | yes | plain, universal, ladder, password, pdf, png, png-pdf, zip-png, zip-png-pdf, zip64 |
1716
-
1717
- The classes follow the bytes, not the options. `preventAppendedData` puts the
1718
- relocated variant in class B, but it suppresses *markup* after the archive, not the PNG
1719
- face's tail: an archive combining that option with a PNG face has bytes after the EOCD
1720
- and is class C.
1721
-
1722
- | ZIP reader | A | B | C | Behavior |
1723
- |---|---|---|---|---|
1724
- | Info-ZIP `unzip`, `zipinfo` | ✔ | ✔ | ✔ | Lists and extracts every variant. AES entries are skipped — `need PK compat. v5.1 (can do v4.5)` — a limitation of the tool, not of the file; `page.pdf` still extracts because it is never encrypted |
1725
- | Python `zipfile` | ✔ | ✔ | ✔ | Lists and extracts every variant |
1726
- | 7-Zip (`7zz`) | ✔ | ✔ | ✔ | Lists and extracts every variant, AES included |
1727
- | libarchive `bsdtar`, seekable input | ✔ | ✔ | ✔ | Lists and extracts every variant. Its EOCD scan is the narrowest measured, so a class-C file whose appended run pushes the record past 16383 bytes from the end is rejected with `Unrecognized archive format`; §5.2's default budget is sized to this window, and the ✔ holds for files that respect it |
1728
- | libarchive `bsdtar`, piped input | ✔ | ✘ | ✘ | `Unrecognized archive format` — the forward-only case of §1.2, measured |
1729
- | Java `java.util.zip` (`jar tf`) | ✔ | ✔ | ✘ | `zip END header not found` whenever bytes follow the EOCD undeclared. Declaring them as the archive comment makes the same file open, measured on every class-C variant (§4.2) |
1730
- | macOS `ditto -x -k` | ✔ | ✘ | ✘ | `Couldn't read PKZip signature` — requires a local file header at offset 0, so prepended data alone defeats it |
1731
-
1732
- The backward scans behind the class-C column differ by an order of magnitude, and they
1733
- are what §5.2's budget is sized against. Measured by padding a working archive until
1734
- the record fell out of reach, the largest distance from end-of-file at which each
1735
- reader still finds the EOCD record is: libarchive 16383, perl `Archive::Zip` 32768,
1736
- Python `zipfile` 65557, zip.js 65536, Info-ZIP `unzip` 68000, and 7-Zip beyond 1 MiB,
1737
- which scans the whole file. libarchive binds, and its window less the 22-byte record
1738
- is the 16361-byte default budget of §5.2. macOS `ditto` is not on this axis at all: it
1739
- requires a local file header at offset 0 whatever the tail looks like.
1740
-
1741
- The cost of the declared form was measured on the same tools: it is a display cost, not
1742
- a compatibility one. An archive whose trailing bytes are declared as the comment has
1743
- them printed back on ordinary listings; `unzip -l` reproduces the whole run — in
1744
- universal mode that is the `-->`, the entire `<sfz-extra-data>` element and the end tags
1745
- — under the archive's own header. The raw form is the default for that reason (§4.2).
1746
-
1747
- The other faces were exercised on the variants that carry them, and all succeeded:
1748
- `pdftotext` extracts the page text from every PDF-face variant, including the
1749
- password-protected one (`page.pdf` is never encrypted, §5.6), and `sips` reports the
1750
- screenshot's true dimensions for every PNG-face variant. macOS Quick Look renders the
1751
- PNG face of the self-extracting PNG specimen, and on the all-four-faces specimen it
1752
- renders both faces of the same bytes: renamed to `.png` it yields the screenshot,
1753
- renamed to `.pdf` the rendered document.
1754
-
1755
- The PDF header window of §4.3 was measured rather than assumed, by moving `%PDF-`
1756
- progressively later in otherwise identical archives. PDFium (build 153.0.7999.0, the
1757
- engine of every Chromium-based browser) loads the document while the header starts at
1758
- offset 1024 or less and fails with a data-format error from 1025 on — the documented
1759
- 1024-byte figure, enforced exactly. poppler `pdftotext` and macOS PDFKit render the
1760
- same files with the header at 1000222 bytes, so they impose no window at all. A
1761
- writer that keeps the header inside 1024 bytes satisfies every engine tested.
1762
-
1763
- `file(1)` disagrees with itself across the variants. What it reports depends on
1764
- where the archive falls relative to the fixed buffer it sniffs, not on the variant as
1765
- such: a small self-extracting file whose archive starts within that buffer is
1766
- reported as `data` where a large one is reported as HTML. On these specimens:
1767
- `HTML document text` for the plain, universal and ladder specimens, `PNG image data`
1768
- for every PNG-face variant, `PDF document` for a
1769
- PDF-first archive, `Zip archive data` for a pure archive **and for the relocated
1770
- variant**, whose first bytes are a doctype, and `data` where the HTML head carries
1771
- the embedded PDF.
1772
-
1773
- Two of these results bear on §1.1's customary tolerances. The EOCD backward scan and
1774
- the tolerance of undeclared trailing bytes are near-universal but not unanimous: Java
1775
- is the measured exception, and not a niche one, since `java.util.zip` is what
1776
- Android and most JVM tooling use. Prepended data is tolerated by every ZIP reader
1777
- measured except Apple's `ditto`.
1778
-
1779
- ### 8.2 Anatomy of a small archive
1780
-
1781
- Offsets in `universal.sfz.html` (123077 bytes, two entries, saved from `example.com`
1782
- with the §8.3 command, on the build named there).
1783
- The layout is the *universal* row of the byte map (§3).
1784
-
1785
- These numbers are one capture, not a contract. Everything from the bootstrap onward
1786
- moves whenever the inlined ZIP library changes size, so treat the table as an
1787
- illustration of the shape and not as values to compare a file against. What *is* fixed
1788
- is the set of relations between the rows — the doctype opening the file with the root
1789
- element start tag immediately after it, the charset declaration immediately after that
1790
- and the comment immediately after that, the identifier's twelve bytes ahead of the
1791
- region, the EOCD's directory offset being an absolute file position, and the entry
1792
- order. Those are checked by `test/sfz-harness/byte-map.js`, which builds an equivalent
1793
- specimen without a network.
1794
-
1795
- | Offset | Bytes | Region |
1796
- |---|---|---|
1797
- | 0 | `<!DOCTYPE html>` | `html-prologue` begins |
1798
- | 15 | `<html data-sfz>` | root element start tag; the attribute is the reference implementation's own marker (§1.3) |
1799
- | 30 | `<meta charset=windows-1252>` | the charset rule, inside the first 1024 bytes (§2.1) |
1800
- | 57 | `<!--` … `-->` (ends at 200) | comment written by the implementation; its content is implementation-defined, but where it may appear is not (§3.1, §4.6, §5.6). It follows the charset declaration so it cannot push it out of the prescan window |
1801
- | 200 | `<title>` … `</title>` (ends at 229) | the page title, as numeric character references (§4.6) |
1802
- | 677 | `<style>` | the stylesheet of the blank-page backstop (§4.1) |
1803
- | 855 | `<body hidden>` | start of the blank-page backstop (§4.1) |
1804
- | 868, 913 | wait and error messages | the two visible elements |
1805
- | 1134 | `<script>` … `</script>` (ends at 121993) | `bootstrap`: ZIP reader, extractor, display, acquisition |
1806
- | 121993 | `<!--sfz-data` | wrapper start tag, opening the ZIP region and carrying its identifier (§5.1) |
1807
- | 122005 | `PK\3\4` | first local file header, `index.html` (1110 bytes, deflated to 593) — the ZIP region begins |
1808
- | 122647 | `PK\3\4` | local file header, `manifest.json` (168 bytes, deflated to 130) |
1809
- | 122829 | `PK\1\2` | central directory: `index.html`, then `manifest.json` |
1810
- | 122981 | `PK\5\6` | EOCD: 2 entries, directory size 152, directory offset 122829 — an absolute file position (§5.3) — comment length 0 |
1811
- | 123003 | `-->` | wrapper close tag; the ZIP region ends here |
1812
- | 123006 | `<sfz-extra-data>` … `</sfz-extra-data>` | recovery payload, appended placement (§5.2); 24 base64 characters for this archive |
1813
- | 123063 | `</body></html>` | end tags; end of file at 123077 |
1814
-
1815
- The appended run is 74 bytes, well inside the 16361-byte default budget (§5.2). The ZIP region
1816
- is the 998 bytes from 122005 to 123003; the universal extractor reproduces the first 996
1817
- of them and supplies the last two itself (§1.3).
1818
-
1819
- ### 8.3 Specimens
1820
-
1821
- Generated with the command-line client running `single-file-core` against
1822
- `example.com`. The anatomy of §8.2 was regenerated against 1.5.108; the compatibility
1823
- results of §8.1 were measured on the 1.5.107 build of the same specimen set, which
1824
- differs only inside the prologue and so falls in the same classes: those are grouped
1825
- by whether bytes precede the archive and follow the EOCD, which no prologue change
1826
- alters. The declared-form results are the exception, `declareAppendedData` being later
1827
- than that build (§8.5); they were measured separately on a build that has it. The
1828
- specimen names carry a `.sfz.html` suffix chosen for the harness; the conventions of
1829
- §2.2 are what the clients produce, not what these files are called.
1830
- `--compress-content` makes the output an archive; `extract-data-from-page`
1831
- defaults to true there, so the plain variant has to switch it off:
1832
-
1833
- | Specimen | Command |
1834
- |---|---|
1835
- | pure zip | `single-file --compress-content --self-extracting-archive=false <url> pure.zip` |
1836
- | plain | `single-file --compress-content --extract-data-from-page=false <url> plain.sfz.html` |
1837
- | universal | `single-file --compress-content <url> universal.sfz.html` |
1838
- | relocated | add `--prevent-appended-data` |
1839
- | pdf | add `--embed-pdf` |
1840
- | png | add `--embed-screenshot` |
1841
- | png-pdf | add `--embed-screenshot --embed-pdf` |
1842
- | zip-pdf, zip-png, zip-png-pdf | add `--self-extracting-archive=false` to the pdf, png and png-pdf rows |
1843
- | password | add `--password=<password>` |
1844
-
1845
- These specimens are deliberately small, and a reader tested only against them is
1846
- undertested: they are all flat archives of two or three entries. None
1847
- exercises a root directory, `frames/<n>/` nesting, a second `index.html`, a `data:`-URL
1848
- entry comment, the optional text body (§4.6), a UTF-8 BOM, zip64
1849
- (§5.7), a payload past the appended-data budget, or a relocated reservation with padding left
1850
- in it. Two omissions matter more than the rest, because they are the parts of §5.1 a
1851
- writer is most likely to get wrong: no specimen defeats a rung by its **start**
1852
- pattern, and none defeats one with an **upper-case** pattern. A writer that tested only
1853
- end patterns, or matched them case-sensitively, produces every specimen here unchanged.
1854
-
1855
- Two specimens cannot be produced from a URL alone. The **ladder** specimen, which
1856
- forces the second rung of §5.1, needs a page referencing an image whose stored bytes
1857
- contain `-->`; the archive then wraps in `<script type=sfz-data>`. The **zip64** specimen requires
1858
- an archive past the thresholds of §5.7, so it is produced by calling the writer
1859
- directly with zip64 forced on the ZIP writer, as `test/sfz-harness/zip64.js` does.
1860
-
1861
- The measurements quoted elsewhere in this document come from the same harness: the
1862
- payload growth rate of §5.2 (86 KB → 181 bytes, 283 KB → 465, 1.07 MB → 1645, 4.2 MB
1863
- → 6497, i.e. one byte of element per 650 bytes of archive) and the zip64 verification
1864
- of §5.7.
1865
-
1866
- ### 8.4 The charset round trip, measured
1867
-
1868
- Two claims of §2.1 were verified. The first is re-derived on every run by
1869
- `test/sfz-harness/charset-round-trip.js`, which reads the tables below out of the
1870
- runtime's own decoders rather than trusting this section, and checks the reverse table
1871
- the extractor ships against the one the rule of §5.5 produces.
1872
-
1873
- **Which encodings qualify.** Decoding all 256 byte values through each encoding
1874
- defined by the WHATWG standard shows 20 that are injective and never produce U+FFFD:
1875
- `windows-1252` (and its `iso-8859-1` labels), `iso-8859-2`, `-4`, `-5`, `-10`, `-13`,
1876
- `-14`, `-15`, `-16`, `koi8-r`, `koi8-u`, `macintosh`, `windows-1250`, `-1251`,
1877
- `-1254`, `-1256`, `-1258`, `x-mac-cyrillic`, `ibm866` and `x-user-defined`. The last of
1878
- those qualifies on the criterion but is a poor choice in practice: it maps 0x80–0xFF
1879
- into the Private Use Area, U+F780–U+F7FF, so the payload's characters have no meaning
1880
- outside this round trip and any tool that touches the text sees private-use code points.
1881
- The remaining single-byte encodings have undefined positions in their index —
1882
- `iso-8859-3`, `-6`, `-7`, `-8`, `windows-874`, `-1253`, `-1255`, `-1257` — and the
1883
- multi-byte ones (`utf-8`, `utf-16le`, `utf-16be`, `gbk`, `gb18030`, `big5`, `euc-jp`,
1884
- `shift_jis`, `euc-kr`, `iso-2022-jp`) decode a lone byte sequence to U+FFFD or to fewer
1885
- than 256 characters.
1886
- The reverse table each one needs ranges from 8 entries (`iso-8859-15`) to 128
1887
- (`koi8-r`, `koi8-u`, `ibm866` and `x-user-defined`); windows-1252 needs 27.
1888
-
1889
- **That the round trip is charset-independent.** The mechanism of §5.5 — parse, then
1890
- re-encode with the reverse table, restoring newlines from the 2-bit codes and NUL from
1891
- U+FFFD — was run in a browser on a 4364-byte payload containing every byte value, 49
1892
- newlines covering LF, CR, CR LF and a trailing CR, and NUL bytes. It recovers the
1893
- payload byte for byte under `windows-1252`, `iso-8859-15`, `iso-8859-5`, `koi8-r`,
1894
- `ibm866` and `macintosh`, and fails under `utf-8`, as the injectivity requirement
1895
- predicts.
1896
-
1897
- ### 8.5 Format history
1898
-
1899
- | When | Change |
1900
- |---|---|
1901
- | before 2023 | The format originates in SingleFileZ, a separate extension pairing a self-extracting HTML page with a ZIP archive. The `data-sfz` and `<sfz-extra-data>` identifiers date from there (§1.3) |
1902
- | October 2023 | SingleFileZ's core is merged into single-file-core; universal mode (`extractDataFromPage`) and the wrapper ladder arrive with it |
1903
- | November 2023 | `preventAppendedData`: archives that end exactly at the EOCD record, with the payload relocated ahead of the archive |
1904
- | January 2024 | The PNG face: a screenshot's chunks wrap the archive, the HTML riding in a `tEXt` chunk |
1905
- | September 2024 | The PDF face: an embedded PDF document placed so its header falls in the scan window, first with the HTML face, then for archives without it |
1906
- | August 2026 | The embedded PDF becomes the `page.pdf` ZIP entry, listed first, so ZIP tools see the document as an ordinary entry (core 1.5.93) |
1907
- | August 2026 | Core 1.5.107: page-text extraction skips `page.pdf` (§4.5), the appended-data budget accounts for the PNG tail (§5.2), and archives served with an HTTP error status fall back to page-text extraction (§4.1) |
1908
- | August 2026 | Core 1.5.108: `<noscript>` leaves the wrapper ladder (§5.1), the title is emitted as character references instead of being dropped in universal mode (§4.6), the doctype is capped when it would push `%PDF-` out of the scan window (§6.1), and password-protected archives withhold the entry comments, the title and the text body (§5.6) |
1909
- | August 2026 | Core 1.5.108: the writer enforces the two remaining HTML restrictions on comment text — a payload may not end with `<!-`, and the PNG face may not open its comment with the `>` or `->` its chunk checksum lands on once in 256 archives, which closed the wrapper and left the image data to the parser (§5.1) |
1910
- | August 2026 | Core 1.5.108: the wrapper ladder escalates to `<script type=sfz-data>` and `<style type=sfz-data>` before the raw-text rungs, the only two whose content local text extractors drop the way they drop a comment (§5.1) |
1911
- | August 2026 | Core 1.5.108: the ZIP region carries the identifier `sfz-data` and the extractor addresses it with that instead of deducing it from its position beside `<sfz-extra-data>` (§4.5). This fixes universal extraction on the `<style type=sfz-data>` rung, where the reference extractor's own relocation of `style` elements into the head moved the region out from under the positional rule |
1912
- | August 2026 | Core 1.5.108: the recovery payload stops two bytes short of the End Of Central Directory record, excluding its comment-length field (§1.3), which lets universal-mode archives declare their appended data as the archive comment — a writer option, for `java.util.zip` and the readers that reject undeclared trailing bytes (§4.2) |
1913
- | August 2026 | Core 1.5.108: the PDF and PNG faces test a wrapper rung's start pattern as well as its end pattern, closing the same script-data escape hole the ZIP region was already guarded against — a face payload holding `<!--` and then `<script` took the `<script type=sfz-data>` rung and swallowed the rest of the document (§5.1) |
1914
- | August 2026 | Core 1.5.108: the retry loop never discards a relocation reservation, so a payload sitting on the appended-data boundary cannot oscillate between the two placements forever (§6.2) |
1915
- | August 2026 | Core 1.5.110: a PDF or PNG face whose payload names every rung is dropped instead of written bare (§5.1). Found by nesting an archive inside itself as both faces: the fifth level exhausts the ladder, and readers then extracted the fourth level's archive — checksums intact, no way to tell (§7.4) |
1916
- | August 2026 | Core 1.5.110: a PNG face leaving the comment rung on its checksum resumes the rung search instead of taking the next rung untested (§5.1). Taking it put a payload holding `</script>` on the script rung, where its own bytes closed the wrapper 93 bytes in and left the image data, the chunk framing and the whole ZIP region to the parser |
1917
- | August 2026 | Core 1.5.110: `<svg><![CDATA[` joins the ladder above `<plaintext>` (§5.1) — the one rung whose terminator, `]]>`, real payloads rarely carry. It gives a payload naming every element rung somewhere to go that does not cost the appended-data placement, and moves the self-nesting limit from the fifth level to the sixth |
1918
- | August 2026 | Core 1.5.115: password-protected archives withhold the provenance comment and the canonical link as well (§5.6). Both wrote the page's own URL into the prologue, beside the title that was already withheld, so the address the archive was saved from stayed in the clear |
1919
- | August 2026 | Core 1.5.119: the inlined ZIP library is built ASCII-only, and §2.1 now requires it of any bootstrap. Its CP437 table had been emitted as literal characters, which the page re-decoded as windows-1252, growing the table from 256 entries to 508 and shifting every lookup by 60 — so the one entry read without the UTF-8 flag, `page.pdf`, came back mangled and no archive with a PDF face extracted in any engine (§5.8) |
1920
- | August 2026 | Core 1.5.120: the hand-built `page.pdf` records set the language encoding flag, like every entry the ZIP writer produces (§5.8). Its name is ASCII, so no decoded name changes; what changes is that no entry in an archive is read through CP437 any more, closing the path the 1.5.119 defect surfaced on |
1921
- | September 2026 | §5.8 no longer requires bit 11 on every entry, deferring to ZIP's own rule: the flag is set when a name or a comment holds a byte outside printable ASCII, and left clear otherwise, because readers disagree about the flag more than they disagree about ASCII. The reference writer's names are all percent-encoded, so in practice none of them carries it now, and the hand-built `page.pdf` records follow the writer instead of overriding it — reversing the 1.5.120 row below, whose reason was that `page.pdf` would otherwise be the only entry read through the legacy path. It no longer is: every name in the archive takes the same path again, the other one |
1922
- | September 2026 | Core 1.5.126: the appended-data budget becomes the `maxAppendedDataLength` writer option and its default drops from 65535 to 16361 bytes, so the EOCD record stays inside libarchive's scan and `bsdtar` opens archives it used to reject (§5.2, §8.1). The 65535-byte comment ceiling is now a separate limit, stated in §4.2: a budget raised past it produces a run that cannot be declared |
1923
-
1924
- This document was itself revised in August 2026, against core 1.5.108, after several
1925
- independent reviews. One of them was a reader built from this specification alone, with
1926
- no access to the reference code. It read every specimen correctly, which is the best
1927
- evidence available that the format is implementable from this text, and it found
1928
- thirteen defects in the text. The load-bearing corrections are in §5.1 (the
1929
- start-pattern test is necessary, not conservative: script data's escape states let a
1930
- payload defeat the end-tag test), §5.5 (the WHATWG index is not the platform codec of
1931
- the same name), §4.5 (the offset shift is derivable from the recovered region, and
1932
- `page.pdf` is reachable, just not by offset) and §1.3 (the appended data may be a
1933
- declared archive comment). A second pass by the same implementation caught a regression
1934
- the revision had introduced: giving `<plaintext>` a start pattern would have let 55
1935
- bytes of ASCII defeat all eight rungs, which the reference writer never did and which
1936
- §6.2's termination argument forbids. Later passes added a normative order for resolving
1937
- the page entry, since "the shallowest `index.html`" had no tiebreak (§7.1); the
1938
- distinction between the logical archive and the ZIP region, which `page.pdf` is the one
1939
- entry to fall outside (§1.3); the layer convention now stated at the head of the
1940
- document; the limits of the reconstructed-`page.pdf` CRC check (§4.5); the durability
1941
- ranking of the faces (§1.1); what each face costs a writer (§6); and the silent loss of
1942
- the other faces to a pipeline that repacks the file (§7.2). One review found a live
1943
- defect rather than a documentation one, the non-monotone retry step recorded above.
1944
-
1945
- A later pass found four places where the document contradicted itself or the standard
1946
- it cites: §7.3 stated that source URLs stay readable under a password while §5.6 said
1947
- the writer withholds them, §4.5 called the recovered region exact while excluding two
1948
- bytes from it, §7.4 rejected a duplicate identifier that §4.5 resolves by tie-break,
1949
- and §2.1 described the HTML encoding prescan as mandatory and 1024 bytes wide when the
1950
- standard makes it optional and only encourages that bound. None of the four changes
1951
- what a writer emits or a reader accepts.
1952
-
1953
- The same pass added the two boundaries universal mode had left unstated: that it
1954
- recovers the region only where the declared charset is what decided the decoding, a
1955
- BOM, a user override and a transport-layer charset all outranking it — narrow in
1956
- practice, since the raw read comes first and no encoding applies to it (§2.1) — and that
1957
- the recovery payload's 32-bit length field caps the region below 2^32 bytes, with
1958
- engine string limits binding well before that (§5.5).
1959
-
1960
- A pass in September 2026, against core 1.5.120, read the text alone first and then
1961
- checked each open question against the writer. It corrected two statements about the
1962
- reference writer that the code contradicted: the retry loop never discards a
1963
- reservation, and the archive time is read once per build, not once per pass (§6.2).
1964
- It added what only the code could say: the PNG build steps that §6.1 had skipped, the
1965
- fields patched after the wrapper check and the size below which they are harmless
1966
- (§5.1), the chunks the PNG face copies (§3.1), the per-frame manifests and the root
1967
- directory's name (§7.1), the `page.pdf` header fields (§6.1), and the range-reading
1968
- failure path (§4.1).