single-file-core 1.6.1 → 1.6.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/core/filename.js +62 -0
- package/core/helper.js +9 -32
- package/modules/css-fonts-minifier.js +34 -13
- package/modules/template-formatter.js +8 -2
- package/package.json +10 -2
- package/processors/compression/compression-constants.js +28 -0
- package/processors/compression/compression.js +3 -1
- package/.github/workflows/publish.yml +0 -88
- package/.github/workflows/test.yml +0 -31
- package/.github/workflows/vendor.yml +0 -39
- package/css-tree-build/README.md +0 -39
- package/css-tree-build/build.js +0 -66
- package/css-tree-build/package-lock.json +0 -511
- package/css-tree-build/package.json +0 -13
- package/deno.lock +0 -19
- package/doc/assets/singlefile-archive-byte-map.svg +0 -276
- package/doc/assets/singlefile-archive-lenses.svg +0 -245
- package/doc/singlefile-archive.md +0 -1968
- package/eslint.config.mjs +0 -74
- package/test/capture/README.md +0 -74
- package/test/capture/canonical-link.js +0 -50
- package/test/capture/common.js +0 -73
- package/test/capture/deferred-content-options.js +0 -71
- package/test/capture/dom.js +0 -26
- package/test/capture/maff-metadata.js +0 -205
- package/test/capture/resource-cap.js +0 -79
- package/test/capture/script-uri-sanitization.js +0 -82
- package/test/fixtures/README.md +0 -18
- package/test/fixtures/classic-digest.html +0 -5
- package/test/fixtures/generate-fixtures.sh +0 -31
- package/test/fixtures/multi-page-dedup.zip.html +0 -70
- package/test/fixtures/multi-page.zip.html +0 -26
- package/test/fixtures/single-page.zip.html +0 -11
- package/test/fixtures/site/multi-page/alpha.html +0 -2
- package/test/fixtures/site/multi-page/beta.html +0 -2
- package/test/fixtures/site/multi-page/index.html +0 -2
- package/test/fixtures/site/multi-page/sub/gamma.html +0 -2
- package/test/fixtures/site/multi-page-dedup/index.html +0 -2
- package/test/fixtures/site/multi-page-dedup/logo.png +0 -0
- package/test/fixtures/site/multi-page-dedup/one.html +0 -2
- package/test/fixtures/site/multi-page-dedup/shared.css +0 -3
- package/test/fixtures/site/multi-page-dedup/two.html +0 -2
- package/test/run.js +0 -109
- package/test/sfz-harness/README.md +0 -87
- package/test/sfz-harness/adopted-stylesheets-hook.js +0 -240
- package/test/sfz-harness/byte-map.js +0 -137
- package/test/sfz-harness/charset-round-trip.js +0 -161
- package/test/sfz-harness/check-determinism.js +0 -57
- package/test/sfz-harness/common.js +0 -154
- package/test/sfz-harness/content-type-sniffing.js +0 -83
- package/test/sfz-harness/css-fonts-minifier.js +0 -235
- package/test/sfz-harness/css-property-filter.js +0 -113
- package/test/sfz-harness/dom-stub.js +0 -13
- package/test/sfz-harness/entry-compression.js +0 -77
- package/test/sfz-harness/filename-characters.js +0 -55
- package/test/sfz-harness/filename-max-length.js +0 -69
- package/test/sfz-harness/font-face-composite.js +0 -135
- package/test/sfz-harness/format-rules.js +0 -617
- package/test/sfz-harness/gen-e2e-page.js +0 -14
- package/test/sfz-harness/inlined-functions.js +0 -82
- package/test/sfz-harness/option-wiring.js +0 -60
- package/test/sfz-harness/pages-archive.js +0 -318
- package/test/sfz-harness/pages-router.js +0 -143
- package/test/sfz-harness/relocation-cost.js +0 -94
- package/test/sfz-harness/search-triggers.js +0 -35
- package/test/sfz-harness/smoke.js +0 -10
- package/test/sfz-harness/stored-trigger.js +0 -95
- package/test/sfz-harness/trigger-seeds.json +0 -120
- package/test/sfz-harness/zip64.js +0 -77
- package/zip-build/README.md +0 -56
- package/zip-build/lib/fflate-streams.js +0 -47
- package/zip-build/lib/fflate.js +0 -2692
- package/zip-build/lib/zip-vendor-worker.js +0 -3
- package/zip-build/lib/zip-vendor.js +0 -9
- package/zip-build/lib/zip.js +0 -25
- package/zip-build/package-lock.json +0 -665
- package/zip-build/package.json +0 -15
- package/zip-build/reserved-property-names.json +0 -10630
- package/zip-build/rollup.config.js +0 -61
|
@@ -1,1968 +0,0 @@
|
|
|
1
|
-
# The SingleFile archive format
|
|
2
|
-
|
|
3
|
-
**Status: draft.** This document specifies the SingleFile archive, the polyglot file
|
|
4
|
-
format produced by [SingleFile](https://github.com/gildas-lormeau/SingleFile) when it
|
|
5
|
-
saves a page as a ZIP archive. It is written against the reference implementation,
|
|
6
|
-
[single-file-core](https://github.com/gildas-lormeau/single-file-core) 1.5.120
|
|
7
|
-
(`processors/compression/`), and every byte-level statement has been verified on
|
|
8
|
-
generated specimen files.
|
|
9
|
-
|
|
10
|
-
The key words MUST, MUST NOT, SHOULD and MAY are to be interpreted as
|
|
11
|
-
described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119) when, and only when,
|
|
12
|
-
they appear in all capitals.
|
|
13
|
-
|
|
14
|
-
Three kinds of statement appear throughout:
|
|
15
|
-
|
|
16
|
-
- **Format requirements**, in RFC 2119 capitals. Their subject is a writer producing an
|
|
17
|
-
archive, or a *conforming reader of this format* — third-party software that reads
|
|
18
|
-
SingleFile archives (§7). It is never the stock HTML, ZIP, PDF or PNG readers the
|
|
19
|
-
document describes: those cannot be placed under an obligation by this document, and
|
|
20
|
-
a requirement that appears to do so is describing what a writer must produce so that
|
|
21
|
-
their existing behavior lands correctly (§4).
|
|
22
|
-
- **Reference behavior** — what `createArchive()` does where the format allows other
|
|
23
|
-
choices. Always marked as the reference writer's, and never a requirement.
|
|
24
|
-
- **Measurements** — observed behavior of specific third-party software, with the
|
|
25
|
-
version and the specimen it was measured on. §8 collects them; where one appears
|
|
26
|
-
inline it is the evidence for a claim, not a guarantee about software in general.
|
|
27
|
-
|
|
28
|
-
## 1. Introduction
|
|
29
|
-
|
|
30
|
-
A SingleFile archive is **one byte string that is simultaneously a valid document in
|
|
31
|
-
several formats**. Every archive is a valid ZIP file containing the saved page and its
|
|
32
|
-
resources. Depending on the options used to produce it, the same byte string is also:
|
|
33
|
-
|
|
34
|
-
- a valid **HTML page** that extracts and displays the archived page when opened in a
|
|
35
|
-
browser, with no external dependency;
|
|
36
|
-
- a valid **PNG image**, typically a screenshot of the page, though the format does not
|
|
37
|
-
require the image to depict it;
|
|
38
|
-
- a valid **PDF document**, typically a rendering of the page, though the format does
|
|
39
|
-
not require the document to depict it either.
|
|
40
|
-
|
|
41
|
-
Each format's reader accepts the file as a complete document of its own format and
|
|
42
|
-
silently ignores the bytes that belong to the other formats. The word is *accepts*,
|
|
43
|
-
not *conforms*: several faces lean on reader tolerances the target standards do not
|
|
44
|
-
promise (§1.1). The large payloads are stored once and
|
|
45
|
-
shared: the archive entries, the PDF document and the PNG pixel data are single
|
|
46
|
-
regions that several readers reach, not per-face copies. The polyglot works by
|
|
47
|
-
*partitioning* the file into regions and arranging each region so that every reader
|
|
48
|
-
either interprets it or skips it. What the optional features add is never a copy of
|
|
49
|
-
another face's payload: the text body (an optional plain-text copy of the page stored
|
|
50
|
-
in the HTML face for text tools and indexers, §4.6) repeats the page text, and the
|
|
51
|
-
image and PDF a writer supplies are separate documents, whether or not they depict the
|
|
52
|
-
archived page.
|
|
53
|
-
|
|
54
|
-
### 1.1 Design goals
|
|
55
|
-
|
|
56
|
-
The format exists to keep saved pages readable for as long as possible, with as
|
|
57
|
-
little software as possible. Each way of opening the file has a simpler fallback:
|
|
58
|
-
|
|
59
|
-
1. In a JavaScript-capable browser, the file opens and displays the page.
|
|
60
|
-
2. When extraction fails, the file displays an error message with recovery
|
|
61
|
-
instructions. Without JavaScript, it renders as a blank page: the document body
|
|
62
|
-
is hidden by construction, so the browser displays neither the page nor raw
|
|
63
|
-
archive bytes (§4.1).
|
|
64
|
-
3. Renamed to `.zip`, the file opens in a ZIP tool; the page and each resource are
|
|
65
|
-
ordinary entries. Two measured readers refuse a self-extracting variant even from
|
|
66
|
-
seekable input; the ranking below names them. A forward-only reader refuses it too,
|
|
67
|
-
and is a non-goal (§1.2).
|
|
68
|
-
4. Renamed to `.pdf` or `.png` (when those faces are present), the file opens in a PDF
|
|
69
|
-
viewer or an image viewer.
|
|
70
|
-
|
|
71
|
-
Three consequences shape everything below:
|
|
72
|
-
|
|
73
|
-
- **The HTML face depends on the ZIP face.** The HTML bootstrap extracts the page
|
|
74
|
-
*from the ZIP structure of its own file*; the ZIP face is not an export feature, it
|
|
75
|
-
is the storage layer the HTML face reads from.
|
|
76
|
-
- **Readers must need no cooperation.** Every face works with stock, unmodified
|
|
77
|
-
readers. The format relies on two kinds of reader behavior: rules the target
|
|
78
|
-
format actually defines (such as HTML's parsing and error-recovery rules, §4.1),
|
|
79
|
-
and *customary tolerances* — behaviors that are near-universal in practice but
|
|
80
|
-
that no standard promises, such as the backward scan ZIP readers use to find the
|
|
81
|
-
End Of Central Directory record (§4.2, §5.2), the PDF header scan and
|
|
82
|
-
trailing-data tolerance (§4.3), and PNG decoders' indifference to what an ancillary
|
|
83
|
-
`tEXt` chunk contains (§4.4). The compatibility appendix records the customary
|
|
84
|
-
tolerances' real-world support (§8).
|
|
85
|
-
- **The faces are not equally durable.** They rest on different amounts of unpromised
|
|
86
|
-
behavior, and "the file is valid in four formats" is not four guarantees. A decision
|
|
87
|
-
about what to rely on years from now should follow this order:
|
|
88
|
-
1. **The ZIP face** is the one to trust, and the only one the format calls storage.
|
|
89
|
-
A variant with no other face is an ordinary ZIP file. The self-extracting variants
|
|
90
|
-
add prepended and appended bytes, which cost two measured readers: `ditto`, which
|
|
91
|
-
requires a local header at offset 0, and `java.util.zip`, which rejects undeclared
|
|
92
|
-
trailing bytes. A writer can fix the second case by declaring those bytes as the
|
|
93
|
-
archive comment (§4.2).
|
|
94
|
-
2. **The HTML face** rests mostly on *normative* behavior: HTML's tokenizer states
|
|
95
|
-
and error recovery are specified, and the blank-page backstop of §4.1 is ordinary
|
|
96
|
-
CSS. It is exposed on two other counts: it depends on the ZIP face beneath it, and
|
|
97
|
-
a browser must run its script for the page to appear.
|
|
98
|
-
3. **The PDF and PNG faces are conveniences.** Both rest entirely on tolerances no
|
|
99
|
-
standard promises: the PDF header scan, where PDFium already enforces its
|
|
100
|
-
1024-byte window exactly with no margin, and PNG decoders' indifference to a
|
|
101
|
-
`tEXt` chunk holding bytes the format does not permit. They are worth having
|
|
102
|
-
because they cost nothing the other faces need. For preservation purposes they
|
|
103
|
-
are exports that happen to share the file; they are not archival copies.
|
|
104
|
-
|
|
105
|
-
### 1.2 Non-goals
|
|
106
|
-
|
|
107
|
-
- **Forward-only ZIP parsers.** Every archive with a face requires central-directory-driven
|
|
108
|
-
reading, since the entries are then preceded by non-ZIP bytes. The variant with no other
|
|
109
|
-
face is an ordinary ZIP file and streams from offset 0 like any other (§8.1, class A);
|
|
110
|
-
parsers that require that are out of scope for the rest (§7).
|
|
111
|
-
- **In-place modification by generic ZIP tools.** The face invariants are global:
|
|
112
|
-
the writer picks each hiding tag only after checking the exact bytes it must hide,
|
|
113
|
-
the recovery payload of universal mode contains a checksum of the ZIP region
|
|
114
|
-
without its comment-length field, and the PDF and PNG structures wrap the
|
|
115
|
-
archive (§5.4). A tool
|
|
116
|
-
that adds, removes or recompresses entries invalidates them, and most rewriters drop
|
|
117
|
-
the prepended and appended regions outright. A generically rewritten file keeps at
|
|
118
|
-
best its ZIP face. Editing an archive means producing a new one through the writer
|
|
119
|
-
rules (§6).
|
|
120
|
-
- **Multi-page archives.** The reference implementation can bundle several saved
|
|
121
|
-
pages into one archive behind a routing bootstrap (`multiPageArchive`). This
|
|
122
|
-
version of the document specifies single-page archives only; the multi-page
|
|
123
|
-
layout is out of scope, and so are the regions it adds to the prologue, which this
|
|
124
|
-
document does not describe.
|
|
125
|
-
- **Confidentiality outside the ZIP entries.** A password encrypts ZIP entry contents
|
|
126
|
-
only (AES). The PDF and PNG faces render the page content and are plaintext by
|
|
127
|
-
design; the writer withholds what it can without breaking a face, as described in
|
|
128
|
-
§5.6. The embedded PDF MAY itself be a PDF-encrypted document, since the format is
|
|
129
|
-
agnostic to the PDF's content, but the reference writer does not produce one.
|
|
130
|
-
|
|
131
|
-
### 1.3 Terminology
|
|
132
|
-
|
|
133
|
-
| Term | Meaning |
|
|
134
|
-
|---|---|
|
|
135
|
-
| **face** | One of the formats the file is valid in: HTML, ZIP, PNG, PDF. |
|
|
136
|
-
| **region** | A byte range with a single producer, named in §3. Regions are the units the rest of this document reasons about; a region can appear in several pieces — `html-prologue` resumes after the embedded PDF document in the PDF variants, and after the `tEXt "ZIP"` chunk header in the PNG ones, so with all four faces it comes in three. |
|
|
137
|
-
| **universal mode** | The variant whose HTML face can extract the archive from the *parsed page text*, the text and comment nodes the HTML parser produced, and therefore needs no access to its own raw bytes. Named "universal" because it works from any location, including the `file:` protocol. |
|
|
138
|
-
| **wrapper tag** | The HTML construct that hides a binary region from the HTML parser, `<!--`…`-->` by default (§5.1). |
|
|
139
|
-
| **appended data** | Bytes after the ZIP End Of Central Directory record. A reader tolerates them as far back as its EOCD scan reaches, and how far that is varies by an order of magnitude: 65557 bytes from the end of the file for Python `zipfile` (the 22-byte record plus the 65535-byte maximum comment length), but 16383 for libarchive and 32768 for perl `Archive::Zip` (§8.1). No reader's window is guaranteed, so a writer keeps its own narrower budget (§5.2). The format's one hard limit is the 65535-byte comment field, and it binds only a run the writer declares (§4.2). It may be left undeclared or declared as the archive comment; both forms are valid ZIP and readers MUST accept both (§4.2). The recovery payload can be computed before that choice is made because it stops two bytes short of the record, excluding its comment-length field (see *recovered range* below). |
|
|
140
|
-
| **ZIP region** | The contiguous byte range holding the archive proper: from the first local file header the ZIP writer emitted through the last byte of the End Of Central Directory record. It spans the `zip-entries`, `pdf-central-record` (when present) and `central-directory · eocd` blocks of §3, and in the HTML variants it is the content of the last wrapper, exactly so on the element rungs and preceded by the `sfz-data` identifier on the comment rung, which the extractor steps over. It does **not** include `pdf-local-header` or the PDF document, which sit earlier in the file. |
|
|
141
|
-
| **archive** | The *logical* ZIP file: the set of entries the central directory describes, wherever their bytes lie. This is distinct from the ZIP region above, which is a contiguous byte range. Every entry but one has its bytes inside the region; `page.pdf` is the deliberate exception, an entry of the archive whose local header and data sit before the region (§4.2). "Archive" in this document always means the logical file, "ZIP region" always the byte range, and the two differ only in the PDF-with-HTML variants. |
|
|
142
|
-
| **recovered range** | What the universal extractor reproduces (§4.5): the ZIP region minus its last two bytes, the comment-length field of the End Of Central Directory record. That field is the one part of the record whose value depends on what follows the region, so leaving it out is what lets a writer decide the appended-data form after the recovery payload is final (§4.2). The extractor supplies the two bytes itself, as zeroes — the recovered range carries no comment. |
|
|
143
|
-
| **reference writer** | `createArchive()` in single-file-core `processors/compression/compression.js`. |
|
|
144
|
-
| **bootstrap** | The inline script in the HTML face that locates, extracts and displays the archived page. |
|
|
145
|
-
| **`sfz` identifiers** | Three byte-level identifiers carrying the `sfz` prefix matter to this document: `data-sfz`, `<sfz-extra-data>` and `sfz-data`. A conforming file may hold others the format says nothing about — the reference bootstrap gives its own status messages `sfz`-prefixed ids, which no reader has any reason to look for. The prefix is inherited from SingleFileZ, the browser extension the format originated in (since merged into SingleFile), and is kept unchanged for compatibility with existing files. They are wire identifiers, not the format's name. Two of the three have a role in the format: `<sfz-extra-data>` is the element carrying the recovery payload, and `sfz-data` is the identifier the universal extractor addresses the ZIP region with — an `id` attribute on the wrapper element, or the first characters of the wrapper comment's data (§4.5). `data-sfz` is a marker the reference writer happens to put on the root element; this document mentions it only where it describes bytes those files contain. |
|
|
146
|
-
|
|
147
|
-
## 2. Variants: composing faces
|
|
148
|
-
|
|
149
|
-
Every SingleFile archive has the ZIP face. The other faces are enabled independently by writer
|
|
150
|
-
options, and compose. Each row of the table lists its complete option set; every row
|
|
151
|
-
also implies `compressContent`, the option that stores the page and its resources as
|
|
152
|
-
ZIP entries and so makes the output an archive at all:
|
|
153
|
-
|
|
154
|
-
| Writer options (core names) | HTML | PNG | PDF | Extension | Specimen |
|
|
155
|
-
|---|---|---|---|---|---|
|
|
156
|
-
| *(none)* | — | — | — | `.zip` | pure zip |
|
|
157
|
-
| `selfExtractingArchive` | ✔ | — | — | `.zip.html` | plain |
|
|
158
|
-
| `selfExtractingArchive`, `extractDataFromPage` | ✔ universal | — | — | `.u.zip.html` | universal |
|
|
159
|
-
| `selfExtractingArchive`, `extractDataFromPage`, `embeddedImage` | ✔ universal | ✔ | — | `.u.zip.html` | png |
|
|
160
|
-
| `selfExtractingArchive`, `extractDataFromPage`, `embeddedPdf` | ✔ universal | — | ✔ | `.u.zip.html` | pdf |
|
|
161
|
-
| `selfExtractingArchive`, `extractDataFromPage`, `embeddedImage`, `embeddedPdf` | ✔ universal | ✔ | ✔ | `.u.zip.html` | png-pdf |
|
|
162
|
-
| `embeddedPdf` | — | — | ✔ | `.zip` | zip-pdf |
|
|
163
|
-
| `embeddedImage` | — | ✔ | — | `.zip` | zip-png |
|
|
164
|
-
| `embeddedImage`, `embeddedPdf` | — | ✔ | ✔ | `.zip` | zip-png-pdf |
|
|
165
|
-
|
|
166
|
-
`extractDataFromPage` is orthogonal to the PNG and PDF faces. The PNG and PDF rows
|
|
167
|
-
include it because the clients producing those variants enable it by default, but
|
|
168
|
-
`embeddedImage` and `embeddedPdf` compose with a non-universal self-extracting file
|
|
169
|
-
just as well; the extension is then `.zip.html`. The one interaction: `page.pdf` lies
|
|
170
|
-
outside the ZIP region (§1.3), so the page-text extraction path does not recover it
|
|
171
|
-
with the rest. The extractor therefore filters that entry out unconditionally, on
|
|
172
|
-
every acquisition path including the ones that read raw bytes and could return it
|
|
173
|
-
(§4.5). In the last three rows there is no
|
|
174
|
-
HTML face, so the option does not apply. The *Specimen* column names the measured
|
|
175
|
-
reference files this document cites; §8 records how to regenerate them.
|
|
176
|
-
|
|
177
|
-
Other writer options shape the file without adding a face: `preventAppendedData`,
|
|
178
|
-
`declareAppendedData` and `maxAppendedDataLength` (§4.2, §5.2), `includeBOM` (§3.1),
|
|
179
|
-
`insertTextBody` (§4.6),
|
|
180
|
-
`password` (§5.6), `createRootDirectory` (§7.1), and the head-element switches
|
|
181
|
-
`insertCanonicalLink`, `insertMetaNoIndex` and `insertMetaCSP` (§3.1).
|
|
182
|
-
|
|
183
|
-
Notes on composition:
|
|
184
|
-
|
|
185
|
-
- **Universal mode requires the HTML face** (it is a property of the bootstrap) and is
|
|
186
|
-
independent of PNG/PDF. The command-line client enables it by default whenever it
|
|
187
|
-
produces a self-extracting file; the browser extension exposes it as the
|
|
188
|
-
"self-extracting ZIP (universal)" file format.
|
|
189
|
-
- **The extra-data payload is what separates the two self-extracting types in
|
|
190
|
-
practice.** A plain file has no way of its own to reach its bytes from a `file:`
|
|
191
|
-
URL: the bootstrap has no payload to rebuild the archive from and does not attempt
|
|
192
|
-
a self-read there (§4.1), so it goes to its error message. Opening one from disk
|
|
193
|
-
needs cooperating software that reads the file and hands it to the bootstrap — the
|
|
194
|
-
SingleFile extension does this once granted file access ("Allow access to file
|
|
195
|
-
URLs" in Chrome, Edge and Brave; "Disable Local File Restrictions" in Safari). Over
|
|
196
|
-
HTTP the plain file is self-sufficient, fetching its own URL. A
|
|
197
|
-
universal file instead carries the extra-data payload (§3.1, §5.5), so the bootstrap can
|
|
198
|
-
rebuild the archive from the parsed page text with no access to the raw bytes. Such a
|
|
199
|
-
file opens from disk in any browser, with no setting and no assistance.
|
|
200
|
-
- **All four faces at once is a supported combination**: the PDF document rides
|
|
201
|
-
inside the HTML head, which itself rides inside the PNG's first `tEXt` chunk
|
|
202
|
-
(the all-four-faces row of the byte map).
|
|
203
|
-
- **Faces without HTML** are plain polyglots with no self-extraction and no markup:
|
|
204
|
-
the ZIP data is appended after a raw PDF document (which is then plain prepended
|
|
205
|
-
data), or wrapped in PNG chunks — or both, the PNG chunk layout carrying the PDF
|
|
206
|
-
document as the data of its own `tEXt` chunk (keyword `PDF`) placed right after
|
|
207
|
-
`IHDR`, which keeps `%PDF-` within the header scan window (§4.3). In none of
|
|
208
|
-
these variants is `page.pdf` an archive entry.
|
|
209
|
-
|
|
210
|
-
### 2.1 The charset rule
|
|
211
|
-
|
|
212
|
-
The HTML face declares `<meta charset=utf-8>` when universal mode is off. When
|
|
213
|
-
universal mode is on it declares a single-byte charset instead — `windows-1252` in the
|
|
214
|
-
reference writer. The declaration MUST appear within the first 1024 bytes of the file
|
|
215
|
-
so the parser's encoding prescan finds it. That bound is the HTML standard's own
|
|
216
|
-
authoring rule. The prescan it serves is weaker than the rule suggests: the standard
|
|
217
|
-
makes it optional, and only *encourages* scanning the first 1024 bytes. Treat the
|
|
218
|
-
number as a ceiling to write under, never as a budget a parser promises to read. The
|
|
219
|
-
whole `<meta>` tag has to fit: one that straddles the boundary is not seen, and the
|
|
220
|
-
parser falls back to its default encoding. Meeting the declaration later, during
|
|
221
|
-
tokenization, does not rescue the file. The parser does not resume the prescan. It
|
|
222
|
-
re-navigates the document under the new encoding instead, and a writer must not rely
|
|
223
|
-
on that.
|
|
224
|
-
|
|
225
|
-
The declaration decides the decoding only when nothing outranks it. Three things do,
|
|
226
|
-
each returning an encoding with the standard's *certain* confidence, all of them ahead
|
|
227
|
-
of the prescan: a byte order mark, a user's explicit encoding override, and a charset
|
|
228
|
-
stated by the transport layer, which over HTTP means a `Content-Type` header carrying
|
|
229
|
-
its own `charset`. Any of the three replaces the declared charset, the parsed text is
|
|
230
|
-
then not what the writer encoded, and the region cannot be recovered from it. This is
|
|
231
|
-
the one precondition universal mode has that the file cannot satisfy from within
|
|
232
|
-
itself.
|
|
233
|
-
|
|
234
|
-
Where it bites is narrower than that makes it sound, because the parsed text is the
|
|
235
|
-
last rung, not the first. The bootstrap reads the file's raw bytes whenever it can
|
|
236
|
-
(§4.1), and raw bytes carry no encoding; universal extraction is the fallback for when
|
|
237
|
-
they are out of reach. Taking the three in turn:
|
|
238
|
-
|
|
239
|
-
- A **transport charset** exists only over HTTP, and over HTTP the raw read is what
|
|
240
|
-
runs — the bootstrap requests its own URL and takes the response as bytes, which no
|
|
241
|
-
`Content-Type` can reinterpret. It reaches universal extraction only in a double
|
|
242
|
-
failure: the response has to defeat the raw read, through a network or CORS failure
|
|
243
|
-
or a non-200 status, *and* state a charset of its own.
|
|
244
|
-
- A **BOM** is the writer's own doing. It is why universal and PNG variants never carry
|
|
245
|
-
one (§3.1): the reference writer emits a BOM for the plain variant only
|
|
246
|
-
(`includeBOM`), where nothing depends on the declared charset.
|
|
247
|
-
- A **user override** is the one no software can prevent, and the rarest.
|
|
248
|
-
|
|
249
|
-
On `file:` URLs the bootstrap goes straight to page-text extraction, since it attempts
|
|
250
|
-
no raw read there (§4.1) — but there is also no transport layer, so the first of the
|
|
251
|
-
three cannot arise on the very path that depends on the charset most.
|
|
252
|
-
|
|
253
|
-
The failure is safe rather than silent, which is why the precondition is worth stating
|
|
254
|
-
at all. Decoded under the wrong charset the reconstructed bytes are wrong, the payload
|
|
255
|
-
checksum does not match, and the extractor MUST fail to the error message (§4.5)
|
|
256
|
-
instead of displaying a corrupt page. A reader MAY tell the case apart from ordinary
|
|
257
|
-
corruption by comparing the encoding the document was actually decoded with —
|
|
258
|
-
`document.characterSet` in a browser — against the declared one, and say so in the
|
|
259
|
-
error message. Nothing requires it, and the MUST is unaffected either way.
|
|
260
|
-
|
|
261
|
-
Universal mode works in two parts, and the charset carries the first. The archive
|
|
262
|
-
bytes themselves are recovered *from the parsed page text*: the browser decoded
|
|
263
|
-
them as characters when it parsed the file, and the bootstrap re-encodes those
|
|
264
|
-
characters back into bytes. The requirement this places on the charset is
|
|
265
|
-
**injectivity**: under the encoding's index in the
|
|
266
|
-
[WHATWG Encoding Standard](https://encoding.spec.whatwg.org/) — the mapping every
|
|
267
|
-
browser implements — each of the 256 byte values MUST decode to a distinct code
|
|
268
|
-
point, and no byte may decode to U+FFFD. Any encoding with that property carries
|
|
269
|
-
arbitrary bytes through the parse, and 20 of the standard's encodings qualify (§8.4).
|
|
270
|
-
Multi-byte encodings, `utf-8` included, do not: invalid sequences collapse to U+FFFD
|
|
271
|
-
and the bytes cannot be recovered.
|
|
272
|
-
|
|
273
|
-
The reference writer uses `windows-1252`, for reasons beyond injectivity. It is the
|
|
274
|
-
best-supported single-byte encoding there is: the standard resolves the `iso-8859-1`
|
|
275
|
-
and `ascii` label families to it, and it is the fallback the HTML standard prescribes
|
|
276
|
-
for unlabelled content in most locales — so a file whose `<meta charset>` is stripped
|
|
277
|
-
or overridden by a server still tends to be decoded the way the extractor expects. It
|
|
278
|
-
also keeps the reverse table small, at 27 entries (§5.5).
|
|
279
|
-
|
|
280
|
-
The second part is the extra-data payload, which does **not**
|
|
281
|
-
contain the archive: it carries only what the round trip destroys or leaves
|
|
282
|
-
undetermined, namely a checksum, the recovered range's length, and the information
|
|
283
|
-
needed to restore newline bytes, which the parser normalizes.
|
|
284
|
-
The parser also replaces NUL bytes with U+FFFD; since no byte decodes to U+FFFD under
|
|
285
|
-
a qualifying encoding, the extractor maps U+FFFD back to NUL unambiguously and the
|
|
286
|
-
payload needs nothing for it (§5.5).
|
|
287
|
-
|
|
288
|
-
The declared charset governs the **whole document**, not only the regions the format
|
|
289
|
-
reasons about. Everything the parser reads is decoded with it, the bootstrap script
|
|
290
|
-
included, and the writer's own code is therefore subject to the same single-byte
|
|
291
|
-
decoding as the page it carries. In universal mode the bootstrap MUST contain no
|
|
292
|
-
character outside ASCII.
|
|
293
|
-
|
|
294
|
-
Unlike the `<title>` (§4.6), it cannot be rescued by
|
|
295
|
-
character references. A `<script>` element's content is script data, a tokenizer state
|
|
296
|
-
that does not resolve them: `☺` written there stays seven literal characters and
|
|
297
|
-
reaches the program as seven characters. The escape has to happen one level down, in
|
|
298
|
-
the JavaScript source — `\u263A` rather than `☺`, an escape the language resolves when
|
|
299
|
-
the script is compiled, not one the HTML parser resolves when the file is read. A
|
|
300
|
-
minifier will undo this if allowed to, since printing the shortest form is its default
|
|
301
|
-
and the shortest form of `\u263A` is the literal character. A writer that assembles
|
|
302
|
-
the bootstrap through a minifier MUST configure it to emit ASCII only.
|
|
303
|
-
|
|
304
|
-
The consequence of getting this wrong is worse than the mojibake a raw title produces,
|
|
305
|
-
and that is the reason for the MUST. A garbled title is visible; a garbled string
|
|
306
|
-
inside the extractor is not. A lookup table is the sharpest case. Emitted as literal
|
|
307
|
-
characters, a CP437 table is re-decoded as windows-1252 and grows from 256 entries to
|
|
308
|
-
508, shifting every lookup past the first 32 by 60 positions — and nothing about the
|
|
309
|
-
page looks wrong, because the damage is confined to names the table decodes. Under
|
|
310
|
-
§5.8 that can be a single entry, and one is enough when it is the entry the extractor
|
|
311
|
-
matches by name. The requirement is on the whole bootstrap rather than on any table
|
|
312
|
-
inside it, because a minifier does not know which strings are load-bearing.
|
|
313
|
-
|
|
314
|
-
### 2.2 File name conventions
|
|
315
|
-
|
|
316
|
-
The reference implementation names files by variant: `.zip` (no HTML face),
|
|
317
|
-
`.zip.html` (self-extracting), `.u.zip.html` (self-extracting, universal). These are
|
|
318
|
-
conventions for humans and pickers; **readers MUST NOT rely on the file name**. Every
|
|
319
|
-
face is discoverable from the bytes alone: PNG and PDF by their signatures, the ZIP
|
|
320
|
-
face by its End Of Central Directory record, and the HTML face by an `<html` start tag
|
|
321
|
-
occurring before the first local file header — inside the first `tEXt` chunk's data in
|
|
322
|
-
the PNG variants, where the markup begins after the chunk's keyword and its NUL
|
|
323
|
-
separator. Inside the
|
|
324
|
-
archive, the `index.html` and `manifest.json` entries mark it as a saved page; a
|
|
325
|
-
reader should identify it that way (§7.1). The self-extracting variants are told apart
|
|
326
|
-
the same way: only a universal file carries an `<sfz-extra-data>` element.
|
|
327
|
-
|
|
328
|
-
## 3. The byte map
|
|
329
|
-
|
|
330
|
-
Unless a row states otherwise, the layouts below are measured from specimen files
|
|
331
|
-
saved from `example.com` (the generation commands are in §8). The relocated row covers
|
|
332
|
-
two cases with one layout, `preventAppendedData` and a payload over the appended-data
|
|
333
|
-
budget (§5.2): the first is
|
|
334
|
-
measured on the relocated specimen, the second derived from the writer rules, because
|
|
335
|
-
such a payload requires an archive too large for a readable specimen. The figure below shows
|
|
336
|
-
the regions and their order; the glossary of §3.1 is the normative list, and it states
|
|
337
|
-
in text everything the figure conveys.
|
|
338
|
-
|
|
339
|
-

|
|
340
|
-
|
|
341
|
-
### 3.1 Region glossary
|
|
342
|
-
|
|
343
|
-
The names below are the block labels of the figure (where space is tight the figure
|
|
344
|
-
merges adjacent blocks into one label, such as `--></body></html>`). In this table,
|
|
345
|
-
*Producer* names
|
|
346
|
-
the syntax the bytes belong to (*extractor* marks the recovery machinery of
|
|
347
|
-
universal mode), and *Present* names the faces, variants or modes in which the
|
|
348
|
-
region exists. Rows are grouped: the HTML/ZIP core first, then the regions the PDF
|
|
349
|
-
face adds, then the regions the PNG face adds.
|
|
350
|
-
|
|
351
|
-
| Region | Producer | Present | Contents |
|
|
352
|
-
|---|---|---|---|
|
|
353
|
-
| `html-prologue` | HTML | HTML face | Doctype, the root element start tag, `<meta charset>`, an optional implementation-defined comment, title, optional head elements (canonical link, `robots` meta, viewport, Content-Security-Policy), minimal CSS, `<body hidden>`, wait/error messages, optional text body (§4.6). The leading comment, the title, the canonical link and the text body are withheld when a password is set (§5.6). In the plain variant an optional UTF-8 BOM MAY precede the doctype (`includeBOM`); universal and PNG variants never carry one, and the reference writer ignores the option there. In the PNG variants the region is split: everything through `<body hidden>` is the data of the `tEXt "PNG"` chunk, while the messages and the optional text body follow the `tEXt "ZIP"` chunk header; the doctype and the leading comment are dropped. |
|
|
354
|
-
| `bootstrap` | HTML | HTML face | One inline `<script>`: the embedded ZIP reader, the extractor, the display routine, and the content-acquisition logic (§4.1). In universal mode its bytes MUST be pure ASCII, since the declared charset decodes this region like any other and character references do not apply inside script data (§2.1). The wrapper start tag that opens the ZIP region follows it, directly or after a relocated `extra-data`. |
|
|
355
|
-
| `<!--` / `-->` | HTML | HTML face | The wrapper tag pair hiding a binary region from the HTML parser — comment tags by default, another pair when the hidden bytes defeat them — which `-->` is only the commonest way to do, the full test being `<!--`, `--!>`, a trailing `<!-` and, for the PNG payload, a leading `>` or `->` (§5.1). Drawn at each opening and closing position. The close tag is absent whenever the recovery payload is relocated (§5.2): under `preventAppendedData`, when the payload outgrows the appended-data budget, or on the `<plaintext>` wrapper which cannot close. No markup then follows the archive and the wrapper runs to end-of-file. That does not mean the file ends at the EOCD — the PNG face's tail still follows, inside the wrapper, where it parses as text (§5.1). |
|
|
356
|
-
| `zip-entries` | ZIP | always | The archive's local file headers and entry data, written by the ZIP writer. The central directory of an archive written by the reference writer lists `index.html` (the page) first, then `manifest.json` (a JSON description of the archive: original URL, title, save time, resource-to-URL map — informative; the page displays without it), then the resources; the *physical* order of the local headers inside the region is not guaranteed to match, and readers MUST NOT rely on either order — entries are addressed by name (§7.1). |
|
|
357
|
-
| `central-directory · eocd` | ZIP | always | The central-directory records followed by the End Of Central Directory record. All offsets are absolute file positions (§5.3). In the HTML+PDF variants the EOCD accounts for the injected `pdf-central-record` (how the writer achieves that is §6). |
|
|
358
|
-
| `extra-data` | extractor | universal | `<sfz-extra-data>` element holding the base64, deflate-compressed recovery payload (§5.5). It always sits outside the wrapper, so it parses as a real element the extractor can address. Normal placement: after the EOCD, between the wrapper close tag and the end tags. Relocated placement, used when the payload exceeds the appended-data budget (§5.2) or `preventAppendedData` is set: immediately before the wrapper start tag. In the relocated form the element is followed by space padding: its room is reserved before the archive is written, because the region precedes the ZIP data and resizing it would shift every central-directory offset (§6). Neither placement carries positional meaning — the extractor finds the ZIP region by identifier, not relative to this element (§4.5). |
|
|
359
|
-
| `</body></html>` | HTML | HTML face | The end tags closing the document after the wrapper close tag. Omitted whenever the recovery payload is relocated (§5.2), and in the PNG variants so the file can end with the PNG tail. |
|
|
360
|
-
| `pdf-local-header` | ZIP | PDF face with HTML | The hand-built local file header for `page.pdf` (STORE, checksum precomputed, language encoding flag left clear as on every other ASCII name — §5.8), written immediately before the PDF document so ZIP readers see an ordinary entry whose data is the PDF (§6). |
|
|
361
|
-
| `pdf-document` | PDF | PDF face | The raw PDF bytes. With the HTML face, wrapped together with `pdf-local-header` in a wrapper tag pair inside `html-prologue`, placed so `%PDF-` starts at offset 1024 or lower — the range PDF readers search for the header, which is what lets a PDF document start after other bytes at all (§4.3). Without the HTML face and without the PNG face, the file simply *starts* with the PDF document, as prepended data the ZIP face tolerates; `page.pdf` is then not an archive entry at all — no local header, no central record. |
|
|
362
|
-
| `pdf-central-record` | ZIP | PDF face with HTML | The central-directory record for `page.pdf`, injected *before* the writer's own central directory. The start of the central directory is the one place a record can be added without moving any offset the writer already committed, and it makes `page.pdf` the first entry ZIP tools list (§6). |
|
|
363
|
-
| `png-signature · IHDR` | PNG | PNG face | The 8-byte PNG signature and the `IHDR` chunk declaring the source image's dimensions — the first 33 bytes of the file. |
|
|
364
|
-
| `tEXt "PNG"` | PNG | PNG face with HTML | The 12 header bytes of the first `tEXt` chunk: the 4-byte big-endian length, the type, the keyword and its NUL separator. Its data is `html-prologue` (with the PDF face, the embedded PDF document rides inside it too), ending with the wrapper start tag. |
|
|
365
|
-
| `tEXt "PDF"` | PNG | PNG + PDF faces without HTML | The 12 header bytes (length, type, keyword, NUL separator) of a `tEXt` chunk whose data is the raw PDF document. Written only when the PNG and PDF faces combine without HTML — with the HTML face the PDF rides inside `tEXt "PNG"` instead — and placed right after `IHDR` so `%PDF-` stays within the header scan window (§4.3). |
|
|
366
|
-
| `pixel-data chunks` | PNG | PNG face | Every chunk of the source image between `IHDR` and `IEND`, ancillary chunks included, copied unmodified. The reference writer takes `IHDR` as the 25 bytes after the signature and `IEND` as the last 12 bytes of the source, so a source image with bytes after `IEND` is not supported. With the HTML face the chunks sit inside the wrapper so the HTML parser skips them. |
|
|
367
|
-
| `tEXt "ZIP"` | PNG | PNG face | The 12 header bytes (length, type, keyword, NUL separator) of the archive's own `tEXt` chunk — the second one when the HTML face or the PDF face put a chunk ahead of it, the only one otherwise. Its declared length covers everything from there up to but not including the trailing chunk CRC, as a PNG chunk length always does, so the PNG decoder skips the archive — and, with the HTML face, the bootstrap and the appended data — as the data of one chunk. With the HTML face, the wrapper opened at the end of `tEXt "PNG"` closes immediately after these bytes: its content is the first chunk's CRC, the pixel-data chunks and this chunk's own header, and the prologue resumes as markup directly after the close tag. |
|
|
368
|
-
| `crc · IEND` | PNG | PNG face | The `tEXt "ZIP"` chunk's CRC, computed once the archive bytes are final (§6), followed by the empty `IEND` chunk — the last bytes of the file (PNG requires `IEND` to end the stream, which is why the PNG variants drop the end tags). |
|
|
369
|
-
|
|
370
|
-
The reader-by-reader interpretation of these regions is §4; the mechanics that keep
|
|
371
|
-
them from colliding (wrapper-tag selection, checksums, offsets, the appended-data budget) are
|
|
372
|
-
§5.
|
|
373
|
-
|
|
374
|
-
## 4. Reader lenses
|
|
375
|
-
|
|
376
|
-
Each consumer of the file has a defined way of locating its own bytes and a defined
|
|
377
|
-
reason to ignore the rest. This section walks the same file through each reader. The
|
|
378
|
-
figure shows the all-four-faces variant of the byte map once per reader, fading the
|
|
379
|
-
regions that reader ignores; the subsections explain each row.
|
|
380
|
-
|
|
381
|
-

|
|
382
|
-
|
|
383
|
-
This section describes what stock readers do with the file. The convention stated at
|
|
384
|
-
the head of this document applies throughout it: a MUST about a face constrains the
|
|
385
|
-
bytes a writer produces, never the stock reader whose behavior the format cannot
|
|
386
|
-
change.
|
|
387
|
-
|
|
388
|
-
### 4.1 The browser
|
|
389
|
-
|
|
390
|
-
The HTML parser consumes the whole file as one document. Its encoding prescan finds
|
|
391
|
-
the `<meta charset>` declaration within the first 1024 bytes (§2.1) and the file is
|
|
392
|
-
decoded as a single text; every binary region therefore also exists as characters in
|
|
393
|
-
the parsed document, which is what universal mode exploits (§4.5). This holds only
|
|
394
|
-
while the declaration is what decides the decoding: a BOM, a user override or a
|
|
395
|
-
transport-layer charset outranks it, and universal extraction then fails its checksum
|
|
396
|
-
rather than recovering anything (§2.1). The acquisition order below keeps that off the
|
|
397
|
-
common path — the raw bytes are read in preference to the parsed text wherever they
|
|
398
|
-
can be, and no encoding applies to them.
|
|
399
|
-
|
|
400
|
-
The binary regions are kept out of the rendered page by the wrapper tags. The
|
|
401
|
-
default wrapper is an HTML comment, and the HTML standard defines exactly which
|
|
402
|
-
character sequences terminate one (`-->`, and the recovery form `--!>`); the writer
|
|
403
|
-
MUST select a wrapper only after checking the bytes it must hide against that
|
|
404
|
-
wrapper's patterns (the same test for every payload, with a shorter ladder for the
|
|
405
|
-
PDF and PNG payloads and one extra check for the PNG one, §5.1), so hiding relies on
|
|
406
|
-
normative parsing behavior. When no
|
|
407
|
-
wrapper fits a PDF or PNG payload, the face is dropped rather than emitted bare
|
|
408
|
-
(§5.1). Some binary content always sits *outside* a
|
|
409
|
-
wrapper: in the PNG variants, the signature, IHDR and chunk framing bytes that
|
|
410
|
-
precede the root element start tag decode to a short run of text that HTML error recovery
|
|
411
|
-
places in the (hidden) body. The backstop for all these cases is the prologue: it
|
|
412
|
-
declares `<body hidden>`, which only the bootstrap clears, and a stylesheet that
|
|
413
|
-
suppresses everything except the wait and error messages once the body is shown, so
|
|
414
|
-
the page comes up blank rather than showing raw bytes, with or without scripting.
|
|
415
|
-
|
|
416
|
-
The bootstrap script runs at parse time and proceeds in three stages:
|
|
417
|
-
|
|
418
|
-
1. **Acquire the archive bytes.** On `file:` URLs it goes straight to page-text
|
|
419
|
-
extraction (§4.5): whether a `file:` page may read its own bytes varies by
|
|
420
|
-
browser and configuration (§2), so the bootstrap uses the rung that depends on
|
|
421
|
-
neither. On other protocols it requests its own URL, aborting at the response
|
|
422
|
-
headers: when the server advertises `Accept-Ranges: bytes` it switches to HTTP
|
|
423
|
-
range reading, fetching only the central directory and the entries it needs (a
|
|
424
|
-
large archive displays without downloading the ZIP region in full); otherwise it
|
|
425
|
-
downloads the whole file. When the header probe fails it falls back to page-text
|
|
426
|
-
extraction; so does a failure of the full download itself, past the probe, which
|
|
427
|
-
is why the probe leaves the document in place. A failure inside range reading,
|
|
428
|
-
past the probe, is not caught the same way: it goes to the error message. Only
|
|
429
|
-
when every applicable rung fails does the error
|
|
430
|
-
message appear, with recovery instructions that differ by variant (§2).
|
|
431
|
-
2. **Extract.** The embedded ZIP reader reads the archive through the ZIP lens
|
|
432
|
-
(§4.2) and rebuilds the page: text entries are decoded, binary entries become
|
|
433
|
-
in-memory URLs, and references between entries are rewritten deepest-first.
|
|
434
|
-
3. **Display.** The rebuilt page replaces the bootstrap document. Around this stage
|
|
435
|
-
the saved page's `<noscript>` elements are neutralized — rewritten to inert
|
|
436
|
-
placeholders before parsing, restored afterwards — because the page was captured
|
|
437
|
-
with scripting available, so its noscript fallbacks must not activate in the
|
|
438
|
-
viewer.
|
|
439
|
-
|
|
440
|
-
The entry point is exposed as a run-once `globalThis.bootstrap(content)` function,
|
|
441
|
-
so cooperating software MAY hand the bootstrap bytes it acquired itself; `content` is
|
|
442
|
-
the file's bytes as a `Blob` or an array of byte values, or a reader object exposing
|
|
443
|
-
the embedded ZIP reader's `readUint8Array` interface, and the call returns a promise
|
|
444
|
-
that settles when the page has been displayed. This is how
|
|
445
|
-
the SingleFile extension assists a *plain* (non-universal) file on `file:`: granted
|
|
446
|
-
file-URL access, it reads the file and invokes the bootstrap. That is the recovery
|
|
447
|
-
path the plain variant's error message describes.
|
|
448
|
-
|
|
449
|
-
### 4.2 The ZIP reader
|
|
450
|
-
|
|
451
|
-
The ZIP face is read from the end. A reader locates the End Of Central Directory
|
|
452
|
-
record by scanning backward from end-of-file, and how far back it scans is the one
|
|
453
|
-
reader property the format cannot assume (§1.3, *appended data*). Everything the
|
|
454
|
-
writer emits after the record — wrapper close tag, extra-data, end tags, PNG tail —
|
|
455
|
-
fits the appended-data budget of §5.2, and the reference writer sizes that budget to
|
|
456
|
-
the narrowest scan measured in §8.1, so the record stays reachable for every reader
|
|
457
|
-
listed there. Accepting *undeclared* bytes
|
|
458
|
-
in that window is itself a customary tolerance (§1.1): the ZIP specification
|
|
459
|
-
documents the comment, not trailing junk. From the EOCD
|
|
460
|
-
the reader jumps to the central directory and reads only what it references;
|
|
461
|
-
central-directory-driven reading is a requirement of the format (§1.2, §7).
|
|
462
|
-
|
|
463
|
-
Two properties keep ordinary ZIP tools comfortable:
|
|
464
|
-
|
|
465
|
-
- **Offsets MUST be absolute file positions** (§5.3). The stored central-directory offset
|
|
466
|
-
equals the record's actual position, and each entry's local-header offset points at
|
|
467
|
-
a real local header — tools that cross-check offset arithmetic (rather than
|
|
468
|
-
tolerating a uniform shift from prepended data) accept the file as-is.
|
|
469
|
-
- **The non-ZIP regions are invisible to the reader.** Bytes before the first local
|
|
470
|
-
header and after the EOCD are simply never referenced. The one deliberate
|
|
471
|
-
exception: in the PDF-with-HTML variants the first central-directory record points
|
|
472
|
-
*back into the HTML head*, where the hand-built `page.pdf` local header and the PDF
|
|
473
|
-
document sit (`pdf-local-header`, §3.1) — an ordinary STORE entry that happens to
|
|
474
|
-
live inside the prepended region.
|
|
475
|
-
|
|
476
|
-
A listing shows `page.pdf` first (when the PDF face is present with HTML), then
|
|
477
|
-
`index.html`, `manifest.json` and the page's resources. That order and the two
|
|
478
|
-
conventions below describe the reference writer rather than constraining the format —
|
|
479
|
-
readers address entries by name (§7.1):
|
|
480
|
-
|
|
481
|
-
- Entries for resources fetched from a URL carry that URL in their *comment* field,
|
|
482
|
-
and so does each `index.html`, whose comment is the URL of the page or frame it
|
|
483
|
-
holds; a resource that came from a `data:` URL carries the literal marker `data:`
|
|
484
|
-
instead, and `manifest.json` and `page.pdf` have no comment. Comments are omitted entirely
|
|
485
|
-
from a password-protected archive, because the central directory is not encrypted
|
|
486
|
-
(§5.6).
|
|
487
|
-
- Entries whose content is already compressed (images, fonts, media, PDF) are STOREd
|
|
488
|
-
and the rest are deflated, a size optimization no reader depends on.
|
|
489
|
-
|
|
490
|
-
One rule here is a requirement. With a password, entry contents are AES-encrypted —
|
|
491
|
-
except `page.pdf`, which MUST stay unencrypted and MUST be STOREd, because its bytes
|
|
492
|
-
double as the PDF face (§5.6). The encryption is WinZip's AES scheme, the one ZIP
|
|
493
|
-
tools implement under compression method 99 with the `0x9901` extra field: AE-2,
|
|
494
|
-
AES-256, PBKDF2-HMAC-SHA1 key derivation and an HMAC-SHA1 authentication code. A
|
|
495
|
-
reader that already supports encrypted ZIP entries needs nothing specific to this
|
|
496
|
-
format, and the [WinZip AES specification](https://www.winzip.com/en/support/aes-encryption/)
|
|
497
|
-
is normative for it. A reader implementing the scheme from primitives rather than from
|
|
498
|
-
a ZIP library needs six parameters that specification supplies and this paragraph's
|
|
499
|
-
names do not: 1000 PBKDF2 iterations; a 16-byte salt at AES-256 strength, stored
|
|
500
|
-
before the data; a derived key of 32 + 32 + 2 bytes, read as encryption key,
|
|
501
|
-
authentication key, then a password verifier the reader MUST check before decrypting;
|
|
502
|
-
the HMAC-SHA1 code truncated to its first 10 bytes and stored after the data; and a
|
|
503
|
-
CTR counter that increments **little-endian**, starting at 1, which general-purpose
|
|
504
|
-
CTR interfaces do not do.
|
|
505
|
-
|
|
506
|
-
Appended data comes in two forms and both are valid ZIP, so readers MUST accept
|
|
507
|
-
both. **Raw** — the EOCD declares a zero-length comment and the trailing bytes are
|
|
508
|
-
simply outside the archive — is the default, because tools print a declared archive
|
|
509
|
-
comment on ordinary operations (§8.1), and in universal mode that comment is the
|
|
510
|
-
whole base64 recovery payload. **Declared** — the EOCD's comment length covers every
|
|
511
|
-
byte after the record, `declareAppendedData` in the reference writer — is the later
|
|
512
|
-
addition, and it is the only form some
|
|
513
|
-
readers accept at all: `java.util.zip`, and therefore Android and most JVM tooling,
|
|
514
|
-
rejects an archive with undeclared trailing bytes outright (§8.1). A writer SHOULD
|
|
515
|
-
offer both and default to raw.
|
|
516
|
-
|
|
517
|
-
The declared form carries a ceiling the raw form does not. The comment length is a
|
|
518
|
-
16-bit field, so a run longer than 65535 bytes cannot be declared at all. A writer
|
|
519
|
-
whose appended-data budget (§5.2) is raised past that ceiling MUST leave such a run
|
|
520
|
-
undeclared rather than write its length back modulo 65536, and readers that accept
|
|
521
|
-
only the declared form then reject the file with no diagnostic. The budget and the
|
|
522
|
-
ceiling are two separate limits, and a writer that exposes the first as an option
|
|
523
|
-
SHOULD say so where it documents it.
|
|
524
|
-
|
|
525
|
-
Neither form constrains the other faces, and universal mode supports both, because
|
|
526
|
-
the recovery payload describes the recovered range rather than the whole region: the
|
|
527
|
-
comment-length field is excluded (§1.3), so its value can be decided after the
|
|
528
|
-
payload is final. A writer that instead covered the field would have to solve a
|
|
529
|
-
circular dependency — the field's value depends on the size of the appended run,
|
|
530
|
-
which contains the payload, whose content depends on the field.
|
|
531
|
-
|
|
532
|
-
### 4.3 The PDF viewer
|
|
533
|
-
|
|
534
|
-
The PDF face relies on two customary reader behaviors — conventions PDF
|
|
535
|
-
implementations follow rather than guarantees of the PDF specification, so the
|
|
536
|
-
compatibility appendix records real-world support (§8):
|
|
537
|
-
|
|
538
|
-
- **The header scan.** Viewers accept a file whose `%PDF-` header starts at offset
|
|
539
|
-
1024 or lower — the bound is on the header's first byte, and 1024 itself passes —
|
|
540
|
-
and treat the header's position as byte 0 of the document: every
|
|
541
|
-
offset in the file (cross-reference entries, `startxref`) is interpreted relative
|
|
542
|
-
to it. The writer MUST place the header inside that window; the document's own
|
|
543
|
-
offsets then need no rewriting. Engines differ in how strictly they hold to the
|
|
544
|
-
1024-byte figure, and the strict ones are the common ones: PDFium — Chrome, Edge and
|
|
545
|
-
everything else Chromium-based — accepts a header starting at offset 1024 and
|
|
546
|
-
rejects one at 1025 outright, while poppler and macOS PDFKit impose no limit at all
|
|
547
|
-
(§8.1).
|
|
548
|
-
- **Tolerance of trailing data.** The archive continues after `%%EOF`, so the
|
|
549
|
-
document's tail is not the file's tail. Viewers cope by searching backward for the
|
|
550
|
-
trailer over a larger window or by reconstructing the cross-reference table from
|
|
551
|
-
the objects themselves.
|
|
552
|
-
|
|
553
|
-
With the HTML face, the PDF document sits inside the head, wrapped in its own
|
|
554
|
-
wrapper-tag pair chosen against the local header and the PDF together (§5.1) — a PDF
|
|
555
|
-
containing `-->` steps the ladder, and so can the header, whose CRC-32 and size fields
|
|
556
|
-
hold arbitrary bytes — and preceded by the `page.pdf` local header so the same bytes
|
|
557
|
-
are also a ZIP entry. That dual role is why the entry MUST be STOREd and MUST NOT be
|
|
558
|
-
encrypted: a viewer reads the entry's data region directly, and any transformation
|
|
559
|
-
of it would break the face. Without the HTML face the document needs no wrapper, and
|
|
560
|
-
where it sits depends on the PNG face: alone with the ZIP face it simply starts the
|
|
561
|
-
file, at offset 0, exercising only the trailing-data tolerance; with the PNG face it is
|
|
562
|
-
the data of a `tEXt "PDF"` chunk placed right after `IHDR` (§3.1), which puts `%PDF-`
|
|
563
|
-
at offset 45 exactly, inside the window but not at its start.
|
|
564
|
-
|
|
565
|
-
### 4.4 The PNG decoder
|
|
566
|
-
|
|
567
|
-
PNG offers no header scan: the standard requires the signature to be the first
|
|
568
|
-
8 bytes of the stream and `IEND` to be its last chunk. The PNG face therefore owns
|
|
569
|
-
both ends of the file — the HTML face gives up its doctype and leading comment
|
|
570
|
-
at the front, and the closing `</body></html>` at the back (§3.1).
|
|
571
|
-
|
|
572
|
-
After the signature, a decoder walks chunks — length, type, data, CRC — and skips
|
|
573
|
-
ancillary chunks it does not use. The face hides all foreign bytes inside two `tEXt`
|
|
574
|
-
chunks (ancillary by construction, their type starting lowercase):
|
|
575
|
-
|
|
576
|
-
- **`tEXt` with keyword `PNG`** carries the head of the HTML prologue, through
|
|
577
|
-
`<body hidden>` — and, in the all-faces variant, the embedded PDF document inside
|
|
578
|
-
it — ending with the wrapper start tag. The rest of the prologue, the messages and
|
|
579
|
-
the optional text body, comes later, inside the second chunk.
|
|
580
|
-
- **`tEXt` with keyword `ZIP`** declares a length that covers everything from its
|
|
581
|
-
keyword to the trailing chunk CRC: the rest of the HTML, the bootstrap, the whole
|
|
582
|
-
ZIP region and the appended data. The decoder hops over all of it as the data of
|
|
583
|
-
one chunk.
|
|
584
|
-
|
|
585
|
-
The `tEXt "ZIP"` chunk carries text the PNG standard does not strictly permit: a
|
|
586
|
-
`tEXt` text string is Latin-1 text, and its payload contains NUL bytes — 78 in the
|
|
587
|
-
`png` specimen, 101 in the `png-pdf` one. The first chunk is pure printable ASCII in
|
|
588
|
-
the plain `png` specimen, where the doctype and the provenance comment are suppressed
|
|
589
|
-
and the title is escaped to character references; only the `-pdf` variants put NULs
|
|
590
|
-
in it. Decoders skip
|
|
591
|
-
ancillary chunks without inspecting their text, so this passes everywhere tested
|
|
592
|
-
(§8.1). It exercises the PNG tolerance §1.1 lists at its limit: what decoders ignore
|
|
593
|
-
in a `tEXt` chunk is text PNG does not permit.
|
|
594
|
-
|
|
595
|
-
Both chunks MUST carry correct CRCs — decoders are entitled to verify them, and the
|
|
596
|
-
second chunk's CRC can only be computed once the archive bytes are final (§6). The
|
|
597
|
-
`pixel-data chunks` region is copied bit-identically from the source image, so the
|
|
598
|
-
decoded image is exactly that image.
|
|
599
|
-
|
|
600
|
-
### 4.5 The universal-mode extractor
|
|
601
|
-
|
|
602
|
-
The last reader is the format's own: the extraction path of universal mode, used
|
|
603
|
-
when the raw bytes are unreachable (§4.1). Its input is not the file but the *parsed
|
|
604
|
-
document* — the characters the HTML parser produced — and its output is the ZIP
|
|
605
|
-
region reconstructed byte for byte, with one deliberate exception: the two bytes of
|
|
606
|
-
the EOCD comment-length field, which the payload does not describe and the extractor
|
|
607
|
-
always writes as zero (step 2 below, and the row in §7.4).
|
|
608
|
-
|
|
609
|
-
It works in three steps:
|
|
610
|
-
|
|
611
|
-
1. **Locate.** The extractor finds the `<sfz-extra-data>` element for the payload, and
|
|
612
|
-
the ZIP region's node by its identifier `sfz-data` (§1.3): the element returned by
|
|
613
|
-
`getElementById`, or, when the wrapper is a comment, the first comment in the
|
|
614
|
-
document whose data starts with those characters. An element bearing the identifier
|
|
615
|
-
wins over a comment when both resolve; a reader that finds an id-bearing element
|
|
616
|
-
which is not one of §5.1's wrapper rungs SHOULD fall back to the comment, since the
|
|
617
|
-
`id` is then something else in the page; the reference extractor does not, and
|
|
618
|
-
takes whatever element bears the identifier. The two placements (§3.1) need no
|
|
619
|
-
telling apart, and neither the region's position in the tree nor its depth carries
|
|
620
|
-
meaning — a document that moved the node before extraction resolves the same way,
|
|
621
|
-
which matters because the reference extractor relocates `meta` and `style` elements
|
|
622
|
-
into the head before this step and one wrapper rung is a `style` element. With a
|
|
623
|
-
comment wrapper the identifier is part of the node's data, so the re-encoding in
|
|
624
|
-
step 3 starts after it.
|
|
625
|
-
2. **Decode the payload.** The element's text is base64 of a raw-deflate stream, and
|
|
626
|
-
the writer puts nothing else inside the element — the padding of the relocated
|
|
627
|
-
placement sits outside it (§6.1) — though a reader SHOULD ignore whitespace there
|
|
628
|
-
rather than reject the file. It
|
|
629
|
-
inflates to four fields: a checksum of the recovered range, its byte length, the
|
|
630
|
-
newline count, and the packed sequence of 2-bit codes recording each original
|
|
631
|
-
newline (LF, CR or CR LF). §5.5 gives their wire format — little-endian 32-bit
|
|
632
|
-
words, the codes packed 16 per word, least-significant pair first — which a reader
|
|
633
|
-
needs before it can read any of what follows.
|
|
634
|
-
Every one of the four describes the *recovered range*,
|
|
635
|
-
not the whole ZIP region: a newline formed by the two excluded bytes is neither
|
|
636
|
-
counted nor coded, and the checksum does not cover them. The declared length is the
|
|
637
|
-
**only** bound on the re-encoding: the extractor MUST stop there and append two zero
|
|
638
|
-
bytes to complete the EOCD record.
|
|
639
|
-
Those two bytes are its comment-length field, and only that field: every other byte
|
|
640
|
-
of the record is reconstructed from the parsed text like the rest of the region. The
|
|
641
|
-
field is excluded because its value depends on what follows the region, so a payload
|
|
642
|
-
covering it could not be computed until the appended data was final (§1.3); a
|
|
643
|
-
recovered archive therefore always declares a zero-length comment, whatever the
|
|
644
|
-
original declared. Never infer the bound from the node instead: the wrapper's close tag is
|
|
645
|
-
absent whenever appended data is prevented, and under `<plaintext>` the node always
|
|
646
|
-
runs to end of file. The final word is padded to 16 codes,
|
|
647
|
-
so codes beyond the declared newline count carry no meaning and a reader MUST ignore
|
|
648
|
-
them; within the count, the value 3 is unassigned and a reader MUST reject a payload
|
|
649
|
-
that uses it.
|
|
650
|
-
3. **Re-encode the characters.** Apply the inverse of the declared charset, exactly
|
|
651
|
-
as §5.5 defines it: a character whose code point is a byte value that decodes to
|
|
652
|
-
itself becomes that byte, every other code point goes through the reverse table,
|
|
653
|
-
and U+FFFD becomes NUL. Under windows-1252 this is nearly an identity — 229 of the
|
|
654
|
-
256 values map to themselves — but the shortcut "any code point ≤ 255 is that
|
|
655
|
-
byte" is **not** a valid substitute: under other qualifying charsets (§2.1) code
|
|
656
|
-
points below 256 can belong to a different byte, 75 of them under `macintosh`, and
|
|
657
|
-
the shortcut would silently corrupt the region. The reference extractor takes it,
|
|
658
|
-
and is correct only because it supports windows-1252 alone. The two things parsing destroyed
|
|
659
|
-
are restored from the payload: each parsed newline consumes the next 2-bit code to
|
|
660
|
-
reproduce the original byte sequence.
|
|
661
|
-
|
|
662
|
-
In the PDF-with-HTML variants the recovered region is a complete archive except for
|
|
663
|
-
one entry: its central directory still lists `page.pdf`, but that entry's local
|
|
664
|
-
header and data sit in the HTML head, outside the ZIP region (§1.3). The extractor
|
|
665
|
-
skips the entry instead of failing — the displayed page never references it, and
|
|
666
|
-
the PDF stays reachable through every raw-bytes path (§4.2).
|
|
667
|
-
|
|
668
|
-
The entry's offset is unusable *within the recovered region* — it is a true file
|
|
669
|
-
position like every other (§5.3), and a reader of the whole file follows it normally;
|
|
670
|
-
what fails is only the shifted arithmetic of the paragraph below. Its bytes are not out
|
|
671
|
-
of reach either: the local header
|
|
672
|
-
and the document sit in a wrapper in the head, so they are in the parsed page like any
|
|
673
|
-
other region, and a reader MAY recover them by the same round trip — find a local file
|
|
674
|
-
header naming `page.pdf` among the other parsed nodes, take the declared number of
|
|
675
|
-
bytes after it, and check them against the CRC-32 the central directory holds. The CRC
|
|
676
|
-
check is mandatory. The recovery payload's newline codes cover the ZIP region only, so
|
|
677
|
-
a newline in the PDF block has no code and its original bytes must be guessed: assume
|
|
678
|
-
LF, the byte the parser normalized *to*. A PDF routinely contains CR, so the guess
|
|
679
|
-
often fails, and nothing may be written unless the CRC-32 agrees. Take the length from
|
|
680
|
-
the central-directory record, which is the entry's authority for it; the hand-built
|
|
681
|
-
local header of §6.1 carries the same value, but a reader cannot tell that from a
|
|
682
|
-
length deferred to a data descriptor. A matching CRC-32 is a 32-bit non-cryptographic
|
|
683
|
-
check over a reconstruction that differs from the original in at most a few newline
|
|
684
|
-
bytes, so an undetected error is improbable but cannot be ruled out. A reader MUST NOT
|
|
685
|
-
present a reconstructed `page.pdf` as verified, and MUST NOT let one displace bytes
|
|
686
|
-
obtained from a raw-bytes read.
|
|
687
|
-
|
|
688
|
-
Recovering the entry is therefore optional; the reference extractor skips it, which
|
|
689
|
-
conforms. A reader that *presents the archive's contents* — a listing, an
|
|
690
|
-
extract-to-disk, an entry enumeration offered to a caller (§7.1) — MUST report
|
|
691
|
-
`page.pdf` as present and unretrieved rather than omit it, whether it skipped the
|
|
692
|
-
recovery or tried and failed the CRC check. Its listing is then the same as a raw-bytes
|
|
693
|
-
reader's, with only the bytes missing. A reader that has no such surface is outside the
|
|
694
|
-
rule: the display path of §4.1 rebuilds a page from the entries it needs, `page.pdf` is
|
|
695
|
-
referenced by nothing in that page, and the reference extractor accordingly filters the
|
|
696
|
-
entry out on every acquisition path, including the ones that read raw bytes and could
|
|
697
|
-
return it.
|
|
698
|
-
|
|
699
|
-
The extractor MUST verify the three checkable payload fields — byte length, newline
|
|
700
|
-
count and checksum — and fail to the error message on any mismatch. It MUST also fail
|
|
701
|
-
on the unassigned newline code of step 2 rather than decode it, so that a payload
|
|
702
|
-
written against a later revision of the format is named as unsupported instead of
|
|
703
|
-
silently reconstructing the wrong bytes.
|
|
704
|
-
|
|
705
|
-
The recovered region is a complete archive but **not an offset-self-contained one**.
|
|
706
|
-
Its offsets are still absolute positions in the original file (§5.3), so every
|
|
707
|
-
local-header offset in its central directory, and the central-directory offset in its
|
|
708
|
-
EOCD record, overshoot by exactly the region's start position in the file. A reader of
|
|
709
|
-
the recovered region MUST therefore apply a uniform negative shift of that amount, the
|
|
710
|
-
prepended-data compensation ZIP readers already implement: from the region's point of
|
|
711
|
-
view the missing bytes look like a prefix that was stripped. In the
|
|
712
|
-
specimen of §8.2 the shift is 122005 bytes: Info-ZIP reports `missing 122005 bytes in
|
|
713
|
-
zipfile`, adds `(attempting to process anyway)` and lists both entries, while readers
|
|
714
|
-
that compensate silently, such as Python's `zipfile`, show no diagnostic at all. The
|
|
715
|
-
shift also puts `page.pdf` out of the
|
|
716
|
-
offset-following path: its local header lies *before* the region, so its compensated
|
|
717
|
-
offset is negative and no reader can seek to it. That offset is the header's own
|
|
718
|
-
position — which §4.3 keeps inside the first 1024 bytes — minus the region's start,
|
|
719
|
-
which lies past the whole bootstrap, so it is negative for every archive and its
|
|
720
|
-
magnitude is essentially the region's start, so it moves with the size of the inlined
|
|
721
|
-
ZIP library and no particular value should be read into it. On
|
|
722
|
-
success the shifted bytes enter the normal extraction path (§4.2).
|
|
723
|
-
|
|
724
|
-
The shift is a file offset, and a universal-mode reader has no file. It does not need
|
|
725
|
-
one: the region carries the shift within itself, since its EOCD declares both the size
|
|
726
|
-
of the central directory and its absolute offset, while the directory's position inside
|
|
727
|
-
the region is known. With `region` the recovered bytes,
|
|
728
|
-
|
|
729
|
-
```
|
|
730
|
-
shift = eocd.centralDirectoryOffset - (eocdPosition - eocd.centralDirectorySize)
|
|
731
|
-
```
|
|
732
|
-
|
|
733
|
-
where `eocdPosition` is the EOCD record's own offset within `region`, found by scanning
|
|
734
|
-
backward for its signature the way any ZIP reader finds it. A reader arrives at the
|
|
735
|
-
same number as a ZIP library's prepended-data compensation, which derives it from the
|
|
736
|
-
record's position rather than from the buffer's end. Do not substitute
|
|
737
|
-
`region.length - 22` for `eocdPosition`. The two are in fact equal for every recovered
|
|
738
|
-
region, which always ends at the EOCD's last byte and declares a zero-length comment
|
|
739
|
-
(§1.3), but the habit fails the moment the same code is pointed at a file rather than a
|
|
740
|
-
recovered region: there a non-empty archive comment puts bytes after the record. Zip64
|
|
741
|
-
does not — its records precede the EOCD, which stays last.
|
|
742
|
-
|
|
743
|
-
Under zip64 (§5.7) both of those EOCD fields are the `0xFFFFFFFF` sentinel, and the
|
|
744
|
-
zip64 end of central directory record carries the real values. Take them from there,
|
|
745
|
-
using the same `eocdPosition` arithmetic against that record's own position — the
|
|
746
|
-
zip64 locator states an absolute offset in the original file, so it needs the shift
|
|
747
|
-
this formula produces and cannot be used to find it. A reader that instead uses the
|
|
748
|
-
sentinels arithmetically gets a shift in the billions, with no diagnostic.
|
|
749
|
-
|
|
750
|
-
Which leaves the record itself to be located without the offset that normally points at
|
|
751
|
-
it. Scan backward from the locator for the `PK\x06\x06` signature and confirm each
|
|
752
|
-
candidate against the record's own size field, the 8 bytes at `p + 4`, which by
|
|
753
|
-
definition excludes the leading 12:
|
|
754
|
-
|
|
755
|
-
```
|
|
756
|
-
p + 12 + size == locatorPosition
|
|
757
|
-
```
|
|
758
|
-
|
|
759
|
-
A well-formed archive puts the record immediately before the locator, where it is 56
|
|
760
|
-
bytes long if it carries no extensible data sector, so `locatorPosition - 56` is worth
|
|
761
|
-
testing before scanning at all. The confirmation matters on the archives that miss:
|
|
762
|
-
past the record the scan walks back through the central directory, whose file names and
|
|
763
|
-
extra fields are arbitrary bytes, and past that through entry data, and a four-byte
|
|
764
|
-
signature turns up in bytes nothing constrains. The test settles each candidate against
|
|
765
|
-
the record's own field, so it needs no offset it does not already have.
|
|
766
|
-
|
|
767
|
-
### 4.6 Text tools
|
|
768
|
-
|
|
769
|
-
The optional text body (`insertTextBody`) addresses one more consumer: software that
|
|
770
|
-
reads the file as plain text — `grep`, desktop search, indexers — and will never run
|
|
771
|
-
the bootstrap or unzip anything. It is a `<main hidden>` element at the end of the
|
|
772
|
-
visible prologue holding the page's text content, so the page stays searchable
|
|
773
|
-
without any extraction. It is searchable by everyone, so it is not written at all
|
|
774
|
-
when a password is set (§5.6): its text would be readable without the password.
|
|
775
|
-
|
|
776
|
-
The text body is always written in UTF-8, regardless of the declared charset. In
|
|
777
|
-
universal mode this cuts its audience in two: a charset-oblivious tool that reads
|
|
778
|
-
raw bytes — `grep`, plain-text search — sees intact UTF-8, while any consumer that
|
|
779
|
-
honors the declared `<meta charset>` decodes it as windows-1252 and garbles
|
|
780
|
-
non-ASCII text. That includes the HTML parser itself — harmless there, because the
|
|
781
|
-
region is hidden and replaced (§4.1) — but also HTML-aware indexers. In universal mode the text body
|
|
782
|
-
opens with the page title, for the same raw-byte
|
|
783
|
-
audience — it is the first *text* in the element, which is not necessarily the
|
|
784
|
-
element's first line: the reference writer's serialization puts a newline before it.
|
|
785
|
-
Outside universal mode the title is not repeated there, since the prologue's own
|
|
786
|
-
`<title>` is already readable as bytes.
|
|
787
|
-
|
|
788
|
-
The `<title>` element takes the opposite route, and so does every other piece of
|
|
789
|
-
prologue text the writer assembles itself. Character references are resolved against
|
|
790
|
-
Unicode independently of the declared encoding — in RCDATA, where the title's content
|
|
791
|
-
sits, in ordinary element text, and in attribute values alike — so the writer emits
|
|
792
|
-
every character outside printable ASCII, along with `&`, `<`, `>` and `"`, as a
|
|
793
|
-
numeric reference. Those bytes are therefore pure ASCII and the text survives the
|
|
794
|
-
single-byte declaration intact: a page titled 日本語 shows as 日本語 in the browser tab
|
|
795
|
-
and to any conforming parser. The reference writer passes the title, the canonical
|
|
796
|
-
link's `href` and the viewport value — attribute values, which is what the `"` is for —
|
|
797
|
-
through one shared escaper. Writers that emit such text raw MUST NOT do so in
|
|
798
|
-
universal mode, where the same bytes decode as mojibake.
|
|
799
|
-
|
|
800
|
-
The text body above is the deliberate exception, not an oversight: it is left as raw
|
|
801
|
-
UTF-8 because its audience reads bytes rather than parsed text. The bootstrap script
|
|
802
|
-
is the other region outside this rule, and it is outside for a harder reason — script
|
|
803
|
-
data does not resolve character references at all, so the escape must happen in the
|
|
804
|
-
JavaScript source instead (§2.1).
|
|
805
|
-
|
|
806
|
-
An implementation-defined comment (§3.1) is the third region outside the rule, and the
|
|
807
|
-
only one with no escape available at all. Comment data does not resolve character
|
|
808
|
-
references either, and unlike script data it has no second language of its own to
|
|
809
|
-
escape in: `é` written in a comment stays `é` in every reader, so the escaper
|
|
810
|
-
does not restore the character, it replaces one unreadable form with another. The
|
|
811
|
-
reference writer therefore leaves the comment's characters alone and serializes it as
|
|
812
|
-
UTF-8 with the rest of the prologue, deliberately. What it does rewrite is the
|
|
813
|
-
comment's own terminators: a space goes before the `>` of `-->` and `--!>`, before a
|
|
814
|
-
leading `>` or `->`, and after a trailing `<!-`, so the comment cannot close itself
|
|
815
|
-
(§5.1). Its audience is whoever opens the raw file in an
|
|
816
|
-
editor or runs a text tool over it, and those decode the bytes as UTF-8 whatever the
|
|
817
|
-
declaration says; only a browser's raw view, which honors the declared charset, shows
|
|
818
|
-
the text as mojibake in universal mode. The bytes are harmless to extraction, since
|
|
819
|
-
the recovery payload covers the ZIP region alone, far past the prologue. A reader MUST
|
|
820
|
-
NOT rely on decoding this comment through the declared charset, and a writer that
|
|
821
|
-
wants it readable everywhere restricts it to printable ASCII. The page's own copy of
|
|
822
|
-
the comment, inside `index.html`, is UTF-8 in a UTF-8 document and is the one the
|
|
823
|
-
displayed page and the infobar carry.
|
|
824
|
-
|
|
825
|
-
## 5. Cross-cutting mechanics
|
|
826
|
-
|
|
827
|
-
Section 3 named the regions and §4 read them one reader at a time. What remains are
|
|
828
|
-
the rules that span readers: how a payload is hidden, how much room is left at the
|
|
829
|
-
end of the file, which numbers are offsets into what, what each checksum covers, how
|
|
830
|
-
the character round trip is inverted, what a password protects, and what changes when
|
|
831
|
-
the archive is large enough to need zip64.
|
|
832
|
-
|
|
833
|
-
### 5.1 Wrapper-tag selection
|
|
834
|
-
|
|
835
|
-
Binary payloads inside the HTML face are hidden by a wrapper tag pair. What the format
|
|
836
|
-
requires of a wrapper is that the HTML parser not treat its content as markup, so the
|
|
837
|
-
payload survives parsing as text, and that the payload not contain the construct's
|
|
838
|
-
terminator. Any construct with those properties works; the reference writer picks from
|
|
839
|
-
this ladder, in order:
|
|
840
|
-
|
|
841
|
-
| Order | Wrapper | Parser treatment of the content | Terminated by |
|
|
842
|
-
|---|---|---|---|
|
|
843
|
-
| 1 | `<!--` … `-->` | comment | `-->`, or the recovery form `--!>` |
|
|
844
|
-
| 2 | `<script type=sfz-data>` | script data, not executed (the type is not a JavaScript MIME type) | `</script` followed by whitespace, `/` or `>` |
|
|
845
|
-
| 3 | `<style type=sfz-data>` | raw text, no style sheet built (the type is not a CSS MIME type) | `</style` + delimiter |
|
|
846
|
-
| 4 | `<noframes>` | raw text | `</noframes` + delimiter |
|
|
847
|
-
| 5 | `<noembed>` | raw text | `</noembed` + delimiter |
|
|
848
|
-
| 6 | `<iframe>` | raw text | `</iframe` + delimiter |
|
|
849
|
-
| 7 | `<xmp>` | raw text | `</xmp` + delimiter |
|
|
850
|
-
| 8 | `<svg><![CDATA[` … `]]></svg>` | CDATA section | `]]>` |
|
|
851
|
-
| 9 | `<plaintext>` | everything to end of file | nothing — the element cannot be closed |
|
|
852
|
-
|
|
853
|
-
The terminators are written lower case above, but HTML matches end tag names ASCII
|
|
854
|
-
case-insensitively: `</XMP>` and `</Script ` close their elements just as `</xmp>` and
|
|
855
|
-
`</script>` do. A writer's test MUST be case-insensitive, on the start patterns as well
|
|
856
|
-
as the end ones. A stored, uncompressed resource is the realistic source of an
|
|
857
|
-
upper-case one.
|
|
858
|
-
|
|
859
|
-
Every rung hides its content unconditionally, which is why `<noscript>` is not one. It
|
|
860
|
-
has the right terminator, but it is the one construct whose content is raw text only
|
|
861
|
-
while scripting is enabled and markup when it is not, so on a page opened without
|
|
862
|
-
scripting the archive bytes would reach the tree builder as tags. The rungs below it
|
|
863
|
-
hide the same payloads at no extra cost, so there is nothing to weigh against that.
|
|
864
|
-
|
|
865
|
-
The order under the comment is not arbitrary. Every rung hides its content from an HTML
|
|
866
|
-
parser, but text extractors differ, and the ZIP region is large enough that the
|
|
867
|
-
difference is a user-visible one. Measured on macOS: Spotlight's HTML importer indexes
|
|
868
|
-
the content of `<noframes>`, `<noembed>`, `<iframe>`, `<xmp>` and `<plaintext>`, and
|
|
869
|
-
`textutil` reads the last two of those, `<xmp>` and `<plaintext>`, while the comment
|
|
870
|
-
and the `script` and `style` rungs are dropped by both. Those two therefore sit
|
|
871
|
-
directly under the comment, so that an escalating writer keeps the archive out of the
|
|
872
|
-
reader's local search index
|
|
873
|
-
for as long as the payload allows. Both are inert at those types: the script is not
|
|
874
|
-
executed and no style sheet is built.
|
|
875
|
-
|
|
876
|
-
That ordering is the one place the format optimizes against measured third-party
|
|
877
|
-
behavior rather than against a rule, and unlike the tolerances of §1.1 nothing depends
|
|
878
|
-
on the measurement holding. An extractor that starts reading `<script type=sfz-data>`,
|
|
879
|
-
or stops reading `<xmp>`, changes only which archives end up in a local search index;
|
|
880
|
-
every rung still hides its content from the HTML parser, and a writer whose ladder is
|
|
881
|
-
ordered differently produces files that are just as correct.
|
|
882
|
-
|
|
883
|
-
The CDATA rung sits where it does for the same reason, and it is the only rung whose
|
|
884
|
-
placement understates it. A CDATA section is a CDATA section only in foreign content,
|
|
885
|
-
which is what the `<svg>` element is there for — in HTML content `<![CDATA[` is a
|
|
886
|
-
bogus comment, and the payload would be markup. Given the `<svg>`, the construct is the
|
|
887
|
-
strongest on the ladder: `]]>` is the whole of its terminator, and it is a sequence real
|
|
888
|
-
payloads carry far less often than `-->` or `</script>`. That matters most for the one
|
|
889
|
-
thing the ladder cannot otherwise avoid — an archive nested inside another as a face
|
|
890
|
-
carries the terminator of every rung it climbed, so each rung is spent once and only
|
|
891
|
-
once (§5.1, ladder depth). A writer MUST place the identifier on the `<svg>` element and
|
|
892
|
-
not on the markup declaration, which takes no attributes: `<svg id=sfz-data><![CDATA[`.
|
|
893
|
-
|
|
894
|
-
Two properties of the CDATA section state are worth stating because a writer is tempted
|
|
895
|
-
to guard against both and needs neither. Sections do not nest, so a `<![CDATA[` inside
|
|
896
|
-
the payload is text like any other — the start-pattern test on this rung is the same
|
|
897
|
-
conservatism the raw-text rungs get, not a necessity. And trailing brackets are safe: a
|
|
898
|
-
payload ending `]]` against the writer's `]]>` produces `]]]]>`, and the tokenizer's
|
|
899
|
-
CDATA section end state emits the payload's own two brackets before closing, so the
|
|
900
|
-
recovered bytes are exact.
|
|
901
|
-
|
|
902
|
-
Verified in Blink, Gecko and WebKit, and against html5lib: a universal-mode archive on
|
|
903
|
-
this rung recovers from the parsed document byte for byte, with the checksum of §4.5
|
|
904
|
-
matching, indistinguishably from the same archive on the comment rung. The same probe
|
|
905
|
-
covered a payload holding every byte value, every rung's patterns, and the near-misses
|
|
906
|
-
`]]x>`, `] ]>`, `]>` and `]]`, in both the prologue position and mid-document.
|
|
907
|
-
|
|
908
|
-
Two things about the ladder *are* required. Whatever order a writer gives the eight
|
|
909
|
-
closable rungs, it MUST apply the selection test below to every rung it considers, and
|
|
910
|
-
MUST keep `<plaintext>` available as the rung of last resort: §6.2's termination
|
|
911
|
-
argument needs one rung no payload can defeat.
|
|
912
|
-
|
|
913
|
-
The wrapper of the ZIP region also carries the identifier the extractor addresses it
|
|
914
|
-
with (§4.5): an element rung takes it as an `id` attribute — `<script type=sfz-data
|
|
915
|
-
id=sfz-data>`, `<noframes id=sfz-data>` — and the comment rung as the first characters of
|
|
916
|
-
its data, `<!--sfz-data`. The wrappers hiding the PDF and PNG faces MUST NOT carry it:
|
|
917
|
-
those payloads are found by byte structure, and a second node bearing the identifier
|
|
918
|
-
would shadow the archive. For the same reason no comment ahead of the wrapper may
|
|
919
|
-
begin with those characters, the implementation-defined comment of §3.1 included,
|
|
920
|
-
since the lookup takes the first that does.
|
|
921
|
-
|
|
922
|
-
The reference writer walks the ladder from the top and takes the first rung the payload
|
|
923
|
-
does not defeat. The test it applies is the format's, and is the same for every payload;
|
|
924
|
-
what differs is how far the ladder goes:
|
|
925
|
-
|
|
926
|
-
- **The ZIP region** rejects a rung whose *end* pattern the payload contains, and also
|
|
927
|
-
a rung whose *start* pattern it contains. A rung's start pattern is the tag's opening
|
|
928
|
-
delimiter and name, without attributes: `<!--` for the comment, then `<script`,
|
|
929
|
-
`<style`, `<noframes`, `<noembed`, `<iframe`, `<xmp` for the elements, and
|
|
930
|
-
`<![CDATA[` for the CDATA rung.
|
|
931
|
-
|
|
932
|
-
`<plaintext>` is exempt from **both** tests. It has no terminator to occur and no
|
|
933
|
-
tokenizer states to escape into — a `<plaintext` inside a `<plaintext>` is inert
|
|
934
|
-
text like everything else — so no payload can defeat it. §6.2's termination argument
|
|
935
|
-
rests on that exemption.
|
|
936
|
-
|
|
937
|
-
The other eight are all tested, and a writer MUST test all eight rather than the one
|
|
938
|
-
that needs it. The `<script>` rung needs it
|
|
939
|
-
to be correct at all, because script data has escape states no other rung has: `<!--`
|
|
940
|
-
in script data enters *script data escaped*, and a `<script` after that enters *script
|
|
941
|
-
data double escaped*, where `</script>` does **not** close the element. So a payload
|
|
942
|
-
can hold `<!--` and then `<script`, contain no `</script` anywhere, pass the end test —
|
|
943
|
-
and the wrapper then swallows its own end tag, the extra-data element and the rest of
|
|
944
|
-
the document. On the other seven the start test is genuine conservatism: a nested `<!--`
|
|
945
|
-
is a parse error inside a comment but does not close it, and the raw-text rungs hold a
|
|
946
|
-
flat run of characters with no states at all, while CDATA sections do not nest. The
|
|
947
|
-
rule is uniform deliberately: the
|
|
948
|
-
exemption would save one pattern match per rung on bytes already in memory, at the
|
|
949
|
-
cost of a special case an implementer has to remember correctly about the single rung
|
|
950
|
-
where forgetting it destroys the document.
|
|
951
|
-
- **The PDF and PNG payloads** apply the same two tests, for the same reason — a face
|
|
952
|
-
that took the `<script>` rung on a payload holding `<!--` and `<script` would swallow
|
|
953
|
-
the rest of the document, title, bootstrap and extra-data element included — but the
|
|
954
|
-
`<plaintext>` rung is excluded from their ladder: those payloads sit in the middle
|
|
955
|
-
of the file, so a wrapper that can never close is not an option. When no rung fits,
|
|
956
|
-
the writer MUST omit the face and emit the archive without it. It MUST NOT write the
|
|
957
|
-
payload bare. Bare renders acceptably — the blank-page backstop of §4.1 keeps it
|
|
958
|
-
invisible — but the payload's markup joins the document, and a payload that is itself
|
|
959
|
-
a SingleFile archive then contributes an `sfz-data` node ahead of the file's own. A
|
|
960
|
-
reader looking for one node finds two, takes the first, and returns an archive that
|
|
961
|
-
passes every check it has (§7.4). A face is a convenience; the archive is not.
|
|
962
|
-
|
|
963
|
-
The comment rung has one restriction more than a terminator. HTML forbids comment text
|
|
964
|
-
that *starts* with `>` or `->`, and the tokenizer enforces it: it closes the comment
|
|
965
|
-
right there, spilling the payload into the parser. What starts the comment differs by
|
|
966
|
-
payload — the ZIP region begins with the identifier, the PDF face with a local file
|
|
967
|
-
header or `%PDF-` — but the PNG face begins with the CRC of the chunk carrying the
|
|
968
|
-
start tag, four bytes that are only settled once the tag is chosen, and one in 256 of
|
|
969
|
-
them is `>`. A writer using a comment there MUST compute that checksum and leave the
|
|
970
|
-
comment rung when it opens with `>` or `->`. It MUST leave it by *resuming the rung
|
|
971
|
-
search* below it, not by taking the rung that follows: the checksum says only that the
|
|
972
|
-
comment is unusable, and which rung is usable remains the payload's to say. The test
|
|
973
|
-
itself cannot cascade — only the comment rung carries the restriction, and every rung
|
|
974
|
-
below it is an element — but the payload's terminators still apply, and a payload
|
|
975
|
-
holding `</script>` sends a writer that stepped rather than searched onto the one rung
|
|
976
|
-
it is guaranteed to close. HTML also forbids comment text ending
|
|
977
|
-
with `<!-`, which the terminator check covers by testing that pattern anchored at the
|
|
978
|
-
payload's end.
|
|
979
|
-
|
|
980
|
-
Choosing the last rung has consequences that reach the rest of the file: because
|
|
981
|
-
`<plaintext>` cannot be closed, selecting it sets `preventAppendedData` — no *markup*
|
|
982
|
-
may follow the ZIP region, which forces the relocated placement of the extra-data
|
|
983
|
-
element (§5.2) and drops the closing `</body></html>`.
|
|
984
|
-
|
|
985
|
-
That constraint is about markup, not about the last byte of the file, so the PNG face
|
|
986
|
-
composes with this rung: the `tEXt` chunk's checksum and the `IEND` chunk still follow
|
|
987
|
-
the region, as the PNG face requires, and `<plaintext>` reads them as the text they are.
|
|
988
|
-
Verified on a build forced onto this rung with a screenshot embedded: the file ends
|
|
989
|
-
`49 45 4e 44 ae 42 60 82`, decodes as a PNG, and its archive extracts from the parsed
|
|
990
|
-
page. The termination argument of §6.2 therefore holds for the faced variants too.
|
|
991
|
-
|
|
992
|
-
The ladder has a depth, and nesting reaches it. An archive used as the PDF or PNG
|
|
993
|
-
payload of another one carries the terminator of every rung its own faces climbed
|
|
994
|
-
through, and `-->`, `</script>` and `</style>` besides, which every prologue emits.
|
|
995
|
-
Each level of nesting therefore burns exactly one rung, and the eight a face may use
|
|
996
|
-
run out at the sixth: the reference writer selects `<!--`, then `<noframes>`,
|
|
997
|
-
`<noembed>`, `<iframe>`, `<xmp>`, the CDATA rung, and then has nothing left. No ladder
|
|
998
|
-
of fixed length avoids this; adding a rung moves the limit by one level, which is the
|
|
999
|
-
limit of what the CDATA rung buys here — its value is that real payloads rarely hold
|
|
1000
|
-
`]]>`, not that it makes nesting unbounded. That is why the paragraph
|
|
1001
|
-
above states a MUST rather than a quality-of-implementation preference — exhaustion is
|
|
1002
|
-
reachable by construction, not only by a payload built to provoke it.
|
|
1003
|
-
|
|
1004
|
-
The selection test above — both patterns, on every rung the writer considers — MUST
|
|
1005
|
-
be applied to the payload's bytes in their final form, for every payload the writer
|
|
1006
|
-
hides and in every variant that hides one. *Final* is the whole of the requirement:
|
|
1007
|
-
bytes the writer has yet to settle have not been tested. Every rejection restarts the build (§6):
|
|
1008
|
-
the wrapper choice changes the bytes preceding the archive, so the archive must be
|
|
1009
|
-
rewritten at its new position.
|
|
1010
|
-
|
|
1011
|
-
Two fields are patched after that test, and each needs one of its own. The EOCD
|
|
1012
|
-
comment-length field sits at the end of the ZIP region and is patched under the
|
|
1013
|
-
declared form (§6.1, step 11); the writer tests the bytes around it again with the
|
|
1014
|
-
final value in place and keeps the raw form when that value would complete a pattern,
|
|
1015
|
-
since the raw form is always valid. The `tEXt "ZIP"` length field sits inside the pixel-data wrapper,
|
|
1016
|
-
with the fixed `tEXt` type and `ZIP` keyword after it, and is written last (step 12).
|
|
1017
|
-
The header is tested with the rest of the payload, the length as zeros, which cannot
|
|
1018
|
-
join a pattern; the real length is big-endian, so a pattern byte in it would have to be
|
|
1019
|
-
the most significant byte of the chunk's size, and the smallest byte any pattern
|
|
1020
|
-
contains, `-` at 0x2D, puts that size at 0x2D000000 bytes, about 755 MB. The writer
|
|
1021
|
-
refuses to build a self-extracting PNG variant whose chunk reaches that size rather than
|
|
1022
|
-
re-check the field.
|
|
1023
|
-
|
|
1024
|
-
### 5.2 The appended-data budget
|
|
1025
|
-
|
|
1026
|
-
The run the writer emits after the EOCD record has two limits, and only one of them
|
|
1027
|
-
comes from the format. A run *declared* as the archive comment MUST fit in 65535
|
|
1028
|
-
bytes, the largest value a comment-length field can hold (§4.2). A run left *raw* has
|
|
1029
|
-
no format limit at all: the bytes are outside the archive, and nothing in ZIP bounds
|
|
1030
|
-
them. What bounds both in practice is the reader. Locating the EOCD record means
|
|
1031
|
-
scanning backward from end-of-file, and the searches measured in §8.1 stop at 16383
|
|
1032
|
-
bytes for libarchive, 32768 for perl `Archive::Zip` and 65557 for Python `zipfile`, so
|
|
1033
|
-
a run sized to the comment ceiling is already invisible to the narrowest of them. A
|
|
1034
|
-
writer therefore keeps a *budget*, sized to the readers it means to satisfy rather
|
|
1035
|
-
than to the format. The appended run is:
|
|
1036
|
-
|
|
1037
|
-
```
|
|
1038
|
-
wrapper close tag + extra-data element + end tags + (PNG face: 4-byte chunk CRC + 12-byte IEND)
|
|
1039
|
-
```
|
|
1040
|
-
|
|
1041
|
-
and the writer compares its total against that budget before committing to it. The
|
|
1042
|
-
EOCD record's own 22 bytes sit inside a reader's window as well, which is what turns a
|
|
1043
|
-
65535-byte run into the 65557 bytes of §1.3 and the reference writer's budget into
|
|
1044
|
-
libarchive's 16383.
|
|
1045
|
-
|
|
1046
|
-
The reference writer exposes the budget as `maxAppendedDataLength` and defaults it to
|
|
1047
|
-
16361 bytes: libarchive's window less the 22 bytes of the record, which is the largest
|
|
1048
|
-
run behind which every reader of §8.1 still finds the record. A writer MAY choose
|
|
1049
|
-
another value. Raising it above 65535 leaves the run undeclarable: it is emitted, and
|
|
1050
|
-
it is still valid ZIP, but no comment length can cover it, and §4.2 says what that
|
|
1051
|
-
costs.
|
|
1052
|
-
|
|
1053
|
-
Only the extra-data element can outgrow the budget: it carries one 2-bit code per
|
|
1054
|
-
newline sequence in the recovered range — the ZIP region without its comment-length
|
|
1055
|
-
field (§4.5), CR LF counting once, for two bytes (§5.5) — so it
|
|
1056
|
-
grows with the archive. Newline bytes
|
|
1057
|
-
occur at their natural density in compressed and STOREd binary data — about two in
|
|
1058
|
-
every 256 bytes — and the codes are compressed and base64-encoded, which measures at
|
|
1059
|
-
one byte of element per 650 bytes of archive at scale (§8). The default budget is
|
|
1060
|
-
therefore exhausted at an archive of roughly 10 MB, and the 65535-byte ceiling at
|
|
1061
|
-
roughly 40 MB, so the relocated placement is uncommon in practice. That ratio is the large-archive limit and must not be used to size a
|
|
1062
|
-
particular file: deflate's overhead is a fixed cost spread over a growing payload, so
|
|
1063
|
-
small archives are far less efficient. Measured on exact byte counts, a 6099-byte region
|
|
1064
|
-
needs 69 bytes of element — a ratio of 88 — and a 74057-byte region needs 189, a ratio
|
|
1065
|
-
of 392; §8.3's series then runs 475, 609, 650 and 646 as the archive grows from 86 KB to
|
|
1066
|
-
4.2 MB, so the ratio approaches the headline figure from below and levels off rather
|
|
1067
|
-
than climbing past it. A writer sizes its reservation from the payload it actually
|
|
1068
|
-
produced (§6.2), never from this figure. When the payload does not fit, or when
|
|
1069
|
-
`preventAppendedData` is set, the writer switches to the **relocated placement**: the
|
|
1070
|
-
element moves in front of the wrapper start tag, ahead of the archive (§3.1). Room
|
|
1071
|
-
for it MUST be reserved before the ZIP region is written, because inserting bytes
|
|
1072
|
-
ahead of the archive would shift every offset the ZIP writer has already committed;
|
|
1073
|
-
the reservation is padded with spaces and the real payload is written into it once
|
|
1074
|
-
its final size is known (§6). Relocation is final for the build, and a relocated
|
|
1075
|
-
archive carries no appended run at all: the writer emits neither the wrapper's
|
|
1076
|
-
terminator nor the end tags, so outside the PNG face, whose tail still follows (§5.1),
|
|
1077
|
-
the file ends at the EOCD record like a plain ZIP file and the readers that reject
|
|
1078
|
-
trailing bytes open it (§8.1). The parser closes the open
|
|
1079
|
-
comment or element at end of file, and `</body></html>` are implied, so the page
|
|
1080
|
-
renders the same.
|
|
1081
|
-
|
|
1082
|
-
Relocation moves the element rather than copying it, but it is not a move at constant
|
|
1083
|
-
size, and wherever there is an element to move it costs bytes. The appended placement
|
|
1084
|
-
emits the wrapper terminator, the element and the end tags, the element plus 17; the
|
|
1085
|
-
relocated placement emits none of those and reserves room ahead of the archive instead,
|
|
1086
|
-
`Math.ceil(length * 1.01) + 32` bytes in the reference writer, where *length* is the
|
|
1087
|
-
element with its tags. The net is that reservation less the element and less the 17
|
|
1088
|
-
bytes, so about one percent of the element plus fifteen: what relocation costs is the
|
|
1089
|
-
margin, not a second copy. Measured on elements from 61 to 17577 bytes the formula holds
|
|
1090
|
-
to within a few bytes, the residual being the element itself changing length between the
|
|
1091
|
-
two passes, since the reservation lengthens the prologue and moves every
|
|
1092
|
-
central-directory offset with it. The wrapper rung sets the constant: fifteen bytes
|
|
1093
|
-
behind a comment, nine behind `</script>` or `]]></svg>`, six behind `</plaintext>`.
|
|
1094
|
-
With extraction disabled there is no element and nothing is reserved, so suppressing the
|
|
1095
|
-
appended run drops those 17 bytes and nothing else.
|
|
1096
|
-
|
|
1097
|
-
The two cases a writer meets differ by an order of magnitude, and the budget is what
|
|
1098
|
-
separates them. A relocation forced by `preventAppendedData` acts on whatever element
|
|
1099
|
-
exists, which on a small archive is small: 16 bytes on a 2848-byte ZIP region, 35 bytes
|
|
1100
|
-
on a 1.3 MB one. A relocation the budget triggers cannot be cheap, because it happens
|
|
1101
|
-
only once the element no longer fits: at the default 16361 that means an element past
|
|
1102
|
-
16344 bytes, and 185 bytes measured on a 12.7 MB region is near the least it can cost.
|
|
1103
|
-
It keeps rising from there, since a relocated element sits in the prologue and no comment
|
|
1104
|
-
ceiling bounds it — at the ratio above, a 40 MB archive carries roughly 57 KB of element
|
|
1105
|
-
and costs roughly 590 bytes. A writer sizing a file should compute the cost from the
|
|
1106
|
-
element it produced rather than quote any of these figures.
|
|
1107
|
-
|
|
1108
|
-
### 5.3 Offset bookkeeping
|
|
1109
|
-
|
|
1110
|
-
Three coordinate systems coexist in one file, and the format's job is to keep each
|
|
1111
|
-
self-consistent:
|
|
1112
|
-
|
|
1113
|
-
- **ZIP offsets are absolute file positions.** The writer is told the size of
|
|
1114
|
-
everything already emitted before the first local header, so the central directory
|
|
1115
|
-
offset in the EOCD and every local-header offset in the central directory are true
|
|
1116
|
-
file positions (§4.2), so a reader of the *whole file* never needs prepended-data
|
|
1117
|
-
compensation — the repair by which a reader recomputes offsets that disagree with
|
|
1118
|
-
the file size. A reader of the recovered ZIP region alone does need it (§4.5).
|
|
1119
|
-
|
|
1120
|
-
The alternative, offsets relative to the start of the region, is not a compatibility
|
|
1121
|
-
problem in itself: a reader that compensates arrives at the same entries, and 7-Zip
|
|
1122
|
-
opens such a file when told the type. What absolute offsets buy is the step before
|
|
1123
|
-
that. The file is a valid archive read as it stands, so it survives format
|
|
1124
|
-
auto-detection — 7-Zip reports a base of 0 and a physical size covering the whole
|
|
1125
|
-
file — and the compensation is confined to the one path that cannot avoid it,
|
|
1126
|
-
universal-mode recovery. Nothing in the format depends on the choice; a writer using
|
|
1127
|
-
the other form produces files this document's readers still open.
|
|
1128
|
-
- **PDF offsets are header-relative.** The document's own cross-reference offsets are
|
|
1129
|
-
interpreted from the `%PDF-` header, so embedding it needs no rewriting; the writer
|
|
1130
|
-
only MUST keep the header inside the scan window (§4.3).
|
|
1131
|
-
- **PNG has no offsets, only lengths.** Each chunk declares its data length. The
|
|
1132
|
-
`tEXt "ZIP"` chunk's length covers the whole archive, and the appended data too in
|
|
1133
|
-
the variants that have it, so it can only be written once the file's final size is
|
|
1134
|
-
known, and the writer patches it in place at the end (§6). A `tEXt` chunk precedes it
|
|
1135
|
-
whenever the HTML face or the PDF face is present — carrying the prologue or the PDF
|
|
1136
|
-
document respectively — and under the PNG face alone it is the only one. Appended
|
|
1137
|
-
data follows it only under the HTML face.
|
|
1138
|
-
|
|
1139
|
-
The injected `page.pdf` central record exploits a fourth, deliberate discrepancy. It
|
|
1140
|
-
is written directly to the output stream, bypassing the ZIP writer's own byte
|
|
1141
|
-
counter, at exactly the position where the central directory is about to start.
|
|
1142
|
-
The writer's counter is therefore left *behind* the true stream position by exactly
|
|
1143
|
-
the record's length, so the central-directory offset it stores lands on the injected
|
|
1144
|
-
record rather than after it: the stored offset needs no correction and the record
|
|
1145
|
-
becomes the first entry of the directory. The accounting does need correcting: after
|
|
1146
|
-
the archive is closed the writer increments the entry counts and adds the record's
|
|
1147
|
-
length to the directory size (§6).
|
|
1148
|
-
|
|
1149
|
-
### 5.4 Checksum inventory
|
|
1150
|
-
|
|
1151
|
-
Four independent integrity mechanisms cover overlapping byte ranges. All three CRC-32
|
|
1152
|
-
variants are the standard ZIP and PNG CRC-32: reflected polynomial `0xEDB88320`,
|
|
1153
|
-
initial value `0xFFFFFFFF`, final complement, processing each byte
|
|
1154
|
-
least-significant-bit first — the function `zlib.crc32` and its equivalents compute.
|
|
1155
|
-
One table therefore serves all three, but they cover different ranges and live in
|
|
1156
|
-
different structures:
|
|
1157
|
-
|
|
1158
|
-
| Checksum | Covers | Stored in |
|
|
1159
|
-
|---|---|---|
|
|
1160
|
-
| ZIP entry CRC-32 | one entry's *uncompressed* content | local file header and central-directory record of that entry, including the hand-built `page.pdf` records. Zero for AES-encrypted entries, whose integrity comes from their authentication code instead |
|
|
1161
|
-
| PNG chunk CRC-32 | one chunk's type and data bytes | the 4 bytes following each chunk's data. For `tEXt "ZIP"` this spans the whole ZIP region and the appended data |
|
|
1162
|
-
| Universal payload CRC-32 | the recovered range as the extractor re-encodes it — the ZIP region without its comment-length field (§1.3) | the recovery payload, with the range's length and newline count (§4.5) |
|
|
1163
|
-
| AES authentication code | one encrypted entry's stored bytes | that entry's data, when a password is set |
|
|
1164
|
-
|
|
1165
|
-
The PDF face contributes none: PDF has no whole-file checksum, so the document can sit
|
|
1166
|
-
inside a larger file unchanged.
|
|
1167
|
-
|
|
1168
|
-
Two of these — the `tEXt "ZIP"` chunk CRC and the universal payload — can only be
|
|
1169
|
-
computed when the file is otherwise final, which fixes the last steps of the writer's
|
|
1170
|
-
order (§6).
|
|
1171
|
-
|
|
1172
|
-
### 5.5 The character round trip
|
|
1173
|
-
|
|
1174
|
-
Universal mode recovers the ZIP region from characters rather than bytes (§2.1). The
|
|
1175
|
-
inverse mapping the extractor applies is, for the declared charset:
|
|
1176
|
-
|
|
1177
|
-
1. **A code point equal to a byte value that decodes to itself → that byte.** Under
|
|
1178
|
-
windows-1252 this covers 229 of the 256 values.
|
|
1179
|
-
2. **Every other code point → a fixed reverse table.** The table is the inverse of the
|
|
1180
|
-
encoding's index in the WHATWG standard, restricted to the byte values it does not
|
|
1181
|
-
map to themselves: 27 entries for windows-1252 — the printable characters it places
|
|
1182
|
-
in the 0x80–0x9F range (typographic quotes, dashes, the euro sign and so on). That
|
|
1183
|
-
count is this rule's table alone; an implementation that folds rule 3 into the same
|
|
1184
|
-
lookup, as the reference extractor does, has 28.
|
|
1185
|
-
Deriving this table is mechanical, so a reader supports any qualifying charset the
|
|
1186
|
-
same way — but it MUST be derived from the WHATWG index and not from the platform's
|
|
1187
|
-
codec of the same name, which is usually not the same mapping. The WHATWG index
|
|
1188
|
-
assigns every byte a code point; most platform codecs leave five positions of
|
|
1189
|
-
windows-1252 undefined:
|
|
1190
|
-
|
|
1191
|
-
| Byte | 0x81 | 0x8D | 0x8F | 0x90 | 0x9D |
|
|
1192
|
-
|---|---|---|---|---|---|
|
|
1193
|
-
| WHATWG | U+0081 | U+008D | U+008F | U+0090 | U+009D |
|
|
1194
|
-
| Python `cp1252`, Java `windows-1252` | undefined | undefined | undefined | undefined | undefined |
|
|
1195
|
-
|
|
1196
|
-
Those five bytes occur in ordinary compressed data, so a strict platform decode
|
|
1197
|
-
raises on essentially every archive. Configuring the decoder to replace what it
|
|
1198
|
-
cannot map is worse: it emits U+FFFD, which rule 3 below turns into NUL, corrupting
|
|
1199
|
-
one byte per occurrence. Measured over ten specimen archives, 42 to 1468 bytes per
|
|
1200
|
-
file would be lost this way. The payload checksum catches it.
|
|
1201
|
-
3. **U+FFFD → 0x00.** No byte decodes to U+FFFD under a qualifying encoding (§2.1), so
|
|
1202
|
-
the replacement character can only have come from a NUL byte. This holds because
|
|
1203
|
-
the payload is inside a wrapper: in every tokenizer state the ladder of §5.1
|
|
1204
|
-
produces — comment, raw text, script data, CDATA section, plaintext — the parser
|
|
1205
|
-
replaces NUL with U+FFFD.
|
|
1206
|
-
4. **Newlines from the payload.** The parser normalizes CR and CR LF to LF, so the
|
|
1207
|
-
original byte sequence is unrecoverable from the text alone; each newline consumes
|
|
1208
|
-
the next 2-bit code (0 = LF, 1 = CR, 2 = CR LF).
|
|
1209
|
-
|
|
1210
|
-
The payload itself is a sequence of little-endian 32-bit words — checksum, recovered
|
|
1211
|
-
range length, newline count, then the codes packed 16 per word, least-significant pair
|
|
1212
|
-
first — raw-deflated and base64-encoded with the standard alphabet and padding.
|
|
1213
|
-
|
|
1214
|
-
Those word widths cap what the payload can describe. A writer MUST NOT use universal
|
|
1215
|
-
mode for a ZIP region of 2^32 bytes or more, since the length field cannot express it.
|
|
1216
|
-
The cap is not enforced by the wire format itself: a writer that ignores it stores the
|
|
1217
|
-
length modulo 2^32 and produces a file that looks well-formed, and the mismatch
|
|
1218
|
-
surfaces only when a reader verifies the field (§4.5). The reference writer is in that
|
|
1219
|
-
position — it assigns the length into a `Uint32Array`, where the truncation is silent
|
|
1220
|
-
— and reaches the cap in no saved page. This bound and zip64 (§5.7) are separate
|
|
1221
|
-
things: zip64 is reachable at any archive size through the 65535-entry trigger and
|
|
1222
|
-
stays compatible with universal mode, and it is only a region large enough to need
|
|
1223
|
-
zip64's 64-bit *offsets* that runs past what the payload can describe.
|
|
1224
|
-
|
|
1225
|
-
An engine limit binds long before the format's. The extractor holds the region as one
|
|
1226
|
-
JavaScript string, and the maximum string length is engine-specific: V8 caps it at
|
|
1227
|
-
2^29 − 24 characters, 536870888, measured on V8 15.0.245. A universal-mode archive
|
|
1228
|
-
whose ZIP region approaches half a gigabyte is therefore already unreadable in Chrome,
|
|
1229
|
-
Edge and Node, whatever the payload declares. Other engines set the limit elsewhere.
|
|
1230
|
-
The practical ceiling on universal mode is this one, not the 4 GiB above.
|
|
1231
|
-
|
|
1232
|
-
### 5.6 Password scope
|
|
1233
|
-
|
|
1234
|
-
A password encrypts the *contents* of ZIP entries with AES, and nothing else. A reader
|
|
1235
|
-
gets no protection beyond that. Four consequences follow:
|
|
1236
|
-
|
|
1237
|
-
- **`page.pdf` is never encrypted** and never compressed: its bytes double as the PDF
|
|
1238
|
-
face, which a viewer reads directly from the entry's data region (§4.3).
|
|
1239
|
-
- **The PNG and PDF faces stay in the clear.** They render the page, and a viewer
|
|
1240
|
-
reads their bytes directly (§4.3, §4.4), so they cannot be encrypted without
|
|
1241
|
-
destroying the face. A password on an archive that also has one of them protects the
|
|
1242
|
-
archived resources, not the page's visible content.
|
|
1243
|
-
- **Entry metadata is never encrypted.** Names, uncompressed sizes and dates remain
|
|
1244
|
-
readable in the central directory, so the resource list of an encrypted archive is
|
|
1245
|
-
public. This is standard ZIP behavior, not a property of this format; §7 restates it.
|
|
1246
|
-
- **What the writer withholds instead.** Five things are not forced into the clear by
|
|
1247
|
-
the format, and so are withheld when a password is set. Three of them state a URL:
|
|
1248
|
-
the entry comments, which publish every resource's source URL (§4.2), and two
|
|
1249
|
-
prologue fields carrying the address the page was saved from — the provenance
|
|
1250
|
-
comment an implementation may write there, and the canonical `<link>` among the head
|
|
1251
|
-
elements (§3.1). The other two are the `<title>` element's text, leaving an empty
|
|
1252
|
-
`<title></title>` in the prologue, and the optional text body, which repeats the
|
|
1253
|
-
whole page text outside the archive (§4.6). Nothing is lost by leaving any of them
|
|
1254
|
-
out: `manifest.json` holds the page URL, the title and the resource-URL map, and it
|
|
1255
|
-
is an encrypted entry like the rest. Unlike the PNG and PDF faces, none of the five
|
|
1256
|
-
is load-bearing for a reader, so a writer that emits them in a password-protected
|
|
1257
|
-
archive publishes what the password is meant to cover for no gain.
|
|
1258
|
-
|
|
1259
|
-
Encrypted entries are stamped AE-2, so their CRC-32 field is zero (§5.4). `page.pdf`
|
|
1260
|
-
stays unencrypted, so in a password-protected archive its checksum is the only one a
|
|
1261
|
-
ZIP tool can verify.
|
|
1262
|
-
|
|
1263
|
-
### 5.7 zip64
|
|
1264
|
-
|
|
1265
|
-
The archive uses the zip64 end of central directory structures whenever the ordinary
|
|
1266
|
-
records cannot express it: a central directory starting beyond 4 GiB (the prefix counts
|
|
1267
|
-
toward the offset, §5.3), a directory 4 GiB or longer, or 65535 entries or more. A
|
|
1268
|
-
single entry of 4 GiB or more also produces zip64 extra fields, in that entry's local
|
|
1269
|
-
and central headers, without any zip64 end of central directory record. The reference
|
|
1270
|
-
writer never requests zip64 explicitly, so it appears only when reached, and given how
|
|
1271
|
-
large that is, effectively never in a saved page.
|
|
1272
|
-
|
|
1273
|
-
When it is reached, the EOCD record carries the sentinel values `0xFFFF` and
|
|
1274
|
-
`0xFFFFFFFF`, preceded by a zip64 end of central directory record and its locator.
|
|
1275
|
-
The sentinels are not selective: the writer saturates the entry counts, the directory
|
|
1276
|
-
size and the directory offset together once zip64 is emitted, whichever one of them
|
|
1277
|
-
overflowed. The `page.pdf` record injection then applies its accounting to the zip64
|
|
1278
|
-
record instead — entry counts and directory size there, and
|
|
1279
|
-
the locator's pointer moved by the record's length — while leaving each saturated
|
|
1280
|
-
field at its sentinel. A writer MUST NOT let the injection push a 16-bit or 32-bit
|
|
1281
|
-
field to its sentinel value without emitting the corresponding zip64 record: a count
|
|
1282
|
-
of `0xFFFF` sends readers looking for a zip64 record that does not exist.
|
|
1283
|
-
|
|
1284
|
-
`test/sfz-harness/zip64.js` covers this: the sentinels stay, the counts and the
|
|
1285
|
-
directory size land in the zip64 record, its directory offset points at the injected
|
|
1286
|
-
record, `page.pdf` is the first record in the directory, and a reader lists every entry.
|
|
1287
|
-
|
|
1288
|
-
zip64 does not conflict with universal mode. Its commonest trigger, 65535 entries or
|
|
1289
|
-
more, is reached at any archive size, and §4.5 gives the offset arithmetic for a
|
|
1290
|
-
recovered region whose EOCD fields are sentinels. What universal mode cannot carry is
|
|
1291
|
-
a ZIP region of 2^32 bytes or more, which the recovery payload's 32-bit length field
|
|
1292
|
-
cannot express (§5.5) — a size bound, not a zip64 one.
|
|
1293
|
-
|
|
1294
|
-
### 5.8 Entry name encoding
|
|
1295
|
-
|
|
1296
|
-
Entry names in this format are arbitrary Unicode, and how a name is decoded is an
|
|
1297
|
-
interoperability question rather than a detail.
|
|
1298
|
-
|
|
1299
|
-
The reference writer never exercises that range. Its names are a fixed prefix, an
|
|
1300
|
-
index and an extension — `index.html`, `manifest.json`, `stylesheet_0.css`,
|
|
1301
|
-
`images/1.png`, `fonts/2.woff2`, `scripts/3.js`, `frames/4/`, `page.pdf` — and the
|
|
1302
|
-
extension comes either from a table of content types or from a URL pathname, which is
|
|
1303
|
-
percent-encoded. Every name it writes is therefore ASCII, whatever the language of the
|
|
1304
|
-
captured page. That is a property of this writer, not a guarantee of the format: a
|
|
1305
|
-
conforming writer may name entries after the resources themselves, and §7.3's rule
|
|
1306
|
-
that entry names are untrusted assumes one does.
|
|
1307
|
-
|
|
1308
|
-
How a name is encoded is ZIP's own business, not this format's: bit 11 of the general
|
|
1309
|
-
purpose bit flag selects UTF-8, and its absence selects the legacy code page. This
|
|
1310
|
-
document adds two requirements to that and specifies nothing else about it.
|
|
1311
|
-
|
|
1312
|
-
**A writer MUST set bit 11 whenever a name or a comment needs it**, and the rule for
|
|
1313
|
-
when it does is ZIP's, not this format's: an encoded name or comment holding a byte
|
|
1314
|
-
outside printable ASCII needs it, one holding only printable ASCII does not, since the
|
|
1315
|
-
two encodings agree there. Control characters count as needing it, the legacy code page
|
|
1316
|
-
mapping them to graphic characters rather than to themselves. Setting it on names that
|
|
1317
|
-
do not need it is allowed and used to be required here; it was dropped because readers
|
|
1318
|
-
disagree about the flag more than they disagree about ASCII, so the safest name is the
|
|
1319
|
-
one that does not exercise the question. A writer MUST NOT set it on a name it then
|
|
1320
|
-
encodes in the legacy code page, which is the one combination that is simply wrong.
|
|
1321
|
-
|
|
1322
|
-
**A reader MUST honor the flag** rather than assume one encoding, and MUST expect to
|
|
1323
|
-
meet a clear one — which, in an archive from the reference writer, is most of them:
|
|
1324
|
-
that writer percent-encodes every name it produces, so every name is printable ASCII
|
|
1325
|
-
and carries no flag, while an entry comment holding the original URL of a resource can
|
|
1326
|
-
carry one when that URL is not ASCII. The hand-built `page.pdf` records (§3.1, §6) are
|
|
1327
|
-
the only ones the reference writer does not produce through its ZIP writer, and they
|
|
1328
|
-
follow the same rule: `page.pdf` is ASCII, so they carry no flag either, and no entry
|
|
1329
|
-
in the archive is decoded differently from the rest. Archives written before this rule
|
|
1330
|
-
was relaxed carry the flag on every entry instead. Both decode identically, which is
|
|
1331
|
-
the point, but a reader that hardcodes either answer meets the other one eventually.
|
|
1332
|
-
|
|
1333
|
-
A name is not a path. §7.3's rule that entry names are untrusted applies to the decoded
|
|
1334
|
-
name, and decoding is the step before that check, not a substitute for it.
|
|
1335
|
-
|
|
1336
|
-
## 6. Writer algorithm
|
|
1337
|
-
|
|
1338
|
-
This section specifies the reference writer's build order. It is normative in the
|
|
1339
|
-
sense that a file produced differently but satisfying every rule above is a valid
|
|
1340
|
-
SingleFile archive; the order matters because several values can only be computed
|
|
1341
|
-
once later bytes exist.
|
|
1342
|
-
|
|
1343
|
-
Most of that difficulty is optional, and a writer should know how much of it each face
|
|
1344
|
-
buys. The cost is not evenly spread:
|
|
1345
|
-
|
|
1346
|
-
| To produce | The writer needs |
|
|
1347
|
-
|---|---|
|
|
1348
|
-
| The ZIP face alone | Nothing from this section. Write an ordinary archive with `index.html` first and a `manifest.json`; no wrapper, no retry, no patching |
|
|
1349
|
-
| Plus the HTML face | The prologue and bootstrap, and the wrapper ladder of §5.1 — one scan of the finished archive, and a rebuild if the rung changes |
|
|
1350
|
-
| Plus universal mode | The recovery payload, the character round trip of §5.5, the appended-data budget of §5.2, and the retry loops of §6.2. This is where the real complexity lives, and it buys opening the file from `file:` with no cooperation |
|
|
1351
|
-
| Plus the PDF or PNG face | The header window of §4.3 or the chunk patching of §5.3, plus a second wrapper choice for the embedded payload |
|
|
1352
|
-
|
|
1353
|
-
The second row already needs a rebuild when the rung changes; the third adds the rest of
|
|
1354
|
-
the retry loops and the first value computed only once the archive is final, the
|
|
1355
|
-
recovery payload; the fourth adds the second such value, the PNG chunk length and CRC
|
|
1356
|
-
(§5.4). A writer that only wants durable saved
|
|
1357
|
-
pages can stop at the first row; the files it produces are accepted by every reader in
|
|
1358
|
-
§8.1.
|
|
1359
|
-
|
|
1360
|
-
### 6.1 Build order
|
|
1361
|
-
|
|
1362
|
-
1. **PNG head.** With the PNG face, copy the signature and `IHDR` from the source
|
|
1363
|
-
image unchanged. With the HTML face, choose the wrapper for the pixel-data payload
|
|
1364
|
-
(§5.1) and emit the `tEXt "PNG"` chunk: its 12 header bytes, the head of the
|
|
1365
|
-
prologue through `<body hidden>` as built in steps 2 and 3, the wrapper start tag,
|
|
1366
|
-
and the chunk CRC. Without the HTML face but with the PDF face, emit the
|
|
1367
|
-
`tEXt "PDF"` chunk holding the PDF document here instead, so its header falls
|
|
1368
|
-
inside the PDF scan window (§4.3). Then copy every source chunk between `IHDR` and
|
|
1369
|
-
`IEND`, write the `tEXt "ZIP"` chunk header with a zero length that step 12 patches,
|
|
1370
|
-
and, with the HTML face, the pixel-data wrapper end tag.
|
|
1371
|
-
2. **HTML prologue.** With the HTML face, emit the doctype (omitted under the PNG
|
|
1372
|
-
face, which owns the start of the file), the root element start tag, the
|
|
1373
|
-
`<meta charset>` required by §2.1, any comment the implementation adds — after the
|
|
1374
|
-
charset declaration, since a comment carrying the page URL has no bound and would
|
|
1375
|
-
otherwise push that declaration out of the first 1024 bytes. The doctype is the
|
|
1376
|
-
other unbounded region ahead of the declaration, copied from the saved page with its
|
|
1377
|
-
identifiers verbatim, so a writer MUST emit a minimal doctype in its place when
|
|
1378
|
-
keeping it would push the declaration past 1024 bytes.
|
|
1379
|
-
|
|
1380
|
-
**Replace it; do not truncate it, and do not drop it.** Truncation is unsafe:
|
|
1381
|
-
a cut inside a quoted identifier leaves the tokenizer in the system-identifier
|
|
1382
|
-
state, where it consumes the markup that follows until the next `>` — swallowing
|
|
1383
|
-
the root element start tag and the `data-sfz` marker on it (§1.3), so the document
|
|
1384
|
-
loses both. Dropping the doctype parses
|
|
1385
|
-
cleanly but puts the document in quirks mode, which is the mode the blank-page
|
|
1386
|
-
backstop, the wait message and the error message are then rendered under (§4.1) —
|
|
1387
|
-
the error message most of all, since it is what a reader sees precisely when
|
|
1388
|
-
nothing else has worked. A minimal doctype is 15 bytes, keeps standards mode, and
|
|
1389
|
-
costs nothing else: the extracted page is written into the document with its own
|
|
1390
|
-
doctype (§4.1), so the outer one never governs the restored page.
|
|
1391
|
-
|
|
1392
|
-
The PNG face is the exception, and nothing is available to it either way. A PNG file
|
|
1393
|
-
MUST begin with its 8-byte signature, so no doctype can precede it, and one written
|
|
1394
|
-
after the PNG head is discarded: those bytes are character data, so the parser has
|
|
1395
|
-
left its initial insertion mode and ignores a DOCTYPE token. The variant renders in
|
|
1396
|
-
quirks mode until the extracted page replaces it, whatever the writer does, so this
|
|
1397
|
-
section requires nothing about the doctype there. A writer MAY drop it, as the
|
|
1398
|
-
reference writer does, or keep it — but a kept one is content like any other, and
|
|
1399
|
-
both windows are measured from the start of the *file*, which under this face begins
|
|
1400
|
-
45 bytes before the HTML does: the signature, `IHDR`, and the chunk length, type,
|
|
1401
|
-
keyword and NUL separator. That is 45 bytes less room than the arithmetic above
|
|
1402
|
-
suggests.
|
|
1403
|
-
|
|
1404
|
-
Then the head elements (the
|
|
1405
|
-
`<title>` and the canonical link among them), the CSS and `<body hidden>`,
|
|
1406
|
-
the wait and error messages, the optional text body, and the
|
|
1407
|
-
bootstrap script. With a password, five of those are left out: the comment, the
|
|
1408
|
-
title, the canonical link, the text body and the entry comments of step 6 (§5.6).
|
|
1409
|
-
With the PNG face the head of this region,
|
|
1410
|
-
through `<body hidden>`, is the data of the `tEXt "PNG"` chunk and the remainder is
|
|
1411
|
-
emitted after the `tEXt "ZIP"` chunk header, which step 1 has already written and
|
|
1412
|
-
step 12 only patches; with the PDF face the
|
|
1413
|
-
region is interrupted by step 3 as well.
|
|
1414
|
-
|
|
1415
|
-
Whatever a writer puts in the prologue, closing every element it opens before the
|
|
1416
|
-
wrapper start tag is good practice but not a requirement: the extractor addresses
|
|
1417
|
-
the ZIP region by identifier (§4.5), so an element left open only makes the archive
|
|
1418
|
-
a descendant of it, and the lookup resolves the same way.
|
|
1419
|
-
3. **Embedded PDF.** With the PDF face and the HTML face, the prologue is *split*
|
|
1420
|
-
around the PDF, which MUST come early enough for `%PDF-` to start at offset 1024
|
|
1421
|
-
or lower (§4.3). Only what a parser needs first precedes it — the doctype, the
|
|
1422
|
-
root element and the charset declaration — and everything
|
|
1423
|
-
else in the head (title, link and meta elements, the stylesheet, `<body hidden>`,
|
|
1424
|
-
the messages, the optional text body) follows it. Emit the
|
|
1425
|
-
wrapper start tag chosen for the PDF payload (§5.1), the hand-built `page.pdf`
|
|
1426
|
-
local file header, the PDF document, the wrapper end tag, and record the local
|
|
1427
|
-
header's absolute position; then resume the prologue. The reference writer's
|
|
1428
|
-
header declares version 2.0, no general purpose bit flag, method STORE, the
|
|
1429
|
-
build's modification date in DOS form, the precomputed CRC-32, the document's
|
|
1430
|
-
length as both sizes, and no extra field; its central record adds a Unix
|
|
1431
|
-
"made by" version and external attributes of a regular file, mode 0644.
|
|
1432
|
-
|
|
1433
|
-
The window is reachable but not structurally guaranteed, and it is the one place
|
|
1434
|
-
where the format depends on the writer rather than on its own layout. The
|
|
1435
|
-
irreducible part of the prefix is small: the root element start tag, the charset
|
|
1436
|
-
declaration, the wrapper start tag and the 38-byte local file header for
|
|
1437
|
-
`page.pdf`, plus a minimal doctype — 92 bytes in the reference layout with a
|
|
1438
|
-
`utf-8` label, 99 with `windows-1252`, and a few more with a wrapper past the first
|
|
1439
|
-
rung (§5.1). But two
|
|
1440
|
-
regions ahead of the header have no length the format controls: the doctype, which
|
|
1441
|
-
is copied from the saved page and carries its public and system identifiers
|
|
1442
|
-
verbatim, and any comment the implementation chooses to write there. Real doctypes
|
|
1443
|
-
are small; the longest in common use, XHTML 1.1 with MathML and SVG, is about 140
|
|
1444
|
-
bytes, though a crafted one is bounded only by what the parser accepts. Step 2's
|
|
1445
|
-
MUST already caps the doctype, but only far enough to keep the charset declaration
|
|
1446
|
-
inside the window; this header sits further into the file, behind the wrapper tag
|
|
1447
|
-
and a 38-byte local header, so it needs the tighter bound below and the comment
|
|
1448
|
-
needs one too. A writer MUST cap them itself, keeping
|
|
1449
|
-
everything before the local file header inside the remaining budget of roughly 930
|
|
1450
|
-
bytes (about 900 with the PNG face, whose signature, `IHDR` and first chunk header
|
|
1451
|
-
take the first 45 bytes of the same window while its variant drops the 15-byte
|
|
1452
|
-
doctype in exchange), shortening, dropping or relocating that content instead of emitting a
|
|
1453
|
-
header outside the window. A writer that places nothing of unbounded length before
|
|
1454
|
-
the PDF block satisfies the rule by construction and needs no check at all.
|
|
1455
|
-
|
|
1456
|
-
The reference writer does both. Its provenance comment is emitted after the PDF
|
|
1457
|
-
block, so the page URL it carries cannot reach the window at all, and the prefix is
|
|
1458
|
-
measured before the header is written: when the page's own doctype would push
|
|
1459
|
-
`%PDF-` past 1024, `<!DOCTYPE html>` is emitted in its place, on step 2's rule. A
|
|
1460
|
-
writer that must keep the page doctype has to find the room elsewhere.
|
|
1461
|
-
|
|
1462
|
-
Without the HTML face, the PDF is simply the first thing in the file and the
|
|
1463
|
-
question does not arise.
|
|
1464
|
-
4. **Reserved extra-data.** In universal mode, when a previous pass determined that
|
|
1465
|
-
the payload must be relocated (§5.2), emit an empty `<sfz-extra-data>` element
|
|
1466
|
-
followed by enough spaces to fill the reservation. The padding sits **outside** the
|
|
1467
|
-
element, so the element's text stays exactly the payload. With
|
|
1468
|
-
`preventAppendedData` set from the start there is still no reservation on the first
|
|
1469
|
-
pass: that pass measures the payload, and the second reserves (§6.2).
|
|
1470
|
-
5. **Wrapper start tag** for the ZIP region, carrying the identifier (§5.1).
|
|
1471
|
-
6. **The archive.** Create the ZIP writer, telling it the number of bytes already
|
|
1472
|
-
written so that its offsets are absolute (§5.3). Add `index.html` first, then
|
|
1473
|
-
`manifest.json`, then the page's resources, preserving that order in the central
|
|
1474
|
-
directory; STORE entries whose content is already compressed and deflate the rest;
|
|
1475
|
-
put each resource's source URL in its entry comment; encrypt entry contents if a
|
|
1476
|
-
password was given.
|
|
1477
|
-
7. **PDF central record.** With the PDF face and the HTML face, write the record for
|
|
1478
|
-
`page.pdf`, with the local header offset from step 3, immediately before closing
|
|
1479
|
-
the writer.
|
|
1480
|
-
8. **Close and patch.** Close the archive, then correct the end of central directory
|
|
1481
|
-
record for the injected record: entry counts, directory size, and the zip64
|
|
1482
|
-
record and locator when present (§5.7).
|
|
1483
|
-
9. **Wrapper check, then the universal payload.** With the HTML face — not only in
|
|
1484
|
-
universal mode, since any self-extracting file needs it — read back the ZIP region
|
|
1485
|
-
and check it against the current wrapper (§5.1); on a collision, restart (§6.2).
|
|
1486
|
-
Then, in universal mode only, compute the region's CRC-32 and its newline codes,
|
|
1487
|
-
build and compress the payload, and decide its placement against the budget
|
|
1488
|
-
(§5.2), restarting when an appended payload turns out not to fit. Relocation is
|
|
1489
|
-
never undone (§6.2).
|
|
1490
|
-
10. **Appended run.** Unless appended data is prevented or the payload is relocated
|
|
1491
|
-
(§5.2), emit the wrapper end tag,
|
|
1492
|
-
the extra-data element when it is appended, and `</body></html>` — the end tags
|
|
1493
|
-
are omitted under the PNG face, which must end with `IEND`.
|
|
1494
|
-
11. **Fill the reservation.** In the relocated placement, write the payload into the
|
|
1495
|
-
space reserved in step 4; if it no longer fits, restart (§6.2). Under
|
|
1496
|
-
`declareAppendedData` (§4.2), the EOCD's comment-length field is patched here too,
|
|
1497
|
-
the appended run's length now being final, unless the value would complete a
|
|
1498
|
-
pattern of the current wrapper, in which case the raw form stays (§5.1).
|
|
1499
|
-
12. **PNG tail.** With the PNG face, patch the `tEXt "ZIP"` chunk's length field, now
|
|
1500
|
-
that the total size is known, compute that chunk's CRC over everything from its
|
|
1501
|
-
type to the last byte written, and append the CRC and the `IEND` chunk.
|
|
1502
|
-
|
|
1503
|
-
### 6.2 The retry loops
|
|
1504
|
-
|
|
1505
|
-
Four conditions restart the build from step 1, and each restart carries forward what
|
|
1506
|
-
the failed pass learned. Nothing a restart changes reaches the entries' bytes: a writer
|
|
1507
|
-
may compress them once and copy them into every pass, rewriting only the central
|
|
1508
|
-
directory's offsets, which is what the reference writer does. The first three
|
|
1509
|
-
terminate because each of them advances a monotone quantity:
|
|
1510
|
-
|
|
1511
|
-
- **Wrapper collision** (§5.1): the next pass starts at the next rung of the ladder.
|
|
1512
|
-
The ladder is finite and its last rung, `<plaintext>`, is exempt from both selection
|
|
1513
|
-
tests, so it always fits.
|
|
1514
|
-
- **Payload does not fit the appended budget** (§5.2): the next pass reserves room
|
|
1515
|
-
ahead of the archive, sized at the measured payload length plus a margin.
|
|
1516
|
-
- **Reservation too small**: relocating the payload changes the file's layout, hence
|
|
1517
|
-
its offsets, hence the payload, which can grow past the room reserved for it. The
|
|
1518
|
-
next pass reserves the new length plus the same margin. This restart fires only
|
|
1519
|
-
when the payload outgrew its reservation and it reserves at least that payload, so
|
|
1520
|
-
every reservation is larger than the one before and the loop cannot revisit a size.
|
|
1521
|
-
What keeps it short is the margin. Shifting the offsets changes a few of the
|
|
1522
|
-
central directory's bytes, which changes the line-ending codes, the deflate output
|
|
1523
|
-
and the base64 rounding, so the payload moves by a few quanta of 4 characters
|
|
1524
|
-
between two layouts: measured between -16 and +20 characters over archives of 8 to
|
|
1525
|
-
2000 entries. A margin smaller than that shift buys a third pass in about one build
|
|
1526
|
-
out of four. The writer reserves the measured length plus 1 % plus 32 characters,
|
|
1527
|
-
which absorbed every shift measured.
|
|
1528
|
-
|
|
1529
|
-
There is no converse of the second: a pass that reserved room never discards it,
|
|
1530
|
-
even when the relocated payload would have fit the appended window. Relocation moves
|
|
1531
|
-
the archive, which changes the offsets, which changes the payload that made the
|
|
1532
|
-
relocation necessary, so a payload lying on the budget boundary can be too large
|
|
1533
|
-
appended and small enough relocated, and a writer that dropped the reservation could
|
|
1534
|
-
rebuild the two placements forever. Relocation is therefore final (§5.2), and the file
|
|
1535
|
-
keeps at most the reservation's own margin of dead padding.
|
|
1536
|
-
|
|
1537
|
-
The fourth stands apart from the other three, and terminates trivially because it can
|
|
1538
|
-
fire only once: if the end of central directory record cannot be patched to account for
|
|
1539
|
-
the injected `page.pdf` record — its signature not where the accounting expects it —
|
|
1540
|
-
the writer rebuilds without that record rather than leave a central directory the EOCD
|
|
1541
|
-
does not count. That is the restart enforcing §5.7's requirement that the injection
|
|
1542
|
-
never leave the two disagreeing, and the rebuilt archive simply has no `page.pdf`
|
|
1543
|
-
entry.
|
|
1544
|
-
|
|
1545
|
-
Given identical inputs, modification date and archive time, the process is
|
|
1546
|
-
deterministic: the same page produces the same bytes, retries included. `manifest.json`
|
|
1547
|
-
records when the archive was made (§7.1), so two builds of one page at two moments
|
|
1548
|
-
differ in that entry and in the entry sizes around it. A writer that retries MUST pin
|
|
1549
|
-
the archive time across the passes of one build rather than read the clock again on
|
|
1550
|
-
each. The reference writer reads the clock once, inside the callback that emits the
|
|
1551
|
-
entries, which runs once per build; every retry reuses the entries that callback
|
|
1552
|
-
produced. Two builds of one page still read the clock twice, so its own determinism
|
|
1553
|
-
test freezes it. A consumer MUST NOT
|
|
1554
|
-
treat the byte identity of two archives of the same page as meaningful.
|
|
1555
|
-
|
|
1556
|
-
## 7. Consuming SingleFile archives safely
|
|
1557
|
-
|
|
1558
|
-
This section addresses software that reads SingleFile archives it did not produce.
|
|
1559
|
-
The ZIP face is the interoperable one, and a reader that follows the rules below
|
|
1560
|
-
handles every variant of §2 without knowing which one it has.
|
|
1561
|
-
|
|
1562
|
-
### 7.1 Reading
|
|
1563
|
-
|
|
1564
|
-
- **Read through the central directory.** Locate the End Of Central Directory record
|
|
1565
|
-
by scanning backward from the end of the file, then follow its offset. A reader that
|
|
1566
|
-
streams local headers from offset 0 will not find an archive in any variant that has
|
|
1567
|
-
a face, since the file then starts with the HTML, PDF or PNG face. The variant with
|
|
1568
|
-
no face is an ordinary ZIP file and streams fine (§1.2, and §8.1 measures what such
|
|
1569
|
-
readers actually do).
|
|
1570
|
-
- **Tolerate bytes before and after the archive.** They are the other faces, not
|
|
1571
|
-
corruption. Offsets are absolute, so no compensation is needed (§5.3).
|
|
1572
|
-
- **Accept both forms of appended data.** The bytes after the EOCD record may be raw
|
|
1573
|
-
or declared as the archive comment; both are valid (§4.2). A reader MUST NOT treat
|
|
1574
|
-
undeclared trailing bytes as a defect.
|
|
1575
|
-
- **Do not identify the format by file name.** Extensions are conventions (§2.2). A
|
|
1576
|
-
SingleFile archive is identifiable from its content, and recognition and extraction
|
|
1577
|
-
use different tests: an `index.html` entry accompanied by a `manifest.json` entry in
|
|
1578
|
-
the same directory is the positive signal for recognizing the format, while
|
|
1579
|
-
`index.html` alone is enough to *extract* from, since `manifest.json` is informative
|
|
1580
|
-
and MUST NOT be required (below). Neither test distinguishes this format from an
|
|
1581
|
-
arbitrary ZIP file laid out the same way, and none is offered, because nothing in the
|
|
1582
|
-
format depends on recognizing it. A reader that treats any archive containing a page
|
|
1583
|
-
entry as a saved page loses nothing.
|
|
1584
|
-
- **Resolve the page entry in this order.** The archive's internal layout is
|
|
1585
|
-
implementation-defined, and two properties of the reference layout matter to a
|
|
1586
|
-
reader. Every entry MAY sit under a single root directory, which the reference
|
|
1587
|
-
writer names `<milliseconds since the epoch>_<tab id>/` when asked to create one
|
|
1588
|
-
(`createRootDirectory`); the page is then `<root>/index.html`. And a page's nested
|
|
1589
|
-
frames are stored as complete pages of their own under `frames/<n>/`, recursively,
|
|
1590
|
-
each with its own `index.html` and `manifest.json`, so an archive normally holds
|
|
1591
|
-
several of both and only the outermost pair is the page. The recognition test above
|
|
1592
|
-
therefore matches every frame directory too. Since neither property
|
|
1593
|
-
is guaranteed, a reader resolves the entry point in three steps, stopping at the
|
|
1594
|
-
first that succeeds:
|
|
1595
|
-
|
|
1596
|
-
1. The `indexFilename` of the `manifest.json` at the smallest directory depth,
|
|
1597
|
-
resolved against that manifest's directory, when the entry exists. This is the
|
|
1598
|
-
only authoritative answer, so a writer that departs from
|
|
1599
|
-
the reference layout SHOULD emit the manifest even though a reader MUST NOT
|
|
1600
|
-
require it.
|
|
1601
|
-
2. Otherwise the `index.html` entry at the smallest directory depth.
|
|
1602
|
-
3. If several `index.html` entries tie at that depth, the archive does not name its
|
|
1603
|
-
page: a reader MUST NOT pick one arbitrarily. Report the ambiguity, or treat the
|
|
1604
|
-
file as a plain ZIP archive.
|
|
1605
|
-
|
|
1606
|
-
The reference writer never produces a tie, since it creates at most one root
|
|
1607
|
-
directory and nests every other page under `frames/<n>/`; step 3 exists for archives
|
|
1608
|
-
from other writers.
|
|
1609
|
-
- **Treat `manifest.json` as informative.** The reference writer records the original
|
|
1610
|
-
URL as `originalUrl`, the title as `title`, the save time as `archiveTime` (an ISO
|
|
1611
|
-
8601 string), the entry name of the page as `indexFilename` and the resource-to-URL
|
|
1612
|
-
map as `resources`. The page displays without any of it, and a reader MUST NOT require
|
|
1613
|
-
the entry or any field of it. `indexFilename` names the page relative to the
|
|
1614
|
-
manifest's own directory, not as a full entry name. A frame's manifest carries the
|
|
1615
|
-
same `archiveTime` as the page's. The set of fields is not closed: a reader
|
|
1616
|
-
MUST ignore what it does not recognize.
|
|
1617
|
-
- **Expect a `page.pdf` entry whose data lies outside the archive proper** (§4.2). It
|
|
1618
|
-
is an ordinary STORE entry at an ordinary offset, so nothing special is needed to
|
|
1619
|
-
read it, but a reader that assumes the entries are contiguous will reject or
|
|
1620
|
-
mislocate it: `page.pdf`'s local header is the first in the file, and the whole
|
|
1621
|
-
bootstrap lies between its data and the next one. It is never placed under the root
|
|
1622
|
-
directory: the PDF face is one document per file, so an archive holds at most one
|
|
1623
|
-
`page.pdf`, and it sits at the top level whatever `createRootDirectory` does to the
|
|
1624
|
-
other entries.
|
|
1625
|
-
|
|
1626
|
-
### 7.2 Modifying
|
|
1627
|
-
|
|
1628
|
-
Do not rewrite the file in place. Adding, removing or recompressing entries moves the
|
|
1629
|
-
ZIP region and invalidates the other faces: the PNG chunk length and CRC that span the
|
|
1630
|
-
archive, the recovery payload's checksum, and the wrapper choice that depends on the
|
|
1631
|
-
archive's exact bytes (§1.2). Most ZIP rewriters also drop the prepended and appended
|
|
1632
|
-
regions, which discards every face but ZIP.
|
|
1633
|
-
|
|
1634
|
-
A tool that wants to produce a modified archive MUST rebuild it through the writer
|
|
1635
|
-
rules of §6. A tool that only wants the page content SHOULD extract rather than
|
|
1636
|
-
rewrite.
|
|
1637
|
-
|
|
1638
|
-
The hazard is automated handling: a deduplicating store, a backup system that
|
|
1639
|
-
recompresses, a mail or chat service that repacks attachments, or any pipeline that
|
|
1640
|
-
round-trips the file through a ZIP library will return an archive whose entries are all
|
|
1641
|
-
intact and whose other faces are gone, with no error at any step and no record of it in
|
|
1642
|
-
the file. Software that stores these archives SHOULD treat them as opaque bytes, and a
|
|
1643
|
-
preservation workflow that cannot guarantee that SHOULD keep a checksum of the original
|
|
1644
|
-
alongside it.
|
|
1645
|
-
|
|
1646
|
-
### 7.3 Security considerations
|
|
1647
|
-
|
|
1648
|
-
- **Entry names are untrusted.** The reference writer's names are a fixed prefix, an
|
|
1649
|
-
index and an extension (§5.8), but nothing in the format requires that, and a writer
|
|
1650
|
-
may name entries after the resources themselves. A reader MUST sanitize them before
|
|
1651
|
-
writing to a filesystem: reject absolute paths and
|
|
1652
|
-
`..` segments, and be aware that names may be long, may collide after case folding,
|
|
1653
|
-
and may contain characters the local filesystem rejects.
|
|
1654
|
-
- **Declared sizes are untrusted.** Do not pre-allocate from the declared uncompressed
|
|
1655
|
-
size, and enforce a limit on the expansion ratio; the archive can be crafted like
|
|
1656
|
-
any other ZIP file.
|
|
1657
|
-
- **The archived page is untrusted web content**, and the HTML face contains a script.
|
|
1658
|
-
Software that displays either MUST do so in a sandboxed context, and MUST NOT run
|
|
1659
|
-
the bootstrap in a privileged one. The format's own display path replaces the
|
|
1660
|
-
document with the extracted page, which is not an isolation boundary by itself.
|
|
1661
|
-
- **A password protects entry contents only** (§5.6). Entry names, sizes and dates
|
|
1662
|
-
stay readable in the central directory, and while the reference writer's names carry
|
|
1663
|
-
no information about the resources (§5.8), another writer's may state their
|
|
1664
|
-
filenames. The PNG and PDF faces render the page regardless. A conforming writer
|
|
1665
|
-
withholds the five fields of §5.6, the source URLs among them, but a reader MUST NOT
|
|
1666
|
-
read their absence as protection: nothing in the format stops a writer from emitting
|
|
1667
|
-
any of them, so an archive of unknown provenance may state every URL in the clear.
|
|
1668
|
-
Software MUST NOT present a password-protected archive as an encrypted document.
|
|
1669
|
-
- **Sniffing disagrees with itself on these files.** `file(1)` reports HTML, PNG, PDF
|
|
1670
|
-
or "data" depending on the variant (§8.1), so a server that guesses the media type
|
|
1671
|
-
from content may serve a saved page as an image. Software that serves SingleFile
|
|
1672
|
-
archives SHOULD set the media type explicitly — `text/html` for the self-extracting
|
|
1673
|
-
variants, `application/zip` otherwise.
|
|
1674
|
-
|
|
1675
|
-
### 7.4 What to reject and what to tolerate
|
|
1676
|
-
|
|
1677
|
-
A reader of a polyglot file meets conditions that look like corruption and are not, and
|
|
1678
|
-
others that look harmless but leave the extracted page wrong. A condition matters if and
|
|
1679
|
-
only if it affects the bytes the page is built from:
|
|
1680
|
-
|
|
1681
|
-
| Condition | Reader behavior |
|
|
1682
|
-
|---|---|
|
|
1683
|
-
| A recovery payload field disagrees with the reconstruction — length, newline count or checksum | **MUST** fail (§4.5). The reconstruction is wrong and nothing built from it can be trusted |
|
|
1684
|
-
| An entry's CRC-32 or AES authentication code does not match | **SHOULD** fail for that entry, and MUST NOT present a page rebuilt from it as intact |
|
|
1685
|
-
| `page.pdf` was reconstructed from the parsed page and its CRC-32 does not match | **MUST** discard the reconstruction (§4.5). The bytes are a guess about newlines the recovery payload does not describe, and the checksum is the only thing that tests it — unlike the row above, there is no read to have gone wrong, only an inference |
|
|
1686
|
-
| Bytes outside the archive proper — before the first local file header, after the EOCD record, or between an entry's data and the next header | **MUST** tolerate: they are the other faces (§7.1). The gap in the middle is not hypothetical: with the PDF face the bootstrap lies between `page.pdf`'s data and the ZIP region |
|
|
1687
|
-
| The appended run exceeds the 65535-byte ceiling (§5.2) | Not a reader's problem: if the EOCD record was found, the archive is readable. Readers MAY warn |
|
|
1688
|
-
| A `tEXt` chunk CRC does not match, or a chunk holds bytes PNG does not permit (§4.4) | Irrelevant to extraction; a reader of the archive MAY ignore both |
|
|
1689
|
-
| `page.pdf` is present but its data does not begin with `%PDF-` | Not an error. The entry is data like any other |
|
|
1690
|
-
| `index.html` is present without `manifest.json` | **MUST** still extract (§7.1) |
|
|
1691
|
-
| More than one candidate carries the `sfz-data` identifier once §4.5's tie-break has been applied | **MUST NOT** extract either silently. The tie-break comes first and settles the ordinary pairing: an id-bearing element that is one of §5.1's wrapper rungs wins over a comment, and one that is not a rung loses to it, since the `id` is then something else in the page. What this row forbids is what the tie-break does not reach — two elements, or two comments, or an element and a comment that both survive it. A conforming writer emits one candidate (§5.1), so a second is a payload that escaped its wrapper, most often a nested archive written by a writer that emitted a face bare. Both extract cleanly and check out, and the checksums say nothing about which one the file was built around |
|
|
1692
|
-
| The recovered region (universal mode) disagrees with the same bytes read directly, in the EOCD's two comment-length bytes only | Expected, not an error. A recovered region always declares a zero-length comment (§4.5), so it differs here from any archive written in the declared form (§4.2). Compare the two only up to those bytes |
|
|
1693
|
-
| The recovered region (universal mode) disagrees with the same bytes read directly, anywhere else | The file is not well-formed, whichever side is at fault, and a reader that has both MUST NOT silently merge them or pick per entry. Prefer the direct read — it is the writer's own output, where the recovered region is a reconstruction of it — and surface the disagreement rather than displaying either as intact |
|
|
1694
|
-
|
|
1695
|
-
Anything the format does not constrain, a reader MUST NOT reject: entries may carry any
|
|
1696
|
-
extra fields, timestamps or data descriptors a ZIP writer would ordinarily emit. The few
|
|
1697
|
-
this document does constrain — the `0x9901` field of an encrypted entry (§4.2), the
|
|
1698
|
-
zip64 records (§5.7), the name-encoding flag (§5.8) — say how an entry is read, not
|
|
1699
|
-
whether it is acceptable, so this row covers them too: each is something a reader meets
|
|
1700
|
-
and reads.
|
|
1701
|
-
|
|
1702
|
-
## 8. Appendices
|
|
1703
|
-
|
|
1704
|
-
### 8.1 Tool compatibility
|
|
1705
|
-
|
|
1706
|
-
Measured on macOS 26 with the specimens of §8.3 (Info-ZIP UnZip 6.00, libarchive
|
|
1707
|
-
3.7.4, Python 3.14, OpenJDK 21, 7-Zip 25.01, poppler `pdftotext`, macOS `ditto`,
|
|
1708
|
-
`sips` and Quick Look). Every result is predicted by two structural properties, so
|
|
1709
|
-
the variants are grouped by them:
|
|
1710
|
-
|
|
1711
|
-
| Class | Bytes before the archive | Bytes after the EOCD | Variants |
|
|
1712
|
-
|---|---|---|---|
|
|
1713
|
-
| A | — | — | pure zip |
|
|
1714
|
-
| B | yes | — | relocated (`preventAppendedData`), zip-pdf |
|
|
1715
|
-
| C | yes | yes | plain, universal, ladder, password, pdf, png, png-pdf, zip-png, zip-png-pdf, zip64 |
|
|
1716
|
-
|
|
1717
|
-
The classes follow the bytes, not the options. `preventAppendedData` puts the
|
|
1718
|
-
relocated variant in class B, but it suppresses *markup* after the archive, not the PNG
|
|
1719
|
-
face's tail: an archive combining that option with a PNG face has bytes after the EOCD
|
|
1720
|
-
and is class C.
|
|
1721
|
-
|
|
1722
|
-
| ZIP reader | A | B | C | Behavior |
|
|
1723
|
-
|---|---|---|---|---|
|
|
1724
|
-
| Info-ZIP `unzip`, `zipinfo` | ✔ | ✔ | ✔ | Lists and extracts every variant. AES entries are skipped — `need PK compat. v5.1 (can do v4.5)` — a limitation of the tool, not of the file; `page.pdf` still extracts because it is never encrypted |
|
|
1725
|
-
| Python `zipfile` | ✔ | ✔ | ✔ | Lists and extracts every variant |
|
|
1726
|
-
| 7-Zip (`7zz`) | ✔ | ✔ | ✔ | Lists and extracts every variant, AES included |
|
|
1727
|
-
| libarchive `bsdtar`, seekable input | ✔ | ✔ | ✔ | Lists and extracts every variant. Its EOCD scan is the narrowest measured, so a class-C file whose appended run pushes the record past 16383 bytes from the end is rejected with `Unrecognized archive format`; §5.2's default budget is sized to this window, and the ✔ holds for files that respect it |
|
|
1728
|
-
| libarchive `bsdtar`, piped input | ✔ | ✘ | ✘ | `Unrecognized archive format` — the forward-only case of §1.2, measured |
|
|
1729
|
-
| Java `java.util.zip` (`jar tf`) | ✔ | ✔ | ✘ | `zip END header not found` whenever bytes follow the EOCD undeclared. Declaring them as the archive comment makes the same file open, measured on every class-C variant (§4.2) |
|
|
1730
|
-
| macOS `ditto -x -k` | ✔ | ✘ | ✘ | `Couldn't read PKZip signature` — requires a local file header at offset 0, so prepended data alone defeats it |
|
|
1731
|
-
|
|
1732
|
-
The backward scans behind the class-C column differ by an order of magnitude, and they
|
|
1733
|
-
are what §5.2's budget is sized against. Measured by padding a working archive until
|
|
1734
|
-
the record fell out of reach, the largest distance from end-of-file at which each
|
|
1735
|
-
reader still finds the EOCD record is: libarchive 16383, perl `Archive::Zip` 32768,
|
|
1736
|
-
Python `zipfile` 65557, zip.js 65536, Info-ZIP `unzip` 68000, and 7-Zip beyond 1 MiB,
|
|
1737
|
-
which scans the whole file. libarchive binds, and its window less the 22-byte record
|
|
1738
|
-
is the 16361-byte default budget of §5.2. macOS `ditto` is not on this axis at all: it
|
|
1739
|
-
requires a local file header at offset 0 whatever the tail looks like.
|
|
1740
|
-
|
|
1741
|
-
The cost of the declared form was measured on the same tools: it is a display cost, not
|
|
1742
|
-
a compatibility one. An archive whose trailing bytes are declared as the comment has
|
|
1743
|
-
them printed back on ordinary listings; `unzip -l` reproduces the whole run — in
|
|
1744
|
-
universal mode that is the `-->`, the entire `<sfz-extra-data>` element and the end tags
|
|
1745
|
-
— under the archive's own header. The raw form is the default for that reason (§4.2).
|
|
1746
|
-
|
|
1747
|
-
The other faces were exercised on the variants that carry them, and all succeeded:
|
|
1748
|
-
`pdftotext` extracts the page text from every PDF-face variant, including the
|
|
1749
|
-
password-protected one (`page.pdf` is never encrypted, §5.6), and `sips` reports the
|
|
1750
|
-
screenshot's true dimensions for every PNG-face variant. macOS Quick Look renders the
|
|
1751
|
-
PNG face of the self-extracting PNG specimen, and on the all-four-faces specimen it
|
|
1752
|
-
renders both faces of the same bytes: renamed to `.png` it yields the screenshot,
|
|
1753
|
-
renamed to `.pdf` the rendered document.
|
|
1754
|
-
|
|
1755
|
-
The PDF header window of §4.3 was measured rather than assumed, by moving `%PDF-`
|
|
1756
|
-
progressively later in otherwise identical archives. PDFium (build 153.0.7999.0, the
|
|
1757
|
-
engine of every Chromium-based browser) loads the document while the header starts at
|
|
1758
|
-
offset 1024 or less and fails with a data-format error from 1025 on — the documented
|
|
1759
|
-
1024-byte figure, enforced exactly. poppler `pdftotext` and macOS PDFKit render the
|
|
1760
|
-
same files with the header at 1000222 bytes, so they impose no window at all. A
|
|
1761
|
-
writer that keeps the header inside 1024 bytes satisfies every engine tested.
|
|
1762
|
-
|
|
1763
|
-
`file(1)` disagrees with itself across the variants. What it reports depends on
|
|
1764
|
-
where the archive falls relative to the fixed buffer it sniffs, not on the variant as
|
|
1765
|
-
such: a small self-extracting file whose archive starts within that buffer is
|
|
1766
|
-
reported as `data` where a large one is reported as HTML. On these specimens:
|
|
1767
|
-
`HTML document text` for the plain, universal and ladder specimens, `PNG image data`
|
|
1768
|
-
for every PNG-face variant, `PDF document` for a
|
|
1769
|
-
PDF-first archive, `Zip archive data` for a pure archive **and for the relocated
|
|
1770
|
-
variant**, whose first bytes are a doctype, and `data` where the HTML head carries
|
|
1771
|
-
the embedded PDF.
|
|
1772
|
-
|
|
1773
|
-
Two of these results bear on §1.1's customary tolerances. The EOCD backward scan and
|
|
1774
|
-
the tolerance of undeclared trailing bytes are near-universal but not unanimous: Java
|
|
1775
|
-
is the measured exception, and not a niche one, since `java.util.zip` is what
|
|
1776
|
-
Android and most JVM tooling use. Prepended data is tolerated by every ZIP reader
|
|
1777
|
-
measured except Apple's `ditto`.
|
|
1778
|
-
|
|
1779
|
-
### 8.2 Anatomy of a small archive
|
|
1780
|
-
|
|
1781
|
-
Offsets in `universal.sfz.html` (123077 bytes, two entries, saved from `example.com`
|
|
1782
|
-
with the §8.3 command, on the build named there).
|
|
1783
|
-
The layout is the *universal* row of the byte map (§3).
|
|
1784
|
-
|
|
1785
|
-
These numbers are one capture, not a contract. Everything from the bootstrap onward
|
|
1786
|
-
moves whenever the inlined ZIP library changes size, so treat the table as an
|
|
1787
|
-
illustration of the shape and not as values to compare a file against. What *is* fixed
|
|
1788
|
-
is the set of relations between the rows — the doctype opening the file with the root
|
|
1789
|
-
element start tag immediately after it, the charset declaration immediately after that
|
|
1790
|
-
and the comment immediately after that, the identifier's twelve bytes ahead of the
|
|
1791
|
-
region, the EOCD's directory offset being an absolute file position, and the entry
|
|
1792
|
-
order. Those are checked by `test/sfz-harness/byte-map.js`, which builds an equivalent
|
|
1793
|
-
specimen without a network.
|
|
1794
|
-
|
|
1795
|
-
| Offset | Bytes | Region |
|
|
1796
|
-
|---|---|---|
|
|
1797
|
-
| 0 | `<!DOCTYPE html>` | `html-prologue` begins |
|
|
1798
|
-
| 15 | `<html data-sfz>` | root element start tag; the attribute is the reference implementation's own marker (§1.3) |
|
|
1799
|
-
| 30 | `<meta charset=windows-1252>` | the charset rule, inside the first 1024 bytes (§2.1) |
|
|
1800
|
-
| 57 | `<!--` … `-->` (ends at 200) | comment written by the implementation; its content is implementation-defined, but where it may appear is not (§3.1, §4.6, §5.6). It follows the charset declaration so it cannot push it out of the prescan window |
|
|
1801
|
-
| 200 | `<title>` … `</title>` (ends at 229) | the page title, as numeric character references (§4.6) |
|
|
1802
|
-
| 677 | `<style>` | the stylesheet of the blank-page backstop (§4.1) |
|
|
1803
|
-
| 855 | `<body hidden>` | start of the blank-page backstop (§4.1) |
|
|
1804
|
-
| 868, 913 | wait and error messages | the two visible elements |
|
|
1805
|
-
| 1134 | `<script>` … `</script>` (ends at 121993) | `bootstrap`: ZIP reader, extractor, display, acquisition |
|
|
1806
|
-
| 121993 | `<!--sfz-data` | wrapper start tag, opening the ZIP region and carrying its identifier (§5.1) |
|
|
1807
|
-
| 122005 | `PK\3\4` | first local file header, `index.html` (1110 bytes, deflated to 593) — the ZIP region begins |
|
|
1808
|
-
| 122647 | `PK\3\4` | local file header, `manifest.json` (168 bytes, deflated to 130) |
|
|
1809
|
-
| 122829 | `PK\1\2` | central directory: `index.html`, then `manifest.json` |
|
|
1810
|
-
| 122981 | `PK\5\6` | EOCD: 2 entries, directory size 152, directory offset 122829 — an absolute file position (§5.3) — comment length 0 |
|
|
1811
|
-
| 123003 | `-->` | wrapper close tag; the ZIP region ends here |
|
|
1812
|
-
| 123006 | `<sfz-extra-data>` … `</sfz-extra-data>` | recovery payload, appended placement (§5.2); 24 base64 characters for this archive |
|
|
1813
|
-
| 123063 | `</body></html>` | end tags; end of file at 123077 |
|
|
1814
|
-
|
|
1815
|
-
The appended run is 74 bytes, well inside the 16361-byte default budget (§5.2). The ZIP region
|
|
1816
|
-
is the 998 bytes from 122005 to 123003; the universal extractor reproduces the first 996
|
|
1817
|
-
of them and supplies the last two itself (§1.3).
|
|
1818
|
-
|
|
1819
|
-
### 8.3 Specimens
|
|
1820
|
-
|
|
1821
|
-
Generated with the command-line client running `single-file-core` against
|
|
1822
|
-
`example.com`. The anatomy of §8.2 was regenerated against 1.5.108; the compatibility
|
|
1823
|
-
results of §8.1 were measured on the 1.5.107 build of the same specimen set, which
|
|
1824
|
-
differs only inside the prologue and so falls in the same classes: those are grouped
|
|
1825
|
-
by whether bytes precede the archive and follow the EOCD, which no prologue change
|
|
1826
|
-
alters. The declared-form results are the exception, `declareAppendedData` being later
|
|
1827
|
-
than that build (§8.5); they were measured separately on a build that has it. The
|
|
1828
|
-
specimen names carry a `.sfz.html` suffix chosen for the harness; the conventions of
|
|
1829
|
-
§2.2 are what the clients produce, not what these files are called.
|
|
1830
|
-
`--compress-content` makes the output an archive; `extract-data-from-page`
|
|
1831
|
-
defaults to true there, so the plain variant has to switch it off:
|
|
1832
|
-
|
|
1833
|
-
| Specimen | Command |
|
|
1834
|
-
|---|---|
|
|
1835
|
-
| pure zip | `single-file --compress-content --self-extracting-archive=false <url> pure.zip` |
|
|
1836
|
-
| plain | `single-file --compress-content --extract-data-from-page=false <url> plain.sfz.html` |
|
|
1837
|
-
| universal | `single-file --compress-content <url> universal.sfz.html` |
|
|
1838
|
-
| relocated | add `--prevent-appended-data` |
|
|
1839
|
-
| pdf | add `--embed-pdf` |
|
|
1840
|
-
| png | add `--embed-screenshot` |
|
|
1841
|
-
| png-pdf | add `--embed-screenshot --embed-pdf` |
|
|
1842
|
-
| zip-pdf, zip-png, zip-png-pdf | add `--self-extracting-archive=false` to the pdf, png and png-pdf rows |
|
|
1843
|
-
| password | add `--password=<password>` |
|
|
1844
|
-
|
|
1845
|
-
These specimens are deliberately small, and a reader tested only against them is
|
|
1846
|
-
undertested: they are all flat archives of two or three entries. None
|
|
1847
|
-
exercises a root directory, `frames/<n>/` nesting, a second `index.html`, a `data:`-URL
|
|
1848
|
-
entry comment, the optional text body (§4.6), a UTF-8 BOM, zip64
|
|
1849
|
-
(§5.7), a payload past the appended-data budget, or a relocated reservation with padding left
|
|
1850
|
-
in it. Two omissions matter more than the rest, because they are the parts of §5.1 a
|
|
1851
|
-
writer is most likely to get wrong: no specimen defeats a rung by its **start**
|
|
1852
|
-
pattern, and none defeats one with an **upper-case** pattern. A writer that tested only
|
|
1853
|
-
end patterns, or matched them case-sensitively, produces every specimen here unchanged.
|
|
1854
|
-
|
|
1855
|
-
Two specimens cannot be produced from a URL alone. The **ladder** specimen, which
|
|
1856
|
-
forces the second rung of §5.1, needs a page referencing an image whose stored bytes
|
|
1857
|
-
contain `-->`; the archive then wraps in `<script type=sfz-data>`. The **zip64** specimen requires
|
|
1858
|
-
an archive past the thresholds of §5.7, so it is produced by calling the writer
|
|
1859
|
-
directly with zip64 forced on the ZIP writer, as `test/sfz-harness/zip64.js` does.
|
|
1860
|
-
|
|
1861
|
-
The measurements quoted elsewhere in this document come from the same harness: the
|
|
1862
|
-
payload growth rate of §5.2 (86 KB → 181 bytes, 283 KB → 465, 1.07 MB → 1645, 4.2 MB
|
|
1863
|
-
→ 6497, i.e. one byte of element per 650 bytes of archive) and the zip64 verification
|
|
1864
|
-
of §5.7.
|
|
1865
|
-
|
|
1866
|
-
### 8.4 The charset round trip, measured
|
|
1867
|
-
|
|
1868
|
-
Two claims of §2.1 were verified. The first is re-derived on every run by
|
|
1869
|
-
`test/sfz-harness/charset-round-trip.js`, which reads the tables below out of the
|
|
1870
|
-
runtime's own decoders rather than trusting this section, and checks the reverse table
|
|
1871
|
-
the extractor ships against the one the rule of §5.5 produces.
|
|
1872
|
-
|
|
1873
|
-
**Which encodings qualify.** Decoding all 256 byte values through each encoding
|
|
1874
|
-
defined by the WHATWG standard shows 20 that are injective and never produce U+FFFD:
|
|
1875
|
-
`windows-1252` (and its `iso-8859-1` labels), `iso-8859-2`, `-4`, `-5`, `-10`, `-13`,
|
|
1876
|
-
`-14`, `-15`, `-16`, `koi8-r`, `koi8-u`, `macintosh`, `windows-1250`, `-1251`,
|
|
1877
|
-
`-1254`, `-1256`, `-1258`, `x-mac-cyrillic`, `ibm866` and `x-user-defined`. The last of
|
|
1878
|
-
those qualifies on the criterion but is a poor choice in practice: it maps 0x80–0xFF
|
|
1879
|
-
into the Private Use Area, U+F780–U+F7FF, so the payload's characters have no meaning
|
|
1880
|
-
outside this round trip and any tool that touches the text sees private-use code points.
|
|
1881
|
-
The remaining single-byte encodings have undefined positions in their index —
|
|
1882
|
-
`iso-8859-3`, `-6`, `-7`, `-8`, `windows-874`, `-1253`, `-1255`, `-1257` — and the
|
|
1883
|
-
multi-byte ones (`utf-8`, `utf-16le`, `utf-16be`, `gbk`, `gb18030`, `big5`, `euc-jp`,
|
|
1884
|
-
`shift_jis`, `euc-kr`, `iso-2022-jp`) decode a lone byte sequence to U+FFFD or to fewer
|
|
1885
|
-
than 256 characters.
|
|
1886
|
-
The reverse table each one needs ranges from 8 entries (`iso-8859-15`) to 128
|
|
1887
|
-
(`koi8-r`, `koi8-u`, `ibm866` and `x-user-defined`); windows-1252 needs 27.
|
|
1888
|
-
|
|
1889
|
-
**That the round trip is charset-independent.** The mechanism of §5.5 — parse, then
|
|
1890
|
-
re-encode with the reverse table, restoring newlines from the 2-bit codes and NUL from
|
|
1891
|
-
U+FFFD — was run in a browser on a 4364-byte payload containing every byte value, 49
|
|
1892
|
-
newlines covering LF, CR, CR LF and a trailing CR, and NUL bytes. It recovers the
|
|
1893
|
-
payload byte for byte under `windows-1252`, `iso-8859-15`, `iso-8859-5`, `koi8-r`,
|
|
1894
|
-
`ibm866` and `macintosh`, and fails under `utf-8`, as the injectivity requirement
|
|
1895
|
-
predicts.
|
|
1896
|
-
|
|
1897
|
-
### 8.5 Format history
|
|
1898
|
-
|
|
1899
|
-
| When | Change |
|
|
1900
|
-
|---|---|
|
|
1901
|
-
| before 2023 | The format originates in SingleFileZ, a separate extension pairing a self-extracting HTML page with a ZIP archive. The `data-sfz` and `<sfz-extra-data>` identifiers date from there (§1.3) |
|
|
1902
|
-
| October 2023 | SingleFileZ's core is merged into single-file-core; universal mode (`extractDataFromPage`) and the wrapper ladder arrive with it |
|
|
1903
|
-
| November 2023 | `preventAppendedData`: archives that end exactly at the EOCD record, with the payload relocated ahead of the archive |
|
|
1904
|
-
| January 2024 | The PNG face: a screenshot's chunks wrap the archive, the HTML riding in a `tEXt` chunk |
|
|
1905
|
-
| September 2024 | The PDF face: an embedded PDF document placed so its header falls in the scan window, first with the HTML face, then for archives without it |
|
|
1906
|
-
| August 2026 | The embedded PDF becomes the `page.pdf` ZIP entry, listed first, so ZIP tools see the document as an ordinary entry (core 1.5.93) |
|
|
1907
|
-
| August 2026 | Core 1.5.107: page-text extraction skips `page.pdf` (§4.5), the appended-data budget accounts for the PNG tail (§5.2), and archives served with an HTTP error status fall back to page-text extraction (§4.1) |
|
|
1908
|
-
| August 2026 | Core 1.5.108: `<noscript>` leaves the wrapper ladder (§5.1), the title is emitted as character references instead of being dropped in universal mode (§4.6), the doctype is capped when it would push `%PDF-` out of the scan window (§6.1), and password-protected archives withhold the entry comments, the title and the text body (§5.6) |
|
|
1909
|
-
| August 2026 | Core 1.5.108: the writer enforces the two remaining HTML restrictions on comment text — a payload may not end with `<!-`, and the PNG face may not open its comment with the `>` or `->` its chunk checksum lands on once in 256 archives, which closed the wrapper and left the image data to the parser (§5.1) |
|
|
1910
|
-
| August 2026 | Core 1.5.108: the wrapper ladder escalates to `<script type=sfz-data>` and `<style type=sfz-data>` before the raw-text rungs, the only two whose content local text extractors drop the way they drop a comment (§5.1) |
|
|
1911
|
-
| August 2026 | Core 1.5.108: the ZIP region carries the identifier `sfz-data` and the extractor addresses it with that instead of deducing it from its position beside `<sfz-extra-data>` (§4.5). This fixes universal extraction on the `<style type=sfz-data>` rung, where the reference extractor's own relocation of `style` elements into the head moved the region out from under the positional rule |
|
|
1912
|
-
| August 2026 | Core 1.5.108: the recovery payload stops two bytes short of the End Of Central Directory record, excluding its comment-length field (§1.3), which lets universal-mode archives declare their appended data as the archive comment — a writer option, for `java.util.zip` and the readers that reject undeclared trailing bytes (§4.2) |
|
|
1913
|
-
| August 2026 | Core 1.5.108: the PDF and PNG faces test a wrapper rung's start pattern as well as its end pattern, closing the same script-data escape hole the ZIP region was already guarded against — a face payload holding `<!--` and then `<script` took the `<script type=sfz-data>` rung and swallowed the rest of the document (§5.1) |
|
|
1914
|
-
| August 2026 | Core 1.5.108: the retry loop never discards a relocation reservation, so a payload sitting on the appended-data boundary cannot oscillate between the two placements forever (§6.2) |
|
|
1915
|
-
| August 2026 | Core 1.5.110: a PDF or PNG face whose payload names every rung is dropped instead of written bare (§5.1). Found by nesting an archive inside itself as both faces: the fifth level exhausts the ladder, and readers then extracted the fourth level's archive — checksums intact, no way to tell (§7.4) |
|
|
1916
|
-
| August 2026 | Core 1.5.110: a PNG face leaving the comment rung on its checksum resumes the rung search instead of taking the next rung untested (§5.1). Taking it put a payload holding `</script>` on the script rung, where its own bytes closed the wrapper 93 bytes in and left the image data, the chunk framing and the whole ZIP region to the parser |
|
|
1917
|
-
| August 2026 | Core 1.5.110: `<svg><![CDATA[` joins the ladder above `<plaintext>` (§5.1) — the one rung whose terminator, `]]>`, real payloads rarely carry. It gives a payload naming every element rung somewhere to go that does not cost the appended-data placement, and moves the self-nesting limit from the fifth level to the sixth |
|
|
1918
|
-
| August 2026 | Core 1.5.115: password-protected archives withhold the provenance comment and the canonical link as well (§5.6). Both wrote the page's own URL into the prologue, beside the title that was already withheld, so the address the archive was saved from stayed in the clear |
|
|
1919
|
-
| August 2026 | Core 1.5.119: the inlined ZIP library is built ASCII-only, and §2.1 now requires it of any bootstrap. Its CP437 table had been emitted as literal characters, which the page re-decoded as windows-1252, growing the table from 256 entries to 508 and shifting every lookup by 60 — so the one entry read without the UTF-8 flag, `page.pdf`, came back mangled and no archive with a PDF face extracted in any engine (§5.8) |
|
|
1920
|
-
| August 2026 | Core 1.5.120: the hand-built `page.pdf` records set the language encoding flag, like every entry the ZIP writer produces (§5.8). Its name is ASCII, so no decoded name changes; what changes is that no entry in an archive is read through CP437 any more, closing the path the 1.5.119 defect surfaced on |
|
|
1921
|
-
| September 2026 | §5.8 no longer requires bit 11 on every entry, deferring to ZIP's own rule: the flag is set when a name or a comment holds a byte outside printable ASCII, and left clear otherwise, because readers disagree about the flag more than they disagree about ASCII. The reference writer's names are all percent-encoded, so in practice none of them carries it now, and the hand-built `page.pdf` records follow the writer instead of overriding it — reversing the 1.5.120 row below, whose reason was that `page.pdf` would otherwise be the only entry read through the legacy path. It no longer is: every name in the archive takes the same path again, the other one |
|
|
1922
|
-
| September 2026 | Core 1.5.126: the appended-data budget becomes the `maxAppendedDataLength` writer option and its default drops from 65535 to 16361 bytes, so the EOCD record stays inside libarchive's scan and `bsdtar` opens archives it used to reject (§5.2, §8.1). The 65535-byte comment ceiling is now a separate limit, stated in §4.2: a budget raised past it produces a run that cannot be declared |
|
|
1923
|
-
|
|
1924
|
-
This document was itself revised in August 2026, against core 1.5.108, after several
|
|
1925
|
-
independent reviews. One of them was a reader built from this specification alone, with
|
|
1926
|
-
no access to the reference code. It read every specimen correctly, which is the best
|
|
1927
|
-
evidence available that the format is implementable from this text, and it found
|
|
1928
|
-
thirteen defects in the text. The load-bearing corrections are in §5.1 (the
|
|
1929
|
-
start-pattern test is necessary, not conservative: script data's escape states let a
|
|
1930
|
-
payload defeat the end-tag test), §5.5 (the WHATWG index is not the platform codec of
|
|
1931
|
-
the same name), §4.5 (the offset shift is derivable from the recovered region, and
|
|
1932
|
-
`page.pdf` is reachable, just not by offset) and §1.3 (the appended data may be a
|
|
1933
|
-
declared archive comment). A second pass by the same implementation caught a regression
|
|
1934
|
-
the revision had introduced: giving `<plaintext>` a start pattern would have let 55
|
|
1935
|
-
bytes of ASCII defeat all eight rungs, which the reference writer never did and which
|
|
1936
|
-
§6.2's termination argument forbids. Later passes added a normative order for resolving
|
|
1937
|
-
the page entry, since "the shallowest `index.html`" had no tiebreak (§7.1); the
|
|
1938
|
-
distinction between the logical archive and the ZIP region, which `page.pdf` is the one
|
|
1939
|
-
entry to fall outside (§1.3); the layer convention now stated at the head of the
|
|
1940
|
-
document; the limits of the reconstructed-`page.pdf` CRC check (§4.5); the durability
|
|
1941
|
-
ranking of the faces (§1.1); what each face costs a writer (§6); and the silent loss of
|
|
1942
|
-
the other faces to a pipeline that repacks the file (§7.2). One review found a live
|
|
1943
|
-
defect rather than a documentation one, the non-monotone retry step recorded above.
|
|
1944
|
-
|
|
1945
|
-
A later pass found four places where the document contradicted itself or the standard
|
|
1946
|
-
it cites: §7.3 stated that source URLs stay readable under a password while §5.6 said
|
|
1947
|
-
the writer withholds them, §4.5 called the recovered region exact while excluding two
|
|
1948
|
-
bytes from it, §7.4 rejected a duplicate identifier that §4.5 resolves by tie-break,
|
|
1949
|
-
and §2.1 described the HTML encoding prescan as mandatory and 1024 bytes wide when the
|
|
1950
|
-
standard makes it optional and only encourages that bound. None of the four changes
|
|
1951
|
-
what a writer emits or a reader accepts.
|
|
1952
|
-
|
|
1953
|
-
The same pass added the two boundaries universal mode had left unstated: that it
|
|
1954
|
-
recovers the region only where the declared charset is what decided the decoding, a
|
|
1955
|
-
BOM, a user override and a transport-layer charset all outranking it — narrow in
|
|
1956
|
-
practice, since the raw read comes first and no encoding applies to it (§2.1) — and that
|
|
1957
|
-
the recovery payload's 32-bit length field caps the region below 2^32 bytes, with
|
|
1958
|
-
engine string limits binding well before that (§5.5).
|
|
1959
|
-
|
|
1960
|
-
A pass in September 2026, against core 1.5.120, read the text alone first and then
|
|
1961
|
-
checked each open question against the writer. It corrected two statements about the
|
|
1962
|
-
reference writer that the code contradicted: the retry loop never discards a
|
|
1963
|
-
reservation, and the archive time is read once per build, not once per pass (§6.2).
|
|
1964
|
-
It added what only the code could say: the PNG build steps that §6.1 had skipped, the
|
|
1965
|
-
fields patched after the wrapper check and the size below which they are harmless
|
|
1966
|
-
(§5.1), the chunks the PNG face copies (§3.1), the per-frame manifests and the root
|
|
1967
|
-
directory's name (§7.1), the `page.pdf` header fields (§6.1), and the range-reading
|
|
1968
|
-
failure path (§4.1).
|