single-file-core 1.5.119 → 1.5.120
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/core/index.js +4 -0
- package/doc/singlefile-archive.md +323 -135
- package/eslint.config.mjs +6 -0
- package/package.json +2 -2
- package/processors/compression/compression-display.js +0 -11
- package/processors/compression/compression-extract.js +0 -4
- package/processors/compression/compression-packager.js +0 -12
- package/processors/compression/compression-router.js +0 -35
- package/processors/compression/compression.js +73 -176
- package/processors/hooks/content/content-hooks-frames-web.js +2 -1
- package/processors/lazy/content/content-lazy-loader.js +21 -18
- package/test/sfz-harness/README.md +10 -1
- package/test/sfz-harness/byte-map.js +137 -0
- package/test/sfz-harness/format-rules.js +132 -2
- package/test/sfz-harness/option-wiring.js +0 -3
- package/test/sfz-harness/zip64.js +77 -0
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
**Status: draft.** This document specifies the SingleFile archive, the polyglot file
|
|
4
4
|
format produced by [SingleFile](https://github.com/gildas-lormeau/SingleFile) when it
|
|
5
5
|
saves a page as a ZIP archive. It is written against the reference implementation,
|
|
6
|
-
[single-file-core](https://github.com/gildas-lormeau/single-file-core) 1.5.
|
|
6
|
+
[single-file-core](https://github.com/gildas-lormeau/single-file-core) 1.5.119
|
|
7
7
|
(`processors/compression/`), and every byte-level statement has been verified on
|
|
8
8
|
generated specimen files.
|
|
9
9
|
|
|
@@ -103,13 +103,15 @@ Three consequences shape everything below:
|
|
|
103
103
|
|
|
104
104
|
### 1.2 Non-goals
|
|
105
105
|
|
|
106
|
-
- **Forward-only ZIP parsers.**
|
|
107
|
-
entries are preceded by non-ZIP bytes
|
|
108
|
-
offset 0
|
|
106
|
+
- **Forward-only ZIP parsers.** Every archive with a face requires central-directory-driven
|
|
107
|
+
reading, since the entries are then preceded by non-ZIP bytes. The variant with no other
|
|
108
|
+
face is an ordinary ZIP file and streams from offset 0 like any other (§8.1, class A);
|
|
109
|
+
parsers that require that are out of scope for the rest (§7).
|
|
109
110
|
- **In-place modification by generic ZIP tools.** The face invariants are global:
|
|
110
111
|
the writer picks each hiding tag only after checking the exact bytes it must hide,
|
|
111
|
-
the recovery payload of universal mode contains a checksum of the
|
|
112
|
-
|
|
112
|
+
the recovery payload of universal mode contains a checksum of the ZIP region
|
|
113
|
+
without its comment-length field, and the PDF and PNG structures wrap the
|
|
114
|
+
archive (§5.4). A tool
|
|
113
115
|
that adds, removes or recompresses entries invalidates them, and most rewriters drop
|
|
114
116
|
the prepended and appended regions outright. A generically rewritten file keeps at
|
|
115
117
|
best its ZIP face. Editing an archive means producing a new one through the writer
|
|
@@ -133,12 +135,12 @@ Three consequences shape everything below:
|
|
|
133
135
|
| **universal mode** | The variant whose HTML face can extract the archive from the *parsed page text*, the text and comment nodes the HTML parser produced, and therefore needs no access to its own raw bytes. Named "universal" because it works from any location, including the `file:` protocol. |
|
|
134
136
|
| **wrapper tag** | The HTML construct that hides a binary region from the HTML parser, `<!--`…`-->` by default (§5.1). |
|
|
135
137
|
| **appended data** | Bytes after the ZIP End Of Central Directory record. Readers tolerate them within the window their EOCD scan already covers: 65557 bytes from the end of the file (the 22-byte record plus the 65535-byte maximum comment length); "the 64 KB window" refers to this. It may be left undeclared or declared as the archive comment; both forms are valid ZIP and readers MUST accept both (§4.2). The recovery payload can be computed before that choice is made because it stops two bytes short of the record, excluding its comment-length field (see *recovered range* below). |
|
|
136
|
-
| **ZIP region** | The contiguous byte range holding the archive proper: from the first local file header the ZIP writer emitted through the last byte of the End Of Central Directory record. It spans the `zip-entries`, `pdf-central-record` (when present) and `central-directory · eocd` blocks of §3, and in the HTML variants it is
|
|
138
|
+
| **ZIP region** | The contiguous byte range holding the archive proper: from the first local file header the ZIP writer emitted through the last byte of the End Of Central Directory record. It spans the `zip-entries`, `pdf-central-record` (when present) and `central-directory · eocd` blocks of §3, and in the HTML variants it is the content of the last wrapper, exactly so on the element rungs and preceded by the `sfz-data` identifier on the comment rung, which the extractor steps over. It does **not** include `pdf-local-header` or the PDF document, which sit earlier in the file. |
|
|
137
139
|
| **archive** | The *logical* ZIP file: the set of entries the central directory describes, wherever their bytes lie. This is distinct from the ZIP region above, which is a contiguous byte range. Every entry but one has its bytes inside the region; `page.pdf` is the deliberate exception, an entry of the archive whose local header and data sit before the region (§4.2). "Archive" in this document always means the logical file, "ZIP region" always the byte range, and the two differ only in the PDF-with-HTML variants. |
|
|
138
140
|
| **recovered range** | What the universal extractor reproduces (§4.5): the ZIP region minus its last two bytes, the comment-length field of the End Of Central Directory record. That field is the one part of the record whose value depends on what follows the region, so leaving it out is what lets a writer decide the appended-data form after the recovery payload is final (§4.2). The extractor supplies the two bytes itself, as zeroes — the recovered range carries no comment. |
|
|
139
141
|
| **reference writer** | `createArchive()` in single-file-core `processors/compression/compression.js`. |
|
|
140
142
|
| **bootstrap** | The inline script in the HTML face that locates, extracts and displays the archived page. |
|
|
141
|
-
| **`sfz` identifiers** | Three byte-level identifiers
|
|
143
|
+
| **`sfz` identifiers** | Three byte-level identifiers carrying the `sfz` prefix matter to this document: `data-sfz`, `<sfz-extra-data>` and `sfz-data`. A conforming file may hold others the format says nothing about — the reference bootstrap gives its own status messages `sfz`-prefixed ids, which no reader has any reason to look for. The prefix is inherited from SingleFileZ, the browser extension the format originated in (since merged into SingleFile), and is kept unchanged for compatibility with existing files. They are wire identifiers, not the format's name. Two of the three have a role in the format: `<sfz-extra-data>` is the element carrying the recovery payload, and `sfz-data` is the identifier the universal extractor addresses the ZIP region with — an `id` attribute on the wrapper element, or the first characters of the wrapper comment's data (§4.5). `data-sfz` is a marker the reference writer happens to put on the root element; this document mentions it only where it describes bytes those files contain. |
|
|
142
144
|
|
|
143
145
|
## 2. Variants: composing faces
|
|
144
146
|
|
|
@@ -164,8 +166,9 @@ include it because the clients producing those variants enable it by default, bu
|
|
|
164
166
|
`embeddedImage` and `embeddedPdf` compose with a non-universal self-extracting file
|
|
165
167
|
just as well; the extension is then `.zip.html`. The one interaction: `page.pdf` lies
|
|
166
168
|
outside the ZIP region (§1.3), so the page-text extraction path does not recover it
|
|
167
|
-
with the rest
|
|
168
|
-
|
|
169
|
+
with the rest. The extractor therefore filters that entry out unconditionally, on
|
|
170
|
+
every acquisition path including the ones that read raw bytes and could return it
|
|
171
|
+
(§4.5). In the last three rows there is no
|
|
169
172
|
HTML face, so the option does not apply. The *Specimen* column names the measured
|
|
170
173
|
reference files this document cites; §8 records how to regenerate them.
|
|
171
174
|
|
|
@@ -267,12 +270,39 @@ or overridden by a server still tends to be decoded the way the extractor expect
|
|
|
267
270
|
also keeps the reverse table small, at 27 entries (§5.5).
|
|
268
271
|
|
|
269
272
|
The second part is the extra-data payload, which does **not**
|
|
270
|
-
contain the archive: it carries only what the round trip destroys
|
|
271
|
-
|
|
273
|
+
contain the archive: it carries only what the round trip destroys or leaves
|
|
274
|
+
undetermined, namely a checksum, the recovered range's length, and the information
|
|
275
|
+
needed to restore newline bytes, which the parser normalizes.
|
|
272
276
|
The parser also replaces NUL bytes with U+FFFD; since no byte decodes to U+FFFD under
|
|
273
277
|
a qualifying encoding, the extractor maps U+FFFD back to NUL unambiguously and the
|
|
274
278
|
payload needs nothing for it (§5.5).
|
|
275
279
|
|
|
280
|
+
The declared charset governs the **whole document**, not only the regions the format
|
|
281
|
+
reasons about. Everything the parser reads is decoded with it, the bootstrap script
|
|
282
|
+
included, and the writer's own code is therefore subject to the same single-byte
|
|
283
|
+
decoding as the page it carries. In universal mode the bootstrap MUST contain no
|
|
284
|
+
character outside printable ASCII.
|
|
285
|
+
|
|
286
|
+
Unlike the `<title>` and the table of contents (§4.6), it cannot be rescued by
|
|
287
|
+
character references. A `<script>` element's content is script data, a tokenizer state
|
|
288
|
+
that does not resolve them: `☺` written there stays seven literal characters and
|
|
289
|
+
reaches the program as seven characters. The escape has to happen one level down, in
|
|
290
|
+
the JavaScript source — `\u263A` rather than `☺`, an escape the language resolves when
|
|
291
|
+
the script is compiled, not one the HTML parser resolves when the file is read. A
|
|
292
|
+
minifier will undo this if allowed to, since printing the shortest form is its default
|
|
293
|
+
and the shortest form of `\u263A` is the literal character. A writer that assembles
|
|
294
|
+
the bootstrap through a minifier MUST configure it to emit ASCII only.
|
|
295
|
+
|
|
296
|
+
The consequence of getting this wrong is worse than the mojibake a raw title produces,
|
|
297
|
+
and that is the reason for the MUST. A garbled title is visible; a garbled string
|
|
298
|
+
inside the extractor is not. A lookup table is the sharpest case. Emitted as literal
|
|
299
|
+
characters, a CP437 table is re-decoded as windows-1252 and grows from 256 entries to
|
|
300
|
+
508, shifting every lookup past the first 32 by 60 positions — and nothing about the
|
|
301
|
+
page looks wrong, because the damage is confined to names the table decodes. Under
|
|
302
|
+
§5.8 that can be a single entry, and one is enough when it is the entry the extractor
|
|
303
|
+
matches by name. The requirement is on the whole bootstrap rather than on any table
|
|
304
|
+
inside it, because a minifier does not know which strings are load-bearing.
|
|
305
|
+
|
|
276
306
|
### 2.2 File name conventions
|
|
277
307
|
|
|
278
308
|
The reference implementation names files by variant: `.zip` (no HTML face),
|
|
@@ -309,21 +339,21 @@ face adds, then the regions the PNG face adds.
|
|
|
309
339
|
|
|
310
340
|
| Region | Producer | Present | Contents |
|
|
311
341
|
|---|---|---|---|
|
|
312
|
-
| `html-prologue` | HTML | HTML face | Doctype, the root element start tag, an optional implementation-defined comment,
|
|
313
|
-
| `bootstrap` | HTML | HTML face | One inline `<script>`: the embedded ZIP reader, the extractor, the display routine, and the content-acquisition logic (§4.1). The wrapper start tag that opens the ZIP region follows it, directly or after a relocated `extra-data`. |
|
|
314
|
-
| `<!--` / `-->` | HTML | HTML face | The wrapper tag pair hiding a binary region from the HTML parser — comment tags by default, another pair when the hidden bytes
|
|
342
|
+
| `html-prologue` | HTML | HTML face | Doctype, the root element start tag, `<meta charset>`, an optional implementation-defined comment, title, optional head elements (canonical link, `robots` meta, viewport, Content-Security-Policy), minimal CSS, `<body hidden>`, wait/error messages, optional table of contents, optional text body (§4.6). The leading comment, the title, the canonical link and the text body are withheld when a password is set (§5.6). In the plain variant an optional UTF-8 BOM MAY precede the doctype (`includeBOM`); universal and PNG variants never carry one. In the PNG variants the region is split: everything through `<body hidden>` is the data of the `tEXt "PNG"` chunk, while the messages, the optional table of contents and the optional text body follow the `tEXt "ZIP"` chunk header; the doctype and the leading comment are dropped. |
|
|
343
|
+
| `bootstrap` | HTML | HTML face | One inline `<script>`: the embedded ZIP reader, the extractor, the display routine, and the content-acquisition logic (§4.1). In universal mode its bytes MUST be pure ASCII, since the declared charset decodes this region like any other and character references do not apply inside script data (§2.1). The wrapper start tag that opens the ZIP region follows it, directly or after a relocated `extra-data`. |
|
|
344
|
+
| `<!--` / `-->` | HTML | HTML face | The wrapper tag pair hiding a binary region from the HTML parser — comment tags by default, another pair when the hidden bytes defeat them — which `-->` is only the commonest way to do, the full test being `<!--`, `--!>` or a trailing `<!-` (§5.1). Drawn at each opening and closing position. The close tag is absent when appended data is prevented (`preventAppendedData`, or the `<plaintext>` wrapper which cannot close): no markup follows the archive and the wrapper runs to end-of-file. That does not mean the file ends at the EOCD — the PNG face's tail still follows, inside the wrapper, where it parses as text (§5.1). |
|
|
315
345
|
| `zip-entries` | ZIP | always | The archive's local file headers and entry data, written by the ZIP writer. The central directory of an archive written by the reference writer lists `index.html` (the page) first, then `manifest.json` (a JSON description of the archive: original URL, title, save time, resource-to-URL map — informative; the page displays without it), then the resources; the *physical* order of the local headers inside the region is not guaranteed to match, and readers MUST NOT rely on either order — entries are addressed by name (§7.1). |
|
|
316
346
|
| `central-directory · eocd` | ZIP | always | The central-directory records followed by the End Of Central Directory record. All offsets are absolute file positions (§5.3). In the HTML+PDF variants the EOCD accounts for the injected `pdf-central-record` (how the writer achieves that is §6). |
|
|
317
347
|
| `extra-data` | extractor | universal | `<sfz-extra-data>` element holding the base64, deflate-compressed recovery payload (§5.5). It always sits outside the wrapper, so it parses as a real element the extractor can address. Normal placement: after the EOCD, between the wrapper close tag and the end tags. Relocated placement, used when the payload exceeds the 64 KB appended-data window or `preventAppendedData` is set: immediately before the wrapper start tag. In the relocated form the element is followed by space padding: its room is reserved before the archive is written, because the region precedes the ZIP data and resizing it would shift every central-directory offset (§6). Neither placement carries positional meaning — the extractor finds the ZIP region by identifier, not relative to this element (§4.5). |
|
|
318
348
|
| `</body></html>` | HTML | HTML face | The end tags closing the document after the wrapper close tag. Omitted when appended data is prevented, and in the PNG variants so the file can end with the PNG tail. |
|
|
319
|
-
| `pdf-local-header` | ZIP | PDF face with HTML | The hand-built local file header for `page.pdf` (STORE, checksum precomputed), written immediately before the PDF document so ZIP readers see an ordinary entry whose data is the PDF (§6). |
|
|
320
|
-
| `pdf-document` | PDF | PDF face | The raw PDF bytes. With the HTML face, wrapped together with `pdf-local-header` in a wrapper tag pair inside `html-prologue`, placed so `%PDF-` starts at offset 1024 or lower — the range PDF readers search for the header, which is what lets a PDF document start after other bytes at all (§4.3). Without the HTML face the file simply *starts* with the PDF document, as prepended data the ZIP face tolerates; `page.pdf` is then not an archive entry at all — no local header, no central record. |
|
|
349
|
+
| `pdf-local-header` | ZIP | PDF face with HTML | The hand-built local file header for `page.pdf` (STORE, checksum precomputed, language encoding flag set as on every other entry — §5.8), written immediately before the PDF document so ZIP readers see an ordinary entry whose data is the PDF (§6). |
|
|
350
|
+
| `pdf-document` | PDF | PDF face | The raw PDF bytes. With the HTML face, wrapped together with `pdf-local-header` in a wrapper tag pair inside `html-prologue`, placed so `%PDF-` starts at offset 1024 or lower — the range PDF readers search for the header, which is what lets a PDF document start after other bytes at all (§4.3). Without the HTML face and without the PNG face, the file simply *starts* with the PDF document, as prepended data the ZIP face tolerates; `page.pdf` is then not an archive entry at all — no local header, no central record. |
|
|
321
351
|
| `pdf-central-record` | ZIP | PDF face with HTML | The central-directory record for `page.pdf`, injected *before* the writer's own central directory. The start of the central directory is the one place a record can be added without moving any offset the writer already committed, and it makes `page.pdf` the first entry ZIP tools list (§6). |
|
|
322
352
|
| `png-signature · IHDR` | PNG | PNG face | The 8-byte PNG signature and the `IHDR` chunk declaring the source image's dimensions — the first 33 bytes of the file. |
|
|
323
353
|
| `tEXt "PNG"` | PNG | PNG face with HTML | The length, type and keyword bytes of the first `tEXt` chunk. Its data is `html-prologue` (with the PDF face, the embedded PDF document rides inside it too), ending with the wrapper start tag. |
|
|
324
354
|
| `tEXt "PDF"` | PNG | PNG + PDF faces without HTML | The length, type and keyword bytes of a `tEXt` chunk whose data is the raw PDF document. Written only when the PNG and PDF faces combine without HTML — with the HTML face the PDF rides inside `tEXt "PNG"` instead — and placed right after `IHDR` so `%PDF-` stays within the header scan window (§4.3). |
|
|
325
355
|
| `pixel-data chunks` | PNG | PNG face | The source image's image-data chunks, copied unmodified. With the HTML face they sit inside the wrapper so the HTML parser skips them. |
|
|
326
|
-
| `tEXt "ZIP"` | PNG | PNG face | The length, type and keyword bytes of the
|
|
356
|
+
| `tEXt "ZIP"` | PNG | PNG face | The length, type and keyword bytes of the archive's own `tEXt` chunk — the second one when the HTML face or the PDF face put a chunk ahead of it, the only one otherwise. Its declared length covers everything from there up to but not including the trailing chunk CRC, as a PNG chunk length always does, so the PNG decoder skips the archive — and, with the HTML face, the bootstrap and the appended data — as the data of one chunk. With the HTML face, the wrapper opened at the end of `tEXt "PNG"` closes immediately after these bytes: its content is the first chunk's CRC, the pixel-data chunks and this chunk's own header, and the prologue resumes as markup directly after the close tag. |
|
|
327
357
|
| `crc · IEND` | PNG | PNG face | The `tEXt "ZIP"` chunk's CRC, computed once the archive bytes are final (§6), followed by the empty `IEND` chunk — the last bytes of the file (PNG requires `IEND` to end the stream, which is why the PNG variants drop the end tags). |
|
|
328
358
|
|
|
329
359
|
The reader-by-reader interpretation of these regions is §4; the mechanics that keep
|
|
@@ -381,8 +411,8 @@ The bootstrap script runs at parse time and proceeds in three stages:
|
|
|
381
411
|
range reading, fetching only the central directory and the entries it needs (a
|
|
382
412
|
large archive displays without downloading the ZIP region in full); otherwise it
|
|
383
413
|
downloads the whole file. When the header probe fails it falls back to page-text
|
|
384
|
-
extraction; a failure of the full download itself, past the probe,
|
|
385
|
-
|
|
414
|
+
extraction; so does a failure of the full download itself, past the probe, which
|
|
415
|
+
is why the probe leaves the document in place. Only when every applicable rung fails does the error
|
|
386
416
|
message appear, with recovery instructions that differ by variant (§2).
|
|
387
417
|
2. **Extract.** The embedded ZIP reader reads the archive through the ZIP lens
|
|
388
418
|
(§4.2) and rebuilds the page: text entries are decoded, binary entries become
|
|
@@ -461,7 +491,7 @@ both. **Raw** — the EOCD declares a zero-length comment and the trailing bytes
|
|
|
461
491
|
simply outside the archive — is the default, because tools print a declared archive
|
|
462
492
|
comment on ordinary operations (§8.1), and in universal mode that comment is the
|
|
463
493
|
whole base64 recovery payload. **Declared** — the EOCD's comment length covers every
|
|
464
|
-
byte after the record — is
|
|
494
|
+
byte after the record — is the later addition, and it is the only form some
|
|
465
495
|
readers accept at all: `java.util.zip`, and therefore Android and most JVM tooling,
|
|
466
496
|
rejects an archive with undeclared trailing bytes outright (§8.1). A writer SHOULD
|
|
467
497
|
offer both and default to raw.
|
|
@@ -495,9 +525,10 @@ compatibility appendix records real-world support (§8):
|
|
|
495
525
|
the objects themselves.
|
|
496
526
|
|
|
497
527
|
With the HTML face, the PDF document sits inside the head, wrapped in its own
|
|
498
|
-
wrapper-tag pair chosen against the
|
|
499
|
-
ladder,
|
|
500
|
-
|
|
528
|
+
wrapper-tag pair chosen against the local header and the PDF together (§5.1) — a PDF
|
|
529
|
+
containing `-->` steps the ladder, and so can the header, whose CRC-32 and size fields
|
|
530
|
+
hold arbitrary bytes — and preceded by the `page.pdf` local header so the same bytes
|
|
531
|
+
are also a ZIP entry. That dual role is why the entry MUST be STOREd and MUST NOT be
|
|
501
532
|
encrypted: a viewer reads the entry's data region directly, and any transformation
|
|
502
533
|
of it would break the face. Without the HTML face the document needs no wrapper, and
|
|
503
534
|
where it sits depends on the PNG face: alone with the ZIP face it simply starts the
|
|
@@ -525,9 +556,12 @@ chunks (ancillary by construction, their type starting lowercase):
|
|
|
525
556
|
ZIP region and the appended data. The decoder hops over all of it as the data of
|
|
526
557
|
one chunk.
|
|
527
558
|
|
|
528
|
-
|
|
529
|
-
string is Latin-1 text, and
|
|
530
|
-
`
|
|
559
|
+
The `tEXt "ZIP"` chunk carries text the PNG standard does not strictly permit: a
|
|
560
|
+
`tEXt` text string is Latin-1 text, and its payload contains NUL bytes — 78 in the
|
|
561
|
+
`png` specimen, 101 in the `png-pdf` one. The first chunk is pure printable ASCII in
|
|
562
|
+
the plain `png` specimen, where the doctype and the provenance comment are suppressed
|
|
563
|
+
and the title is escaped to character references; only the `-pdf` variants put NULs
|
|
564
|
+
in it. Decoders skip
|
|
531
565
|
ancillary chunks without inspecting their text, so this passes everywhere tested
|
|
532
566
|
(§8.1). It exercises the PNG tolerance §1.1 lists at its limit: what decoders ignore
|
|
533
567
|
in a `tEXt` chunk is text PNG does not permit.
|
|
@@ -635,7 +669,10 @@ entry out on every acquisition path, including the ones that read raw bytes and
|
|
|
635
669
|
return it.
|
|
636
670
|
|
|
637
671
|
The extractor MUST verify the three checkable payload fields — byte length, newline
|
|
638
|
-
count and checksum — and fail to the error message on any mismatch.
|
|
672
|
+
count and checksum — and fail to the error message on any mismatch. It MUST also fail
|
|
673
|
+
on the unassigned newline code of step 2 rather than decode it, so that a payload
|
|
674
|
+
written against a later revision of the format is named as unsupported instead of
|
|
675
|
+
silently reconstructing the wrong bytes.
|
|
639
676
|
|
|
640
677
|
The recovered region is a complete archive but **not an offset-self-contained one**.
|
|
641
678
|
Its offsets are still absolute positions in the original file (§5.3), so every
|
|
@@ -649,7 +686,11 @@ zipfile`, adds `(attempting to process anyway)` and lists both entries, while re
|
|
|
649
686
|
that compensate silently, such as Python's `zipfile`, show no diagnostic at all. The
|
|
650
687
|
shift also puts `page.pdf` out of the
|
|
651
688
|
offset-following path: its local header lies *before* the region, so its compensated
|
|
652
|
-
offset is negative
|
|
689
|
+
offset is negative and no reader can seek to it. That offset is the header's own
|
|
690
|
+
position — which §4.3 keeps inside the first 1024 bytes — minus the region's start,
|
|
691
|
+
which lies past the whole bootstrap, so it is negative for every archive and its
|
|
692
|
+
magnitude is essentially the region's start, so it moves with the size of the inlined
|
|
693
|
+
ZIP library and no particular value should be read into it. On
|
|
653
694
|
success the shifted bytes enter the normal extraction path (§4.2).
|
|
654
695
|
|
|
655
696
|
The shift is a file offset, and a universal-mode reader has no file. It does not need
|
|
@@ -665,10 +706,13 @@ where `eocdPosition` is the EOCD record's own offset within `region`, found by s
|
|
|
665
706
|
backward for its signature the way any ZIP reader finds it. A reader arrives at the
|
|
666
707
|
same number as a ZIP library's prepended-data compensation, which derives it from the
|
|
667
708
|
record's position rather than from the buffer's end. Do not substitute
|
|
668
|
-
`region.length - 22` for `eocdPosition
|
|
669
|
-
|
|
709
|
+
`region.length - 22` for `eocdPosition`. The two are in fact equal for every recovered
|
|
710
|
+
region, which always ends at the EOCD's last byte and declares a zero-length comment
|
|
711
|
+
(§1.3), but the habit fails the moment the same code is pointed at a file rather than a
|
|
712
|
+
recovered region: there a non-empty archive comment puts bytes after the record. Zip64
|
|
713
|
+
does not — its records precede the EOCD, which stays last.
|
|
670
714
|
|
|
671
|
-
Under zip64 (§5.
|
|
715
|
+
Under zip64 (§5.7) both of those EOCD fields are the `0xFFFFFFFF` sentinel, and the
|
|
672
716
|
zip64 end of central directory record carries the real values. Take them from there,
|
|
673
717
|
using the same `eocdPosition` arithmetic against that record's own position — the
|
|
674
718
|
zip64 locator states an absolute offset in the original file, so it needs the shift
|
|
@@ -689,18 +733,46 @@ universal mode this cuts its audience in two: a charset-oblivious tool that read
|
|
|
689
733
|
raw bytes — `grep`, plain-text search — sees intact UTF-8, while any consumer that
|
|
690
734
|
honors the declared `<meta charset>` decodes it as windows-1252 and garbles
|
|
691
735
|
non-ASCII text. That includes the HTML parser itself — harmless there, because the
|
|
692
|
-
region is hidden and replaced (§4.1) — but also HTML-aware indexers.
|
|
736
|
+
region is hidden and replaced (§4.1) — but also HTML-aware indexers. In universal mode the text body
|
|
693
737
|
opens with the page title, for the same raw-byte
|
|
694
738
|
audience — it is the first *text* in the element, which is not necessarily the
|
|
695
739
|
element's first line: the reference writer's serialization puts a newline before it.
|
|
696
|
-
|
|
697
|
-
|
|
698
|
-
|
|
699
|
-
|
|
700
|
-
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
740
|
+
Outside universal mode the title is not repeated there, since the prologue's own
|
|
741
|
+
`<title>` is already readable as bytes.
|
|
742
|
+
|
|
743
|
+
The `<title>` element takes the opposite route, and so does every other piece of
|
|
744
|
+
prologue text the writer assembles itself. Character references are resolved against
|
|
745
|
+
Unicode independently of the declared encoding — in RCDATA, where the title's content
|
|
746
|
+
sits, in ordinary element text, and in attribute values alike — so the writer emits
|
|
747
|
+
every character outside printable ASCII, along with `&`, `<`, `>` and `"`, as a
|
|
748
|
+
numeric reference. Those bytes are therefore pure ASCII and the text survives the
|
|
749
|
+
single-byte declaration intact: a page titled 日本語 shows as 日本語 in the browser tab
|
|
750
|
+
and to any conforming parser. The reference writer passes the title and the multi-page
|
|
751
|
+
table of contents — its link text and its `href` values, which is what the `"` is for —
|
|
752
|
+
through one shared escaper. Writers that emit such text raw MUST NOT do so in
|
|
753
|
+
universal mode, where the same bytes decode as mojibake.
|
|
754
|
+
|
|
755
|
+
The text body above is the deliberate exception, not an oversight: it is left as raw
|
|
756
|
+
UTF-8 because its audience reads bytes rather than parsed text. The bootstrap script
|
|
757
|
+
is the other region outside this rule, and it is outside for a harder reason — script
|
|
758
|
+
data does not resolve character references at all, so the escape must happen in the
|
|
759
|
+
JavaScript source instead (§2.1).
|
|
760
|
+
|
|
761
|
+
An implementation-defined comment (§3.1) is the third region outside the rule, and the
|
|
762
|
+
only one with no escape available at all. Comment data does not resolve character
|
|
763
|
+
references either, and unlike script data it has no second language of its own to
|
|
764
|
+
escape in: `é` written in a comment stays `é` in every reader, so the escaper
|
|
765
|
+
does not restore the character, it replaces one unreadable form with another. The
|
|
766
|
+
reference writer therefore leaves the comment alone and serializes it as UTF-8 with
|
|
767
|
+
the rest of the prologue, deliberately. Its audience is whoever opens the raw file in an
|
|
768
|
+
editor or runs a text tool over it, and those decode the bytes as UTF-8 whatever the
|
|
769
|
+
declaration says; only a browser's raw view, which honors the declared charset, shows
|
|
770
|
+
the text as mojibake in universal mode. The bytes are harmless to extraction, since
|
|
771
|
+
the recovery payload covers the ZIP region alone, far past the prologue. A reader MUST
|
|
772
|
+
NOT rely on decoding this comment through the declared charset, and a writer that
|
|
773
|
+
wants it readable everywhere restricts it to printable ASCII. The page's own copy of
|
|
774
|
+
the comment, inside `index.html`, is UTF-8 in a UTF-8 document and is the one the
|
|
775
|
+
displayed page and the infobar carry.
|
|
704
776
|
|
|
705
777
|
## 5. Cross-cutting mechanics
|
|
706
778
|
|
|
@@ -736,12 +808,11 @@ case-insensitively: `</XMP>` and `</Script ` close their elements just as `</xmp
|
|
|
736
808
|
as the end ones. A stored, uncompressed resource is the realistic source of an
|
|
737
809
|
upper-case one.
|
|
738
810
|
|
|
739
|
-
Every rung hides its content unconditionally
|
|
740
|
-
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
demoted.
|
|
811
|
+
Every rung hides its content unconditionally, which is why `<noscript>` is not one. It
|
|
812
|
+
has the right terminator, but it is the one construct whose content is raw text only
|
|
813
|
+
while scripting is enabled and markup when it is not, so on a page opened without
|
|
814
|
+
scripting the archive bytes would reach the tree builder as tags. The rungs below it
|
|
815
|
+
hide the same payloads at no extra cost, so there is nothing to weigh against that.
|
|
745
816
|
|
|
746
817
|
The order under the comment is not arbitrary. Every rung hides its content from an HTML
|
|
747
818
|
parser, but text extractors differ, and the ZIP region is large enough that the
|
|
@@ -786,7 +857,7 @@ matching, indistinguishably from the same archive on the comment rung. The same
|
|
|
786
857
|
covered a payload holding every byte value, every rung's patterns, and the near-misses
|
|
787
858
|
`]]x>`, `] ]>`, `]>` and `]]`, in both the prologue position and mid-document.
|
|
788
859
|
|
|
789
|
-
Two things about the ladder *are* required. Whatever order a writer gives the
|
|
860
|
+
Two things about the ladder *are* required. Whatever order a writer gives the eight
|
|
790
861
|
closable rungs, it MUST apply the selection test below to every rung it considers, and
|
|
791
862
|
MUST keep `<plaintext>` available as the rung of last resort: §6.2's termination
|
|
792
863
|
argument needs one rung no payload can defeat.
|
|
@@ -820,16 +891,13 @@ what differs is how far the ladder goes:
|
|
|
820
891
|
data double escaped*, where `</script>` does **not** close the element. So a payload
|
|
821
892
|
can hold `<!--` and then `<script`, contain no `</script` anywhere, pass the end test —
|
|
822
893
|
and the wrapper then swallows its own end tag, the extra-data element and the rest of
|
|
823
|
-
the document. On the other
|
|
894
|
+
the document. On the other seven the start test is genuine conservatism: a nested `<!--`
|
|
824
895
|
is a parse error inside a comment but does not close it, and the raw-text rungs hold a
|
|
825
896
|
flat run of characters with no states at all, while CDATA sections do not nest. The
|
|
826
897
|
rule is uniform deliberately: the
|
|
827
898
|
exemption would save one pattern match per rung on bytes already in memory, at the
|
|
828
899
|
cost of a special case an implementer has to remember correctly about the single rung
|
|
829
|
-
where forgetting it destroys the document.
|
|
830
|
-
latitude, in the broader form "a writer MAY skip the start test outside universal
|
|
831
|
-
mode", and the reference writer's PDF and PNG faces took it and shipped the bug
|
|
832
|
-
(§8.5).
|
|
900
|
+
where forgetting it destroys the document.
|
|
833
901
|
- **The PDF and PNG payloads** apply the same two tests, for the same reason — a face
|
|
834
902
|
that took the `<script>` rung on a payload holding `<!--` and `<script` would swallow
|
|
835
903
|
the rest of the document, title, bootstrap and extra-data element included — but the
|
|
@@ -902,7 +970,8 @@ and the writer compares its total against 65535 before committing to it. The EOC
|
|
|
902
970
|
record's own 22 bytes sit inside the window too, giving the 65557-byte figure of §1.3.
|
|
903
971
|
|
|
904
972
|
Only the extra-data element can outgrow the budget: it carries one 2-bit code per
|
|
905
|
-
newline sequence in the
|
|
973
|
+
newline sequence in the recovered range — the ZIP region without its comment-length
|
|
974
|
+
field (§4.5), CR LF counting once, for two bytes (§5.5) — so it
|
|
906
975
|
grows with the archive. Newline bytes
|
|
907
976
|
occur at their natural density in compressed and STOREd binary data — about two in
|
|
908
977
|
every 256 bytes — and the codes are compressed and base64-encoded, which measures at
|
|
@@ -921,7 +990,12 @@ element moves in front of the wrapper start tag, ahead of the archive (§3.1). R
|
|
|
921
990
|
for it MUST be reserved before the ZIP region is written, because inserting bytes
|
|
922
991
|
ahead of the archive would shift every offset the ZIP writer has already committed;
|
|
923
992
|
the reservation is padded with spaces and the real payload is written into it once
|
|
924
|
-
its final size is known (§6).
|
|
993
|
+
its final size is known (§6). Relocation is final for the build, and a relocated
|
|
994
|
+
archive carries no appended run at all: the writer emits neither the wrapper's
|
|
995
|
+
terminator nor the end tags, so the file ends at the EOCD record like a plain ZIP file
|
|
996
|
+
and the readers that reject trailing bytes open it (§8.1). The parser closes the open
|
|
997
|
+
comment or element at end of file, and `</body></html>` are implied, so the page
|
|
998
|
+
renders the same.
|
|
925
999
|
|
|
926
1000
|
### 5.3 Offset bookkeeping
|
|
927
1001
|
|
|
@@ -938,9 +1012,12 @@ self-consistent:
|
|
|
938
1012
|
interpreted from the `%PDF-` header, so embedding it needs no rewriting; the writer
|
|
939
1013
|
only MUST keep the header inside the scan window (§4.3).
|
|
940
1014
|
- **PNG has no offsets, only lengths.** Each chunk declares its data length. The
|
|
941
|
-
|
|
942
|
-
can only be written once the file's final size is
|
|
943
|
-
in place at the end (§6).
|
|
1015
|
+
`tEXt "ZIP"` chunk's length covers the whole archive, and the appended data too in
|
|
1016
|
+
the variants that have it, so it can only be written once the file's final size is
|
|
1017
|
+
known, and the writer patches it in place at the end (§6). A `tEXt` chunk precedes it
|
|
1018
|
+
whenever the HTML face or the PDF face is present — carrying the prologue or the PDF
|
|
1019
|
+
document respectively — and under the PNG face alone it is the only one. Appended
|
|
1020
|
+
data follows it only under the HTML face.
|
|
944
1021
|
|
|
945
1022
|
The injected `page.pdf` central record exploits a fourth, deliberate discrepancy. It
|
|
946
1023
|
is written directly to the output stream, bypassing the ZIP writer's own byte
|
|
@@ -1068,25 +1145,28 @@ ZIP tool can verify.
|
|
|
1068
1145
|
|
|
1069
1146
|
### 5.7 zip64
|
|
1070
1147
|
|
|
1071
|
-
The archive uses the zip64 structures whenever the ordinary
|
|
1072
|
-
it: a central directory starting beyond 4 GiB (the prefix counts
|
|
1073
|
-
§5.3), a directory 4 GiB or longer, or 65535 entries or more.
|
|
1074
|
-
|
|
1075
|
-
|
|
1148
|
+
The archive uses the zip64 end of central directory structures whenever the ordinary
|
|
1149
|
+
records cannot express it: a central directory starting beyond 4 GiB (the prefix counts
|
|
1150
|
+
toward the offset, §5.3), a directory 4 GiB or longer, or 65535 entries or more. A
|
|
1151
|
+
single entry of 4 GiB or more also produces zip64 extra fields, in that entry's local
|
|
1152
|
+
and central headers, without any zip64 end of central directory record. The reference
|
|
1153
|
+
writer never requests zip64 explicitly, so it appears only when reached, and given how
|
|
1154
|
+
large that is, effectively never in a saved page.
|
|
1076
1155
|
|
|
1077
1156
|
When it is reached, the EOCD record carries the sentinel values `0xFFFF` and
|
|
1078
|
-
`0xFFFFFFFF
|
|
1079
|
-
|
|
1080
|
-
|
|
1157
|
+
`0xFFFFFFFF`, preceded by a zip64 end of central directory record and its locator.
|
|
1158
|
+
The sentinels are not selective: the writer saturates the entry counts, the directory
|
|
1159
|
+
size and the directory offset together once zip64 is emitted, whichever one of them
|
|
1160
|
+
overflowed. The `page.pdf` record injection then applies its accounting to the zip64
|
|
1161
|
+
record instead — entry counts and directory size there, and
|
|
1081
1162
|
the locator's pointer moved by the record's length — while leaving each saturated
|
|
1082
1163
|
field at its sentinel. A writer MUST NOT let the injection push a 16-bit or 32-bit
|
|
1083
1164
|
field to its sentinel value without emitting the corresponding zip64 record: a count
|
|
1084
1165
|
of `0xFFFF` sends readers looking for a zip64 record that does not exist.
|
|
1085
1166
|
|
|
1086
|
-
|
|
1087
|
-
|
|
1088
|
-
|
|
1089
|
-
the non-zip64 build.
|
|
1167
|
+
`test/sfz-harness/zip64.js` covers this: the sentinels stay, the counts and the
|
|
1168
|
+
directory size land in the zip64 record, its directory offset points at the injected
|
|
1169
|
+
record, `page.pdf` is the first record in the directory, and a reader lists every entry.
|
|
1090
1170
|
|
|
1091
1171
|
zip64 does not conflict with universal mode. Its commonest trigger, 65535 entries or
|
|
1092
1172
|
more, is reached at any archive size, and §4.5 gives the offset arithmetic for a
|
|
@@ -1094,6 +1174,40 @@ recovered region whose EOCD fields are sentinels. What universal mode cannot car
|
|
|
1094
1174
|
a ZIP region of 2^32 bytes or more, which the recovery payload's 32-bit length field
|
|
1095
1175
|
cannot express (§5.5) — a size bound, not a zip64 one.
|
|
1096
1176
|
|
|
1177
|
+
### 5.8 Entry name encoding
|
|
1178
|
+
|
|
1179
|
+
Entry names in this format are arbitrary Unicode, and how a name is decoded is an
|
|
1180
|
+
interoperability question rather than a detail.
|
|
1181
|
+
|
|
1182
|
+
The reference writer never exercises that range. Its names are a fixed prefix, an
|
|
1183
|
+
index and an extension — `index.html`, `manifest.json`, `stylesheet_0.css`,
|
|
1184
|
+
`images/1.png`, `fonts/2.woff2`, `scripts/3.js`, `frames/4/`, `page.pdf` — and the
|
|
1185
|
+
extension comes either from a table of content types or from a URL pathname, which is
|
|
1186
|
+
percent-encoded. Every name it writes is therefore ASCII, whatever the language of the
|
|
1187
|
+
captured page. That is a property of this writer, not a guarantee of the format: a
|
|
1188
|
+
conforming writer may name entries after the resources themselves, and §7.3's rule
|
|
1189
|
+
that entry names are untrusted assumes one does.
|
|
1190
|
+
|
|
1191
|
+
How a name is encoded is ZIP's own business, not this format's: bit 11 of the general
|
|
1192
|
+
purpose bit flag selects UTF-8, and its absence selects the legacy code page. This
|
|
1193
|
+
document adds two requirements to that and specifies nothing else about it.
|
|
1194
|
+
|
|
1195
|
+
**A writer MUST set bit 11 on every entry**, not only on the entries whose names need
|
|
1196
|
+
it. The two encodings agree over printable ASCII, so setting it unconditionally costs
|
|
1197
|
+
nothing, and it means no name in the archive is decoded through the legacy path at all.
|
|
1198
|
+
|
|
1199
|
+
**A reader MUST honor the flag** rather than assume one encoding, and MUST expect to
|
|
1200
|
+
meet a clear one: the hand-built `page.pdf` records (§3.1, §6) are the only ones the
|
|
1201
|
+
reference writer does not produce through its ZIP writer, and an archive may carry them
|
|
1202
|
+
with no flag set at all. That single entry is then decoded as legacy while every other
|
|
1203
|
+
name in the same file is UTF-8.
|
|
1204
|
+
Its name is ASCII, where the two encodings agree, so a correct reader sees `page.pdf`
|
|
1205
|
+
either way — but a reader that hardcodes UTF-8 on the strength of the other entries has
|
|
1206
|
+
not covered it.
|
|
1207
|
+
|
|
1208
|
+
A name is not a path. §7.3's rule that entry names are untrusted applies to the decoded
|
|
1209
|
+
name, and decoding is the step before that check, not a substitute for it.
|
|
1210
|
+
|
|
1097
1211
|
## 6. Writer algorithm
|
|
1098
1212
|
|
|
1099
1213
|
This section specifies the reference writer's build order. It is normative in the
|
|
@@ -1111,7 +1225,8 @@ buys. The cost is not evenly spread:
|
|
|
1111
1225
|
| Plus universal mode | The recovery payload, the character round trip of §5.5, the appended-data budget of §5.2, and the retry loops of §6.2. This is where the real complexity lives, and it buys opening the file from `file:` with no cooperation |
|
|
1112
1226
|
| Plus the PDF or PNG face | The header window of §4.3 or the chunk patching of §5.3, plus a second wrapper choice for the embedded payload |
|
|
1113
1227
|
|
|
1114
|
-
|
|
1228
|
+
The second row already needs a rebuild when the rung changes; the third adds the rest of
|
|
1229
|
+
the retry loops, and only the fourth needs a value that cannot
|
|
1115
1230
|
be computed until the file is otherwise complete. A writer that only wants durable saved
|
|
1116
1231
|
pages can stop at the first row; the files it produces are accepted by every reader in
|
|
1117
1232
|
§8.1.
|
|
@@ -1123,15 +1238,46 @@ pages can stop at the first row; the files it produces are accepted by every rea
|
|
|
1123
1238
|
`tEXt "PDF"` chunk holding the PDF document here, so its header falls inside the
|
|
1124
1239
|
PDF scan window (§4.3).
|
|
1125
1240
|
2. **HTML prologue.** With the HTML face, emit the doctype (omitted under the PNG
|
|
1126
|
-
face, which owns the start of the file), the root element start tag,
|
|
1127
|
-
|
|
1241
|
+
face, which owns the start of the file), the root element start tag, the
|
|
1242
|
+
`<meta charset>` required by §2.1, any comment the implementation adds — after the
|
|
1243
|
+
charset declaration, since a comment carrying the page URL has no bound and would
|
|
1244
|
+
otherwise push that declaration out of the first 1024 bytes. The doctype is the
|
|
1245
|
+
other unbounded region ahead of the declaration, copied from the saved page with its
|
|
1246
|
+
identifiers verbatim, so a writer MUST emit a minimal doctype in its place when
|
|
1247
|
+
keeping it would push the declaration past 1024 bytes.
|
|
1248
|
+
|
|
1249
|
+
**Replace it; do not truncate it, and do not drop it.** Truncation is unsafe:
|
|
1250
|
+
a cut inside a quoted identifier leaves the tokenizer in the system-identifier
|
|
1251
|
+
state, where it consumes the markup that follows until the next `>` — swallowing
|
|
1252
|
+
the root element start tag and the `data-sfz` marker on it (§1.3), so the document
|
|
1253
|
+
loses both. Dropping the doctype parses
|
|
1254
|
+
cleanly but puts the document in quirks mode, which is the mode the blank-page
|
|
1255
|
+
backstop, the wait message and the error message are then rendered under (§4.1) —
|
|
1256
|
+
the error message most of all, since it is what a reader sees precisely when
|
|
1257
|
+
nothing else has worked. A minimal doctype is 15 bytes, keeps standards mode, and
|
|
1258
|
+
costs nothing else: the extracted page is written into the document with its own
|
|
1259
|
+
doctype (§4.1), so the outer one never governs the restored page.
|
|
1260
|
+
|
|
1261
|
+
The PNG face is the exception, and nothing is available to it either way. A PNG file
|
|
1262
|
+
MUST begin with its 8-byte signature, so no doctype can precede it, and one written
|
|
1263
|
+
after the PNG head is discarded: those bytes are character data, so the parser has
|
|
1264
|
+
left its initial insertion mode and ignores a DOCTYPE token. The variant renders in
|
|
1265
|
+
quirks mode until the extracted page replaces it, whatever the writer does, so this
|
|
1266
|
+
section requires nothing about the doctype there. A writer MAY drop it, as the
|
|
1267
|
+
reference writer does, or keep it — but a kept one is content like any other, and
|
|
1268
|
+
both windows are measured from the start of the *file*, which under this face begins
|
|
1269
|
+
45 bytes before the HTML does: the signature, `IHDR`, and the chunk length, type and
|
|
1270
|
+
keyword. That is 45 bytes less room than the arithmetic above suggests.
|
|
1271
|
+
|
|
1272
|
+
Then the head elements (the
|
|
1128
1273
|
`<title>` and the canonical link among them), the CSS and `<body hidden>`,
|
|
1129
1274
|
the wait and error messages, the optional table of contents and text body, and the
|
|
1130
1275
|
bootstrap script. With a password, five of those are left out: the comment, the
|
|
1131
1276
|
title, the canonical link, the text body and the entry comments of step 6 (§5.6).
|
|
1132
1277
|
With the PNG face the head of this region,
|
|
1133
1278
|
through `<body hidden>`, is the data of the `tEXt "PNG"` chunk and the remainder is
|
|
1134
|
-
emitted after the `tEXt "ZIP"` chunk header
|
|
1279
|
+
emitted after the `tEXt "ZIP"` chunk header, which step 1 has already written and
|
|
1280
|
+
step 12 only patches; with the PDF face the
|
|
1135
1281
|
region is interrupted by step 3 as well.
|
|
1136
1282
|
|
|
1137
1283
|
Whatever a writer puts in the prologue, closing every element it opens before the
|
|
@@ -1141,7 +1287,7 @@ pages can stop at the first row; the files it produces are accepted by every rea
|
|
|
1141
1287
|
3. **Embedded PDF.** With the PDF face and the HTML face, the prologue is *split*
|
|
1142
1288
|
around the PDF, which MUST come early enough for `%PDF-` to start at offset 1024
|
|
1143
1289
|
or lower (§4.3). Only what a parser needs first precedes it — the doctype, the
|
|
1144
|
-
root element
|
|
1290
|
+
root element and the charset declaration — and everything
|
|
1145
1291
|
else in the head (title, link and meta elements, the stylesheet, `<body hidden>`,
|
|
1146
1292
|
the messages, the optional table of contents and text body) follows it. Emit the
|
|
1147
1293
|
wrapper start tag chosen for the PDF payload (§5.1), the hand-built `page.pdf`
|
|
@@ -1152,26 +1298,29 @@ pages can stop at the first row; the files it produces are accepted by every rea
|
|
|
1152
1298
|
where the format depends on the writer rather than on its own layout. The
|
|
1153
1299
|
irreducible part of the prefix is small: the root element start tag, the charset
|
|
1154
1300
|
declaration, the wrapper start tag and the 38-byte local file header for
|
|
1155
|
-
`page.pdf`, plus a minimal doctype —
|
|
1156
|
-
|
|
1301
|
+
`page.pdf`, plus a minimal doctype — 92 bytes in the reference layout with a
|
|
1302
|
+
`utf-8` label, 99 with `windows-1252`, and a few more with a wrapper past the first
|
|
1303
|
+
rung (§5.1). But two
|
|
1157
1304
|
regions ahead of the header have no length the format controls: the doctype, which
|
|
1158
1305
|
is copied from the saved page and carries its public and system identifiers
|
|
1159
1306
|
verbatim, and any comment the implementation chooses to write there. Real doctypes
|
|
1160
1307
|
are small; the longest in common use, XHTML 1.1 with MathML and SVG, is about 140
|
|
1161
|
-
bytes
|
|
1162
|
-
|
|
1163
|
-
|
|
1164
|
-
|
|
1165
|
-
|
|
1308
|
+
bytes, though a crafted one is bounded only by what the parser accepts. Step 2's
|
|
1309
|
+
MUST already caps the doctype, but only far enough to keep the charset declaration
|
|
1310
|
+
inside the window; this header sits further into the file, behind the wrapper tag
|
|
1311
|
+
and a 38-byte local header, so it needs the tighter bound below and the comment
|
|
1312
|
+
needs one too. A writer MUST cap them itself, keeping
|
|
1313
|
+
everything before the local file header inside the remaining budget of roughly 930
|
|
1314
|
+
bytes (about 900 with the PNG face, whose signature, `IHDR` and first chunk header
|
|
1315
|
+
take the first 45 bytes of the same window while its variant drops the 15-byte
|
|
1316
|
+
doctype in exchange), shortening, dropping or relocating that content instead of emitting a
|
|
1166
1317
|
header outside the window. A writer that places nothing of unbounded length before
|
|
1167
1318
|
the PDF block satisfies the rule by construction and needs no check at all.
|
|
1168
1319
|
|
|
1169
1320
|
The reference writer does both. Its provenance comment is emitted after the PDF
|
|
1170
1321
|
block, so the page URL it carries cannot reach the window at all, and the prefix is
|
|
1171
1322
|
measured before the header is written: when the page's own doctype would push
|
|
1172
|
-
`%PDF-` past 1024, `<!DOCTYPE html>` is emitted in its place.
|
|
1173
|
-
doctype changes the bootstrap document's rendering mode, which costs nothing here:
|
|
1174
|
-
the extracted page is written into the document with its own doctype (§4.1). A
|
|
1323
|
+
`%PDF-` past 1024, `<!DOCTYPE html>` is emitted in its place, on step 2's rule. A
|
|
1175
1324
|
writer that must keep the page doctype has to find the room elsewhere.
|
|
1176
1325
|
|
|
1177
1326
|
Without the HTML face, the PDF is simply the first thing in the file and the
|
|
@@ -1193,16 +1342,20 @@ pages can stop at the first row; the files it produces are accepted by every rea
|
|
|
1193
1342
|
8. **Close and patch.** Close the archive, then correct the end of central directory
|
|
1194
1343
|
record for the injected record: entry counts, directory size, and the zip64
|
|
1195
1344
|
record and locator when present (§5.7).
|
|
1196
|
-
9. **Universal payload.**
|
|
1197
|
-
|
|
1198
|
-
|
|
1345
|
+
9. **Universal payload.** With the HTML face — not only in universal mode, since any
|
|
1346
|
+
self-extracting file needs it — read back the ZIP region and check it against the
|
|
1347
|
+
current wrapper (§5.1); on a collision, restart (§6.2). Then, in universal mode
|
|
1348
|
+
only, compute the region's CRC-32 and its newline codes, build and compress the payload,
|
|
1199
1349
|
and decide its placement against the budget (§5.2), restarting if the decision
|
|
1200
1350
|
differs from the current pass.
|
|
1201
|
-
10. **Appended run.** Unless appended data is prevented
|
|
1351
|
+
10. **Appended run.** Unless appended data is prevented or the payload is relocated
|
|
1352
|
+
(§5.2), emit the wrapper end tag,
|
|
1202
1353
|
the extra-data element when it is appended, and `</body></html>` — the end tags
|
|
1203
1354
|
are omitted under the PNG face, which must end with `IEND`.
|
|
1204
1355
|
11. **Fill the reservation.** In the relocated placement, write the payload into the
|
|
1205
|
-
space reserved in step 4; if it no longer fits, restart (§6.2).
|
|
1356
|
+
space reserved in step 4; if it no longer fits, restart (§6.2). Under
|
|
1357
|
+
`declareAppendedData` (§4.2), the EOCD's comment-length field is patched here too,
|
|
1358
|
+
the appended run's length now being final.
|
|
1206
1359
|
12. **PNG tail.** With the PNG face, patch the `tEXt "ZIP"` chunk's length field, now
|
|
1207
1360
|
that the total size is known, compute that chunk's CRC over everything from its
|
|
1208
1361
|
type to the last byte written, and append the CRC and the `IEND` chunk.
|
|
@@ -1210,8 +1363,10 @@ pages can stop at the first row; the files it produces are accepted by every rea
|
|
|
1210
1363
|
### 6.2 The retry loops
|
|
1211
1364
|
|
|
1212
1365
|
Four conditions restart the build from step 1, and each restart carries forward what
|
|
1213
|
-
the failed pass learned.
|
|
1214
|
-
|
|
1366
|
+
the failed pass learned. Nothing a restart changes reaches the entries' bytes: a writer
|
|
1367
|
+
may compress them once and copy them into every pass, rewriting only the central
|
|
1368
|
+
directory's offsets, which is what the reference writer does. The first three
|
|
1369
|
+
terminate because each of them advances a monotone quantity:
|
|
1215
1370
|
|
|
1216
1371
|
- **Wrapper collision** (§5.1): the next pass starts at the next rung of the ladder.
|
|
1217
1372
|
The ladder is finite and its last rung, `<plaintext>`, is exempt from both selection
|
|
@@ -1220,29 +1375,42 @@ monotone quantity:
|
|
|
1220
1375
|
ahead of the archive, sized at the measured payload length plus a margin.
|
|
1221
1376
|
- **Reservation too small**: relocating the payload changes the file's layout, hence
|
|
1222
1377
|
its offsets, hence the payload, which can grow past the room reserved for it. The
|
|
1223
|
-
next pass reserves the new length plus the same margin.
|
|
1224
|
-
|
|
1225
|
-
|
|
1226
|
-
|
|
1227
|
-
|
|
1228
|
-
|
|
1229
|
-
|
|
1230
|
-
|
|
1231
|
-
|
|
1232
|
-
|
|
1233
|
-
|
|
1234
|
-
|
|
1235
|
-
|
|
1236
|
-
|
|
1237
|
-
|
|
1238
|
-
|
|
1378
|
+
next pass reserves the new length plus the same margin. This restart fires only
|
|
1379
|
+
when the payload outgrew its reservation and it reserves at least that payload, so
|
|
1380
|
+
every reservation is larger than the one before and the loop cannot revisit a size.
|
|
1381
|
+
What keeps it short is the margin. Shifting the offsets changes a few of the
|
|
1382
|
+
central directory's bytes, which changes the line-ending codes, the deflate output
|
|
1383
|
+
and the base64 rounding, so the payload moves by a few quanta of 4 characters
|
|
1384
|
+
between two layouts: measured between -16 and +20 characters over archives of 8 to
|
|
1385
|
+
2000 entries. A margin smaller than that shift buys a third pass in about one build
|
|
1386
|
+
out of four. The writer reserves the measured length plus 1 % plus 32 characters,
|
|
1387
|
+
which absorbed every shift measured.
|
|
1388
|
+
|
|
1389
|
+
There is no converse of the second: a pass that reserved room never discards it,
|
|
1390
|
+
even when the relocated payload would have fit the appended window. Relocation moves
|
|
1391
|
+
the archive, which changes the offsets, which changes the payload that made the
|
|
1392
|
+
relocation necessary, so a payload lying on the 65535-byte boundary can be too large
|
|
1393
|
+
appended and small enough relocated, and a writer that dropped the reservation could
|
|
1394
|
+
rebuild the two placements forever. Relocation is therefore final (§5.2), and the file
|
|
1395
|
+
keeps at most the reservation's own margin of dead padding.
|
|
1396
|
+
|
|
1397
|
+
The fourth stands apart from the other three, and terminates trivially because it can
|
|
1398
|
+
fire only once: if the end of central directory record cannot be patched to account for
|
|
1399
|
+
the injected `page.pdf` record — its signature not where the accounting expects it —
|
|
1400
|
+
the writer rebuilds without that record rather than leave a central directory the EOCD
|
|
1401
|
+
does not count. That is the restart enforcing §5.7's requirement that the injection
|
|
1402
|
+
never leave the two disagreeing, and the rebuilt archive simply has no `page.pdf`
|
|
1403
|
+
entry.
|
|
1239
1404
|
|
|
1240
1405
|
Given identical inputs, modification date and archive time, the process is
|
|
1241
|
-
deterministic: the same page produces the same bytes, retries included.
|
|
1242
|
-
|
|
1243
|
-
|
|
1244
|
-
|
|
1245
|
-
|
|
1406
|
+
deterministic: the same page produces the same bytes, retries included. `manifest.json`
|
|
1407
|
+
records when the archive was made (§7.1), so two builds of one page at two moments
|
|
1408
|
+
differ in that entry and in the entry sizes around it. A writer that retries MUST pin
|
|
1409
|
+
the archive time across the passes of one build rather than read the clock again on
|
|
1410
|
+
each: the reference writer takes it from the clock inside the callback that emits the
|
|
1411
|
+
entries, which runs once per pass, so a retried build is reproducible only when that
|
|
1412
|
+
clock is frozen — which is what its own determinism test does. A consumer MUST NOT
|
|
1413
|
+
treat the byte identity of two archives of the same page as meaningful.
|
|
1246
1414
|
|
|
1247
1415
|
## 7. Consuming SingleFile archives safely
|
|
1248
1416
|
|
|
@@ -1254,8 +1422,10 @@ handles every variant of §2 without knowing which one it has.
|
|
|
1254
1422
|
|
|
1255
1423
|
- **Read through the central directory.** Locate the End Of Central Directory record
|
|
1256
1424
|
by scanning backward from the end of the file, then follow its offset. A reader that
|
|
1257
|
-
streams local headers from offset 0 will not find an archive
|
|
1258
|
-
the HTML, PDF or PNG face
|
|
1425
|
+
streams local headers from offset 0 will not find an archive in any variant that has
|
|
1426
|
+
a face, since the file then starts with the HTML, PDF or PNG face. The variant with
|
|
1427
|
+
no face is an ordinary ZIP file and streams fine (§1.2, and §8.1 measures what such
|
|
1428
|
+
readers actually do).
|
|
1259
1429
|
- **Tolerate bytes before and after the archive.** They are the other faces, not
|
|
1260
1430
|
corruption. Offsets are absolute, so no compensation is needed (§5.3).
|
|
1261
1431
|
- **Accept both forms of appended data.** The bytes after the EOCD record may be raw
|
|
@@ -1301,8 +1471,9 @@ handles every variant of §2 without knowing which one it has.
|
|
|
1301
1471
|
MUST ignore what it does not recognize.
|
|
1302
1472
|
- **Expect a `page.pdf` entry whose data lies outside the archive proper** (§4.2). It
|
|
1303
1473
|
is an ordinary STORE entry at an ordinary offset, so nothing special is needed to
|
|
1304
|
-
read it, but a reader that assumes
|
|
1305
|
-
|
|
1474
|
+
read it, but a reader that assumes the entries are contiguous will reject or
|
|
1475
|
+
mislocate it: `page.pdf`'s local header is the first in the file, and the whole
|
|
1476
|
+
bootstrap lies between its data and the next one.
|
|
1306
1477
|
|
|
1307
1478
|
### 7.2 Modifying
|
|
1308
1479
|
|
|
@@ -1326,8 +1497,10 @@ alongside it.
|
|
|
1326
1497
|
|
|
1327
1498
|
### 7.3 Security considerations
|
|
1328
1499
|
|
|
1329
|
-
- **Entry names are untrusted.**
|
|
1330
|
-
|
|
1500
|
+
- **Entry names are untrusted.** The reference writer's names are a fixed prefix, an
|
|
1501
|
+
index and an extension (§5.8), but nothing in the format requires that, and a writer
|
|
1502
|
+
may name entries after the resources themselves. A reader MUST sanitize them before
|
|
1503
|
+
writing to a filesystem: reject absolute paths and
|
|
1331
1504
|
`..` segments, and be aware that names may be long, may collide after case folding,
|
|
1332
1505
|
and may contain characters the local filesystem rejects.
|
|
1333
1506
|
- **Declared sizes are untrusted.** Do not pre-allocate from the declared uncompressed
|
|
@@ -1338,8 +1511,9 @@ alongside it.
|
|
|
1338
1511
|
the bootstrap in a privileged one. The format's own display path replaces the
|
|
1339
1512
|
document with the extracted page, which is not an isolation boundary by itself.
|
|
1340
1513
|
- **A password protects entry contents only** (§5.6). Entry names, sizes and dates
|
|
1341
|
-
stay readable in the central directory, and
|
|
1342
|
-
|
|
1514
|
+
stay readable in the central directory, and while the reference writer's names carry
|
|
1515
|
+
no information about the resources (§5.8), another writer's may state their
|
|
1516
|
+
filenames. The PNG and PDF faces render the page regardless. A conforming writer
|
|
1343
1517
|
withholds the five fields of §5.6, the source URLs among them, but a reader MUST NOT
|
|
1344
1518
|
read their absence as protection: nothing in the format stops a writer from emitting
|
|
1345
1519
|
any of them, so an archive of unknown provenance may state every URL in the clear.
|
|
@@ -1361,7 +1535,7 @@ only if it affects the bytes the page is built from:
|
|
|
1361
1535
|
| A recovery payload field disagrees with the reconstruction — length, newline count or checksum | **MUST** fail (§4.5). The reconstruction is wrong and nothing built from it can be trusted |
|
|
1362
1536
|
| An entry's CRC-32 or AES authentication code does not match | **SHOULD** fail for that entry, and MUST NOT present a page rebuilt from it as intact |
|
|
1363
1537
|
| `page.pdf` was reconstructed from the parsed page and its CRC-32 does not match | **MUST** discard the reconstruction (§4.5). The bytes are a guess about newlines the recovery payload does not describe, and the checksum is the only thing that tests it — unlike the row above, there is no read to have gone wrong, only an inference |
|
|
1364
|
-
| Bytes before the first local file header,
|
|
1538
|
+
| Bytes outside the archive proper — before the first local file header, after the EOCD record, or between an entry's data and the next header | **MUST** tolerate: they are the other faces (§7.1). The gap in the middle is not hypothetical: with the PDF face the bootstrap lies between `page.pdf`'s data and the ZIP region |
|
|
1365
1539
|
| The appended run exceeds the 65535-byte budget (§5.2) | Not a reader's problem: if the EOCD record was found, the archive is readable. Readers MAY warn |
|
|
1366
1540
|
| A `tEXt` chunk CRC does not match, or a chunk holds bytes PNG does not permit (§4.4) | Irrelevant to extraction; a reader of the archive MAY ignore both |
|
|
1367
1541
|
| `page.pdf` is present but its data does not begin with `%PDF-` | Not an error. The entry is data like any other |
|
|
@@ -1370,9 +1544,12 @@ only if it affects the bytes the page is built from:
|
|
|
1370
1544
|
| The recovered region (universal mode) disagrees with the same bytes read directly, in the EOCD's two comment-length bytes only | Expected, not an error. A recovered region always declares a zero-length comment (§4.5), so it differs here from any archive written in the declared form (§4.2). Compare the two only up to those bytes |
|
|
1371
1545
|
| The recovered region (universal mode) disagrees with the same bytes read directly, anywhere else | The file is not well-formed, whichever side is at fault, and a reader that has both MUST NOT silently merge them or pick per entry. Prefer the direct read — it is the writer's own output, where the recovered region is a reconstruction of it — and surface the disagreement rather than displaying either as intact |
|
|
1372
1546
|
|
|
1373
|
-
Anything the format does not constrain, a reader MUST NOT reject: entries may carry
|
|
1374
|
-
|
|
1375
|
-
|
|
1547
|
+
Anything the format does not constrain, a reader MUST NOT reject: entries may carry any
|
|
1548
|
+
extra fields, timestamps or data descriptors a ZIP writer would ordinarily emit. The few
|
|
1549
|
+
this document does constrain — the `0x9901` field of an encrypted entry (§4.2), the
|
|
1550
|
+
zip64 records (§5.7), the name-encoding flag (§5.8) — say how an entry is read, not
|
|
1551
|
+
whether it is acceptable, so this row covers them too: each is something a reader meets
|
|
1552
|
+
and reads.
|
|
1376
1553
|
|
|
1377
1554
|
## 8. Appendices
|
|
1378
1555
|
|
|
@@ -1445,15 +1622,25 @@ measured except Apple's `ditto`.
|
|
|
1445
1622
|
### 8.2 Anatomy of a small archive
|
|
1446
1623
|
|
|
1447
1624
|
Offsets in `universal.sfz.html` (123077 bytes, two entries, saved from `example.com`
|
|
1448
|
-
with the §8.3 command,
|
|
1625
|
+
with the §8.3 command, on the build named there).
|
|
1449
1626
|
The layout is the *universal* row of the byte map (§3).
|
|
1450
1627
|
|
|
1628
|
+
These numbers are one capture, not a contract. Everything from the bootstrap onward
|
|
1629
|
+
moves whenever the inlined ZIP library changes size, so treat the table as an
|
|
1630
|
+
illustration of the shape and not as values to compare a file against. What *is* fixed
|
|
1631
|
+
is the set of relations between the rows — the doctype opening the file with the root
|
|
1632
|
+
element start tag immediately after it, the charset declaration immediately after that
|
|
1633
|
+
and the comment immediately after that, the identifier's twelve bytes ahead of the
|
|
1634
|
+
region, the EOCD's directory offset being an absolute file position, and the entry
|
|
1635
|
+
order. Those are checked by `test/sfz-harness/byte-map.js`, which builds an equivalent
|
|
1636
|
+
specimen without a network.
|
|
1637
|
+
|
|
1451
1638
|
| Offset | Bytes | Region |
|
|
1452
1639
|
|---|---|---|
|
|
1453
1640
|
| 0 | `<!DOCTYPE html>` | `html-prologue` begins |
|
|
1454
|
-
|
|
|
1455
|
-
|
|
|
1456
|
-
|
|
|
1641
|
+
| 15 | `<html data-sfz>` | root element start tag; the attribute is the reference implementation's own marker (§1.3) |
|
|
1642
|
+
| 30 | `<meta charset=windows-1252>` | the charset rule, inside the first 1024 bytes (§2.1) |
|
|
1643
|
+
| 57 | `<!--` … `-->` (ends at 200) | comment written by the implementation; its content is implementation-defined, but where it may appear is not (§3.1, §4.6, §5.6). It follows the charset declaration so it cannot push it out of the prescan window |
|
|
1457
1644
|
| 200 | `<title>` … `</title>` (ends at 229) | the page title, as numeric character references (§4.6) |
|
|
1458
1645
|
| 677 | `<style>` | the stylesheet of the blank-page backstop (§4.1) |
|
|
1459
1646
|
| 855 | `<body hidden>` | start of the blank-page backstop (§4.1) |
|
|
@@ -1479,7 +1666,8 @@ Generated with the command-line client running `single-file-core` against
|
|
|
1479
1666
|
results of §8.1 were measured on the 1.5.107 build of the same specimen set, which
|
|
1480
1667
|
differs only inside the prologue and so falls in the same classes: those are grouped
|
|
1481
1668
|
by whether bytes precede the archive and follow the EOCD, which no prologue change
|
|
1482
|
-
alters.
|
|
1669
|
+
alters. The declared-form results are the exception, `declareAppendedData` being later
|
|
1670
|
+
than that build (§8.5); they were measured separately on a build that has it. `--compress-content` makes the output an archive; `extract-data-from-page`
|
|
1483
1671
|
defaults to true there, so the plain variant has to switch it off:
|
|
1484
1672
|
|
|
1485
1673
|
| Specimen | Command |
|
|
@@ -1502,15 +1690,13 @@ entry comment, the optional text body or table of contents (§4.6), a UTF-8 BOM,
|
|
|
1502
1690
|
in it. Two omissions matter more than the rest, because they are the parts of §5.1 a
|
|
1503
1691
|
writer is most likely to get wrong: no specimen defeats a rung by its **start**
|
|
1504
1692
|
pattern, and none defeats one with an **upper-case** pattern. A writer that tested only
|
|
1505
|
-
end patterns, or matched them case-sensitively, produces every specimen here unchanged
|
|
1506
|
-
— and the first of those two mistakes is one the reference writer actually shipped
|
|
1507
|
-
(§8.5).
|
|
1693
|
+
end patterns, or matched them case-sensitively, produces every specimen here unchanged.
|
|
1508
1694
|
|
|
1509
1695
|
Two specimens cannot be produced from a URL alone. The **ladder** specimen, which
|
|
1510
1696
|
forces the second rung of §5.1, needs a page referencing an image whose stored bytes
|
|
1511
1697
|
contain `-->`; the archive then wraps in `<script type=sfz-data>`. The **zip64** specimen requires
|
|
1512
1698
|
an archive past the thresholds of §5.7, so it is produced by calling the writer
|
|
1513
|
-
directly with zip64 forced on the ZIP writer.
|
|
1699
|
+
directly with zip64 forced on the ZIP writer, as `test/sfz-harness/zip64.js` does.
|
|
1514
1700
|
|
|
1515
1701
|
The measurements quoted elsewhere in this document come from the same harness: the
|
|
1516
1702
|
payload growth rate of §5.2 (86 KB → 181 bytes, 283 KB → 465, 1.07 MB → 1645, 4.2 MB
|
|
@@ -1535,7 +1721,7 @@ multi-byte ones (`utf-8`, `utf-16le`, `utf-16be`, `gbk`, `gb18030`, `big5`, `euc
|
|
|
1535
1721
|
`shift_jis`, `euc-kr`, `iso-2022-jp`) decode a lone byte sequence to U+FFFD or to fewer
|
|
1536
1722
|
than 256 characters.
|
|
1537
1723
|
The reverse table each one needs ranges from 8 entries (`iso-8859-15`) to 128
|
|
1538
|
-
(`koi8-r`, `koi8-u` and `
|
|
1724
|
+
(`koi8-r`, `koi8-u`, `ibm866` and `x-user-defined`); windows-1252 needs 27.
|
|
1539
1725
|
|
|
1540
1726
|
**That the round trip is charset-independent.** The mechanism of §5.5 — parse, then
|
|
1541
1727
|
re-encode with the reverse table, restoring newlines from the 2-bit codes and NUL from
|
|
@@ -1567,6 +1753,8 @@ predicts.
|
|
|
1567
1753
|
| August 2026 | Core 1.5.110: a PNG face leaving the comment rung on its checksum resumes the rung search instead of taking the next rung untested (§5.1). Taking it put a payload holding `</script>` on the script rung, where its own bytes closed the wrapper 93 bytes in and left the image data, the chunk framing and the whole ZIP region to the parser |
|
|
1568
1754
|
| August 2026 | Core 1.5.110: `<svg><![CDATA[` joins the ladder above `<plaintext>` (§5.1) — the one rung whose terminator, `]]>`, real payloads rarely carry. It gives a payload naming every element rung somewhere to go that does not cost the appended-data placement, and moves the self-nesting limit from the fifth level to the sixth |
|
|
1569
1755
|
| August 2026 | Core 1.5.115: password-protected archives withhold the provenance comment and the canonical link as well (§5.6). Both wrote the page's own URL into the prologue, beside the title that was already withheld, so the address the archive was saved from stayed in the clear |
|
|
1756
|
+
| August 2026 | Core 1.5.119: the inlined ZIP library is built ASCII-only, and §2.1 now requires it of any bootstrap. Its CP437 table had been emitted as literal characters, which the page re-decoded as windows-1252, growing the table from 256 entries to 508 and shifting every lookup by 60 — so the one entry read without the UTF-8 flag, `page.pdf`, came back mangled and no archive with a PDF face extracted in any engine (§5.8) |
|
|
1757
|
+
| August 2026 | Core 1.5.120: the hand-built `page.pdf` records set the language encoding flag, like every entry the ZIP writer produces (§5.8). Its name is ASCII, so no decoded name changes; what changes is that no entry in an archive is read through CP437 any more, closing the path the 1.5.119 defect surfaced on |
|
|
1570
1758
|
|
|
1571
1759
|
This document was itself revised in August 2026, against core 1.5.108, after several
|
|
1572
1760
|
independent reviews. One of them was a reader built from this specification alone, with
|