wpd-codec 3.0.9 → 3.1.0

package/README.md

@@ -10,21 +10,22 @@ Created for [documents.js#819](https://github.com/ExaDev/documents.js/issues/819
 
 ## Sources
 
-Everything this package does is derived from the vendor's own documentation, and every non-obvious decision in the source cites the page it comes from.
-
-| Source                                                                                                                                                                                                                                                      | What it gives                                                                                                                                                                                                            |
-| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| [WordPerfect File Format SDK help](https://github.com/OneWingedShark/WordPerfect/tree/master/doc/SDK_Help/FileFormats)                                                                                                                                      | The specification itself, mirrored in full: document structure, the prefix packet catalogue, single-byte characters and functions, every variable-length function group, the fixed-length functions, and table formulas. |
-| [WPFF Document Structure](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_DocumentStructure.htm)                                                                                                                    | The file header, the index and packet data areas, the function-code stream's shape, the units glossary, and a complete annotated hex dump of a conforming generic prefix.                                                |
-| [WPFF Single-Byte Characters and Functions](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_SingleByte.htm)                                                                                                         | The character model and the eighty single-byte function codes.                                                                                                                                                           |
-| [WPFF D0 EOL Functions](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_D0-EOL.htm)                                                                                                                                 | The End-of-Line group and, crucially, its "Conversion/Search mappings" column — the specification stating what a converting application should turn each break code into.                                                |
-| [WPFF Fixed-Length Multi-Byte Functions](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_xFixedLength.htm)                                                                                                          | Attribute On/Off, the Extended Character function, and the size of every fixed-length code.                                                                                                                              |
-| [WPFF D3 Paragraph](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_D3-Paragraph.htm) and [D4 Character](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_D4-Character.htm)  | Justification, font face and size changes, colour, and the rest of the paragraph- and character-oriented functions.                                                                                                      |
-| [WPFF D1 Page](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_D1-Page.htm) and [D2 Column](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_D2-Column.htm)                  | Page geometry: the Form function's own page size, and the four margin functions split across the two groups.                                                                                                             |
-| [WPFF DD Style](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_DD-Style.htm) and [DA Display Number](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_DA-DisplayNumber.htm) | The system style number enumeration this package's heading and outline-level recovery rests on, and the paragraph-number display pair.                                                                                   |
-| [WPFF E0 Tab](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_E0-Tab.htm)                                                                                                                                           | The tab definition bitfield, which this group carries in place of a subfunction number.                                                                                                                                  |
-| [WPFF prefix packet catalogue](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_PrefixPkt0-32.htm)                                                                                                                   | The packet types, including the font typeface descriptor layout this package reads a run's font family out of, and the Extended Document Summary the document's own metadata comes from.                                 |
-| [Corel's File Format SDK product page](https://web.archive.org/web/20120125025312/http://apps.corel.com/partners_developers/csp/wordperfect_fileformatsdk.htm)                                                                                              | The provenance: a supported Corel developer product documenting "the entire document format, document prefix and document codes".                                                                                        |
+Everything this package does is derived from the vendor's own documentation, and every non-obvious decision in the source cites the page it comes from — with one deliberate, cited exception: the character-set tables below, which the mirrored SDK pages state the mechanism for but do not themselves tabulate (see the row below and `src/stream/character-sets.ts`'s own top-of-file comment).
+
+| Source                                                                                                                                                                                                                                                      | What it gives                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [WordPerfect File Format SDK help](https://github.com/OneWingedShark/WordPerfect/tree/master/doc/SDK_Help/FileFormats)                                                                                                                                      | The specification itself, mirrored in full: document structure, the prefix packet catalogue, single-byte characters and functions, every variable-length function group, the fixed-length functions, and table formulas.                                                                                                                                                                                                                                                                           |
+| [WPFF Document Structure](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_DocumentStructure.htm)                                                                                                                    | The file header, the index and packet data areas, the function-code stream's shape, the units glossary, and a complete annotated hex dump of a conforming generic prefix.                                                                                                                                                                                                                                                                                                                          |
+| [WPFF Single-Byte Characters and Functions](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_SingleByte.htm)                                                                                                         | The character model and the eighty single-byte function codes.                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+| [WPFF D0 EOL Functions](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_D0-EOL.htm)                                                                                                                                 | The End-of-Line group and, crucially, its "Conversion/Search mappings" column — the specification stating what a converting application should turn each break code into.                                                                                                                                                                                                                                                                                                                          |
+| [WPFF Fixed-Length Multi-Byte Functions](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_xFixedLength.htm)                                                                                                          | Attribute On/Off, the Extended Character function, and the size of every fixed-length code.                                                                                                                                                                                                                                                                                                                                                                                                        |
+| [WPFF D3 Paragraph](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_D3-Paragraph.htm) and [D4 Character](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_D4-Character.htm)  | Justification, font face and size changes, colour, and the rest of the paragraph- and character-oriented functions.                                                                                                                                                                                                                                                                                                                                                                                |
+| [WPFF D1 Page](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_D1-Page.htm) and [D2 Column](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_D2-Column.htm)                  | Page geometry: the Form function's own page size, and the four margin functions split across the two groups.                                                                                                                                                                                                                                                                                                                                                                                       |
+| [WPFF DD Style](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_DD-Style.htm) and [DA Display Number](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_DA-DisplayNumber.htm) | The system style number enumeration this package's heading and outline-level recovery rests on, and the paragraph-number display pair.                                                                                                                                                                                                                                                                                                                                                             |
+| [WPFF E0 Tab](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_E0-Tab.htm)                                                                                                                                           | The tab definition bitfield, which this group carries in place of a subfunction number.                                                                                                                                                                                                                                                                                                                                                                                                            |
+| [WPFF prefix packet catalogue](https://github.com/OneWingedShark/WordPerfect/blob/master/doc/SDK_Help/FileFormats/WPFF_PrefixPkt0-32.htm)                                                                                                                   | The packet types, including the font typeface descriptor layout this package reads a run's font family out of, and the Extended Document Summary the document's own metadata comes from.                                                                                                                                                                                                                                                                                                           |
+| [Corel's File Format SDK product page](https://web.archive.org/web/20120125025312/http://apps.corel.com/partners_developers/csp/wordperfect_fileformatsdk.htm)                                                                                              | The provenance: a supported Corel developer product documenting "the entire document format, document prefix and document codes".                                                                                                                                                                                                                                                                                                                                                                  |
+| [libwpd's `libwpd_internal.cpp`](https://sourceforge.net/p/libwpd/code/ci/master/tree/src/lib/libwpd_internal.cpp) and [`WP6TibetanMap.h`](https://sourceforge.net/p/libwpd/code/ci/master/tree/src/lib/WP6TibetanMap.h)                                    | The character-set-to-Unicode tables for character sets 1 through 14, which the mirrored SDK pages state the mechanism for but do not tabulate. libwpd is the mature LGPL/MPL WordPerfect reader used by AbiWord and LibreOffice; only the factual (set, number) → code point correspondence is transcribed, cross-checked against this package's own independent transcription of the SDK's thirty-two shorthands and against the Unicode Character Database — see `src/stream/character-sets.ts`. |
 
 ## Getting started
 
@@ -135,6 +136,8 @@ The one part of the character model that looks like a bug on first reading, so i
 
 The only mature reader for this format is [libwpd](https://libwpd.sourceforge.net/), which is LGPL C++ — so binding it would forfeit both this family's MIT licensing and its Worker portability in one step, and it could not run in a browser or a Workers isolate at all. Writing the parser by hand against the vendor's own specification is the same bet `markdown-codec` makes against micromark and `pdf-codec` makes against pdf-lib, and here it is not really a bet: the format is documented at byte level by the company that wrote it. An ESLint rule bans importing any libwpd binding by name rather than leaving the decision to memory.
 
+This is a bet against libwpd's _code_, not against citing its _data_: the character-set tables in `src/stream/character-sets.ts` transcribe libwpd's own (character set, character number) → Unicode correspondence, because the mirrored Corel SDK pages state the mechanism but tabulate no character set beyond the thirty-two Default Extended International Characters. A factual mapping table is not the parsing logic the ESLint rule and this section guard against — nothing here imports libwpd, links against it, or depends on its code at runtime.
+
 ## Scope
 
 **Read-only, WordPerfect 6.0 through X6.** Two deliberate exclusions, both decided before any code was written:
@@ -148,7 +151,7 @@ The only mature reader for this format is [libwpd](https://libwpd.sourceforge.ne
 - The file header, with encryption, product type, file type, and major version all checked rather than assumed.
 - The index area and packet data area, with prefix IDs resolvable to packets.
 - The full document-area token stream: characters, all four function-code ranges, prefix ID references, non-deletable data, and gate/size verification.
-- The character model: ASCII, the thirty-two international shorthands, and the Extended Character function for character set 0 and the documented part of set 1.
+- The character model: ASCII, the thirty-two international shorthands, and the Extended Character function for all fourteen of WordPerfect's own named character sets — Multinational, Phonetic Symbols, Box Drawing, Typographic Symbols, Iconic Symbols, Math/Scientific, Math/Scientific Extended, Greek, Hebrew, Cyrillic, Japanese, Tibetan, Arabic, and Arabic Script — transcribed from [libwpd](https://sourceforge.net/p/libwpd/code/ci/master/tree/src/lib/libwpd_internal.cpp)'s own WP6-to-Unicode tables, since the mirrored SDK pages state the (set, number) mechanism but tabulate no character-set table of their own beyond the thirty-two shorthands. See `src/stream/character-sets.ts`'s own top-of-file comment for the source and how it was cross-checked.
 - Paragraph structure from the End-of-Line group, in both its single-byte and multi-byte spellings, using the specification's own conversion table — hard returns become paragraphs, soft returns become spaces, hard end-of-page becomes a `pageBreak` block.
 - Character attributes: bold, italics, underline (plain and double), and strikeout, including the specification's "ignore" bit for a nested duplicate.
 - Font family, from the Desired Font Descriptor packet a Font Face Change names; font size, from a Font Size Change; character colour.
@@ -172,7 +175,6 @@ Everything below is recognised by the tokeniser and skipped by the fold, so a do
 - **The counter groups** (0xD8, 0xD9, 0xDB, 0xDC): setting, numbering-method, increment and decrement carry no text and change no structure this reader models, so only the Display Number group's own paragraph-number pair is read.
 - **Merge codes** (the 0xDE group) and **cross-references** (0xD5). A cross-reference's displayed text survives as ordinary text; its target binding does not. Reported through `wpd/merge-code-dropped` and `wpd/cross-reference-flattened`.
 - **Table formulas** (`WPFF_TableFormulas`): the New Cell Formula embedded subfunction is walked past by its own length so the cells around it still read, but its tokenised formula is not decoded.
-- **Character sets 2 and above**, and the part of set 1 the mirrored SDK pages do not tabulate — the largest remaining fidelity gap, and the one the corpus check measured (see [Evidence](#evidence)): two thirds of the extended characters in real documents name **character set 4** alone, with set 1's untabulated part next, then sets 13, 6, 5, 8, 12 and 3. This is a missing source rather than unfinished work: the mirrored SDK Help states the mechanism — "The high byte is the number of the WordPerfect character set. The low byte contains an offset value into the character set" — and tabulates the thirty-two Default Extended International Characters, but carries no character-set table of its own for any set. It closes when one is transcribed against a citable source, not by inference. An unmapped character renders as U+FFFD and is reported through `wpd/unmapped-character` rather than dropped.
 - **Encrypted documents**, which throw: the specification states that nothing beyond the file header is intelligible without the password, so there is no partial read to offer.
 
 ## Evidence
@@ -190,7 +192,7 @@ What the corpus settles:
 - **The character model is right.** This was the assumption most likely to make every real document read back as nonsense — bytes 1 through 32 are accented-letter shorthands rather than ASCII, so byte `0x20` is the sharp s and a space is the Soft Space function. Real documents read back as correct prose, which they could not do if that were wrong.
 - **Page geometry, tables, metadata, and outline numbering are exercised by real files**, not only by fixtures: a quarter of the corpus states a page size other than US Letter, a third contains a table, and a sixth carries a document summary.
 - **The heading mapping is not.** No document in the corpus uses WordPerfect's own heading styles (system style numbers 68 through 75), so heading recovery is still evidenced by the specification's enumeration alone. The style group itself is heavily exercised — footnote-number, endnote-number, document and hypertext system styles all appear, and all correctly carry no structure.
-- **Character sets 2 and above are the largest remaining fidelity gap, and set 4 is most of it.** Two thirds of the extended characters in the corpus name character set 4, with the untabulated part of set 1 next, then sets 13, 6, 5, 8, 12 and 3. Each renders as U+FFFD today. That makes the missing character-set tables a concrete, measurable piece of work rather than a theoretical one.
+- **Character sets 2 and above were the largest fidelity gap this check found, and set 4 was most of it.** Two thirds of the extended characters in the corpus named character set 4, with the untabulated part of set 1 next, then sets 13, 6, 5, 8, 12 and 3, every one of them rendering as U+FFFD. That measurement is what made the missing character-set tables a concrete, prioritised piece of work rather than a theoretical one — see the section above: all fourteen of WordPerfect's own named character sets now decode, transcribed from libwpd's own tables rather than left as this package's own gap. This bullet stands as the corpus check's original finding, not a re-run against the closed tables; the fixed-byte-sequence tests in `src/stream/characters.test.ts` are what checks the new tables themselves, one decoded example per set against the same cited source.
 
 ### What is still not proven