node-liblzma 1.0.4 → 1.1.4

package/deps/xz/share/doc/xz/xz-file-format.txt

@@ -1,1150 +0,0 @@
-
-The .xz File Format
-===================
-
-Version 1.0.4 (2009-08-27)
-
-
-        0. Preface
-           0.1. Notices and Acknowledgements
-           0.2. Getting the Latest Version
-           0.3. Version History
-        1. Conventions
-           1.1. Byte and Its Representation
-           1.2. Multibyte Integers
-        2. Overall Structure of .xz File
-           2.1. Stream
-                2.1.1. Stream Header
-                       2.1.1.1. Header Magic Bytes
-                       2.1.1.2. Stream Flags
-                       2.1.1.3. CRC32
-                2.1.2. Stream Footer
-                       2.1.2.1. CRC32
-                       2.1.2.2. Backward Size
-                       2.1.2.3. Stream Flags
-                       2.1.2.4. Footer Magic Bytes
-           2.2. Stream Padding
-        3. Block
-           3.1. Block Header
-                3.1.1. Block Header Size
-                3.1.2. Block Flags
-                3.1.3. Compressed Size
-                3.1.4. Uncompressed Size
-                3.1.5. List of Filter Flags
-                3.1.6. Header Padding
-                3.1.7. CRC32
-           3.2. Compressed Data
-           3.3. Block Padding
-           3.4. Check
-        4. Index
-           4.1. Index Indicator
-           4.2. Number of Records
-           4.3. List of Records
-                4.3.1. Unpadded Size
-                4.3.2. Uncompressed Size
-           4.4. Index Padding
-           4.5. CRC32
-        5. Filter Chains
-           5.1. Alignment
-           5.2. Security
-           5.3. Filters
-                5.3.1. LZMA2
-                5.3.2. Branch/Call/Jump Filters for Executables
-                5.3.3. Delta
-                       5.3.3.1. Format of the Encoded Output
-           5.4. Custom Filter IDs
-                5.4.1. Reserved Custom Filter ID Ranges
-        6. Cyclic Redundancy Checks
-        7. References
-
-
-0. Preface
-
-        This document describes the .xz file format (filename suffix
-        ".xz", MIME type "application/x-xz"). It is intended that this
-        this format replace the old .lzma format used by LZMA SDK and
-        LZMA Utils.
-
-
-0.1. Notices and Acknowledgements
-
-        This file format was designed by Lasse Collin
-        <lasse.collin@tukaani.org> and Igor Pavlov.
-
-        Special thanks for helping with this document goes to
-        Ville Koskinen. Thanks for helping with this document goes to
-        Mark Adler, H. Peter Anvin, Mikko Pouru, and Lars Wirzenius.
-
-        This document has been put into the public domain.
-
-
-0.2. Getting the Latest Version
-
-        The latest official version of this document can be downloaded
-        from <http://tukaani.org/xz/xz-file-format.txt>.
-
-        Specific versions of this document have a filename
-        xz-file-format-X.Y.Z.txt where X.Y.Z is the version number.
-        For example, the version 1.0.0 of this document is available
-        at <http://tukaani.org/xz/xz-file-format-1.0.0.txt>.
-
-
-0.3. Version History
-
-        Version   Date          Description
-
-        1.0.4     2009-08-27    Language improvements in Sections 1.2,
-                                2.1.1.2, 3.1.1, 3.1.2, and 5.3.1
-
-        1.0.3     2009-06-05    Spelling fixes in Sections 5.1 and 5.4
-
-        1.0.2     2009-06-04    Typo fixes in Sections 4 and 5.3.1
-
-        1.0.1     2009-06-01    Typo fix in Section 0.3 and minor
-                                clarifications to Sections 2, 2.2,
-                                3.3, 4.4, and 5.3.2
-
-        1.0.0     2009-01-14    The first official version
-
-
-1. Conventions
-
-        The key words "MUST", "MUST NOT", "REQUIRED", "SHOULD",
-        "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
-        document are to be interpreted as described in [RFC-2119].
-
-        Indicating a warning means displaying a message, returning
-        appropriate exit status, or doing something else to let the
-        user know that something worth warning occurred. The operation
-        SHOULD still finish if a warning is indicated.
-
-        Indicating an error means displaying a message, returning
-        appropriate exit status, or doing something else to let the
-        user know that something prevented successfully finishing the
-        operation. The operation MUST be aborted once an error has
-        been indicated.
-
-
-1.1. Byte and Its Representation
-
-        In this document, byte is always 8 bits.
-
-        A "null byte" has all bits unset. That is, the value of a null
-        byte is 0x00.
-
-        To represent byte blocks, this document uses notation that
-        is similar to the notation used in [RFC-1952]:
-
-            +-------+
-            |  Foo  |   One byte.
-            +-------+
-
-            +---+---+
-            |  Foo  |   Two bytes; that is, some of the vertical bars
-            +---+---+   can be missing.
-
-            +=======+
-            |  Foo  |   Zero or more bytes.
-            +=======+
-
-        In this document, a boxed byte or a byte sequence declared
-        using this notation is called "a field". The example field
-        above would be called "the Foo field" or plain "Foo".
-
-        If there are many fields, they may be split to multiple lines.
-        This is indicated with an arrow ("--->"):
-
-            +=====+
-            | Foo |
-            +=====+
-
-                 +=====+
-            ---> | Bar |
-                 +=====+
-
-        The above is equivalent to this:
-
-            +=====+=====+
-            | Foo | Bar |
-            +=====+=====+
-
-
-1.2. Multibyte Integers
-
-        Multibyte integers of static length, such as CRC values,
-        are stored in little endian byte order (least significant
-        byte first).
-
-        When smaller values are more likely than bigger values (for
-        example file sizes), multibyte integers are encoded in a
-        variable-length representation:
-          - Numbers in the range [0, 127] are copied as is, and take
-            one byte of space.
-          - Bigger numbers will occupy two or more bytes. All but the
-            last byte of the multibyte representation have the highest
-            (eighth) bit set.
-
-        For now, the value of the variable-length integers is limited
-        to 63 bits, which limits the encoded size of the integer to
-        nine bytes. These limits may be increased in the future if
-        needed.
-
-        The following C code illustrates encoding and decoding of
-        variable-length integers. The functions return the number of
-        bytes occupied by the integer (1-9), or zero on error.
-
-            #include <stddef.h>
-            #include <inttypes.h>
-
-            size_t
-            encode(uint8_t buf[static 9], uint64_t num)
-            {
-                if (num > UINT64_MAX / 2)
-                    return 0;
-
-                size_t i = 0;
-
-                while (num >= 0x80) {
-                    buf[i++] = (uint8_t)(num) | 0x80;
-                    num >>= 7;
-                }
-
-                buf[i++] = (uint8_t)(num);
-
-                return i;
-            }
-
-            size_t
-            decode(const uint8_t buf[], size_t size_max, uint64_t *num)
-            {
-                if (size_max == 0)
-                    return 0;
-
-                if (size_max > 9)
-                    size_max = 9;
-
-                *num = buf[0] & 0x7F;
-                size_t i = 0;
-
-                while (buf[i++] & 0x80) {
-                    if (i >= size_max || buf[i] == 0x00)
-                        return 0;
-
-                    *num |= (uint64_t)(buf[i] & 0x7F) << (i * 7);
-                }
-
-                return i;
-            }
-
-
-2. Overall Structure of .xz File
-
-        A standalone .xz files consist of one or more Streams which may
-        have Stream Padding between or after them:
-
-            +========+================+========+================+
-            | Stream | Stream Padding | Stream | Stream Padding | ...
-            +========+================+========+================+
-
-        The sizes of Stream and Stream Padding are always multiples
-        of four bytes, thus the size of every valid .xz file MUST be
-        a multiple of four bytes.
-
-        While a typical file contains only one Stream and no Stream
-        Padding, a decoder handling standalone .xz files SHOULD support
-        files that have more than one Stream or Stream Padding.
-
-        In contrast to standalone .xz files, when the .xz file format
-        is used as an internal part of some other file format or
-        communication protocol, it usually is expected that the decoder
-        stops after the first Stream, and doesn't look for Stream
-        Padding or possibly other Streams.
-
-
-2.1. Stream
-
-        +-+-+-+-+-+-+-+-+-+-+-+-+=======+=======+     +=======+
-        |     Stream Header     | Block | Block | ... | Block |
-        +-+-+-+-+-+-+-+-+-+-+-+-+=======+=======+     +=======+
-
-             +=======+-+-+-+-+-+-+-+-+-+-+-+-+
-        ---> | Index |     Stream Footer     |
-             +=======+-+-+-+-+-+-+-+-+-+-+-+-+
-
-        All the above fields have a size that is a multiple of four. If
-        Stream is used as an internal part of another file format, it
-        is RECOMMENDED to make the Stream start at an offset that is
-        a multiple of four bytes.
-
-        Stream Header, Index, and Stream Footer are always present in
-        a Stream. The maximum size of the Index field is 16 GiB (2^34).
-
-        There are zero or more Blocks. The maximum number of Blocks is
-        limited only by the maximum size of the Index field.
-
-        Total size of a Stream MUST be less than 8 EiB (2^63 bytes).
-        The same limit applies to the total amount of uncompressed
-        data stored in a Stream.
-
-        If an implementation supports handling .xz files with multiple
-        concatenated Streams, it MAY apply the above limits to the file
-        as a whole instead of limiting per Stream basis.
-
-
-2.1.1. Stream Header
-
-        +---+---+---+---+---+---+-------+------+--+--+--+--+
-        |  Header Magic Bytes   | Stream Flags |   CRC32   |
-        +---+---+---+---+---+---+-------+------+--+--+--+--+
-
-
-2.1.1.1. Header Magic Bytes
-
-        The first six (6) bytes of the Stream are so called Header
-        Magic Bytes. They can be used to identify the file type.
-
-            Using a C array and ASCII:
-            const uint8_t HEADER_MAGIC[6]
-                    = { 0xFD, '7', 'z', 'X', 'Z', 0x00 };
-
-            In plain hexadecimal:
-            FD 37 7A 58 5A 00
-
-        Notes:
-          - The first byte (0xFD) was chosen so that the files cannot
-            be erroneously detected as being in .lzma format, in which
-            the first byte is in the range [0x00, 0xE0].
-          - The sixth byte (0x00) was chosen to prevent applications
-            from misdetecting the file as a text file.
-
-        If the Header Magic Bytes don't match, the decoder MUST
-        indicate an error.
-
-
-2.1.1.2. Stream Flags
-
-        The first byte of Stream Flags is always a null byte. In the
-        future, this byte may be used to indicate a new Stream version
-        or other Stream properties.
-
-        The second byte of Stream Flags is a bit field:
-
-            Bit(s)  Mask  Description
-             0-3    0x0F  Type of Check (see Section 3.4):
-                              ID    Size      Check name
-                              0x00   0 bytes  None
-                              0x01   4 bytes  CRC32
-                              0x02   4 bytes  (Reserved)
-                              0x03   4 bytes  (Reserved)
-                              0x04   8 bytes  CRC64
-                              0x05   8 bytes  (Reserved)
-                              0x06   8 bytes  (Reserved)
-                              0x07  16 bytes  (Reserved)
-                              0x08  16 bytes  (Reserved)
-                              0x09  16 bytes  (Reserved)
-                              0x0A  32 bytes  SHA-256
-                              0x0B  32 bytes  (Reserved)
-                              0x0C  32 bytes  (Reserved)
-                              0x0D  64 bytes  (Reserved)
-                              0x0E  64 bytes  (Reserved)
-                              0x0F  64 bytes  (Reserved)
-             4-7    0xF0  Reserved for future use; MUST be zero for now.
-
-        Implementations SHOULD support at least the Check IDs 0x00
-        (None) and 0x01 (CRC32). Supporting other Check IDs is
-        OPTIONAL. If an unsupported Check is used, the decoder SHOULD
-        indicate a warning or error.
-
-        If any reserved bit is set, the decoder MUST indicate an error.
-        It is possible that there is a new field present which the
-        decoder is not aware of, and can thus parse the Stream Header
-        incorrectly.
-
-
-2.1.1.3. CRC32
-
-        The CRC32 is calculated from the Stream Flags field. It is
-        stored as an unsigned 32-bit little endian integer. If the
-        calculated value does not match the stored one, the decoder
-        MUST indicate an error.
-
-        The idea is that Stream Flags would always be two bytes, even
-        if new features are needed. This way old decoders will be able
-        to verify the CRC32 calculated from Stream Flags, and thus
-        distinguish between corrupt files (CRC32 doesn't match) and
-        files that the decoder doesn't support (CRC32 matches but
-        Stream Flags has reserved bits set).
-
-
-2.1.2. Stream Footer
-
-        +-+-+-+-+---+---+---+---+-------+------+----------+---------+
-        | CRC32 | Backward Size | Stream Flags | Footer Magic Bytes |
-        +-+-+-+-+---+---+---+---+-------+------+----------+---------+
-
-
-2.1.2.1. CRC32
-
-        The CRC32 is calculated from the Backward Size and Stream Flags
-        fields. It is stored as an unsigned 32-bit little endian
-        integer. If the calculated value does not match the stored one,
-        the decoder MUST indicate an error.
-
-        The reason to have the CRC32 field before the Backward Size and
-        Stream Flags fields is to keep the four-byte fields aligned to
-        a multiple of four bytes.
-
-
-2.1.2.2. Backward Size
-
-        Backward Size is stored as a 32-bit little endian integer,
-        which indicates the size of the Index field as multiple of
-        four bytes, minimum value being four bytes:
-
-            real_backward_size = (stored_backward_size + 1) * 4;
-
-        If the stored value does not match the real size of the Index
-        field, the decoder MUST indicate an error.
-
-        Using a fixed-size integer to store Backward Size makes
-        it slightly simpler to parse the Stream Footer when the
-        application needs to parse the Stream backwards.
-
-
-2.1.2.3. Stream Flags
-
-        This is a copy of the Stream Flags field from the Stream
-        Header. The information stored to Stream Flags is needed
-        when parsing the Stream backwards. The decoder MUST compare
-        the Stream Flags fields in both Stream Header and Stream
-        Footer, and indicate an error if they are not identical.
-
-
-2.1.2.4. Footer Magic Bytes
-
-        As the last step of the decoding process, the decoder MUST
-        verify the existence of Footer Magic Bytes. If they don't
-        match, an error MUST be indicated.
-
-            Using a C array and ASCII:
-            const uint8_t FOOTER_MAGIC[2] = { 'Y', 'Z' };
-
-            In hexadecimal:
-            59 5A
-
-        The primary reason to have Footer Magic Bytes is to make
-        it easier to detect incomplete files quickly, without
-        uncompressing. If the file does not end with Footer Magic Bytes
-        (excluding Stream Padding described in Section 2.2), it cannot
-        be undamaged, unless someone has intentionally appended garbage
-        after the end of the Stream.
-
-
-2.2. Stream Padding
-
-        Only the decoders that support decoding of concatenated Streams
-        MUST support Stream Padding.
-
-        Stream Padding MUST contain only null bytes. To preserve the
-        four-byte alignment of consecutive Streams, the size of Stream
-        Padding MUST be a multiple of four bytes. Empty Stream Padding
-        is allowed. If these requirements are not met, the decoder MUST
-        indicate an error.
-
-        Note that non-empty Stream Padding is allowed at the end of the
-        file; there doesn't need to be a new Stream after non-empty
-        Stream Padding. This can be convenient in certain situations
-        [GNU-tar].
-
-        The possibility of Stream Padding MUST be taken into account
-        when designing an application that parses Streams backwards,
-        and the application supports concatenated Streams.
-
-
-3. Block
-
-        +==============+=================+===============+=======+
-        | Block Header | Compressed Data | Block Padding | Check |
-        +==============+=================+===============+=======+
-
-
-3.1. Block Header
-
-        +-------------------+-------------+=================+
-        | Block Header Size | Block Flags | Compressed Size |
-        +-------------------+-------------+=================+
-
-             +===================+======================+
-        ---> | Uncompressed Size | List of Filter Flags |
-             +===================+======================+
-
-             +================+--+--+--+--+
-        ---> | Header Padding |   CRC32   |
-             +================+--+--+--+--+
-
-
-3.1.1. Block Header Size
-
-        This field overlaps with the Index Indicator field (see
-        Section 4.1).
-
-        This field contains the size of the Block Header field,
-        including the Block Header Size field itself. Valid values are
-        in the range [0x01, 0xFF], which indicate the size of the Block
-        Header as multiples of four bytes, minimum size being eight
-        bytes:
-
-            real_header_size = (encoded_header_size + 1) * 4;
-
-        If a Block Header bigger than 1024 bytes is needed in the
-        future, a new field can be added between the Block Header and
-        Compressed Data fields. The presence of this new field would
-        be indicated in the Block Header field.
-
-
-3.1.2. Block Flags
-
-        The Block Flags field is a bit field:
-
-            Bit(s)  Mask  Description
-             0-1    0x03  Number of filters (1-4)
-             2-5    0x3C  Reserved for future use; MUST be zero for now.
-              6     0x40  The Compressed Size field is present.
-              7     0x80  The Uncompressed Size field is present.
-
-        If any reserved bit is set, the decoder MUST indicate an error.
-        It is possible that there is a new field present which the
-        decoder is not aware of, and can thus parse the Block Header
-        incorrectly.
-
-
-3.1.3. Compressed Size
-
-        This field is present only if the appropriate bit is set in
-        the Block Flags field (see Section 3.1.2).
-
-        The Compressed Size field contains the size of the Compressed
-        Data field, which MUST be non-zero. Compressed Size is stored
-        using the encoding described in Section 1.2. If the Compressed
-        Size doesn't match the size of the Compressed Data field, the
-        decoder MUST indicate an error.
-
-
-3.1.4. Uncompressed Size
-
-        This field is present only if the appropriate bit is set in
-        the Block Flags field (see Section 3.1.2).
-
-        The Uncompressed Size field contains the size of the Block
-        after uncompressing. Uncompressed Size is stored using the
-        encoding described in Section 1.2. If the Uncompressed Size
-        does not match the real uncompressed size, the decoder MUST
-        indicate an error.
-
-        Storing the Compressed Size and Uncompressed Size fields serves
-        several purposes:
-          - The decoder knows how much memory it needs to allocate
-            for a temporary buffer in multithreaded mode.
-          - Simple error detection: wrong size indicates a broken file.
-          - Seeking forwards to a specific location in streamed mode.
-
-        It should be noted that the only reliable way to determine
-        the real uncompressed size is to uncompress the Block,
-        because the Block Header and Index fields may contain
-        (intentionally or unintentionally) invalid information.
-
-
-3.1.5. List of Filter Flags
-
-        +================+================+     +================+
-        | Filter 0 Flags | Filter 1 Flags | ... | Filter n Flags |
-        +================+================+     +================+
-
-        The number of Filter Flags fields is stored in the Block Flags
-        field (see Section 3.1.2).
-
-        The format of each Filter Flags field is as follows:
-
-            +===========+====================+===================+
-            | Filter ID | Size of Properties | Filter Properties |
-            +===========+====================+===================+
-
-        Both Filter ID and Size of Properties are stored using the
-        encoding described in Section 1.2. Size of Properties indicates
-        the size of the Filter Properties field as bytes. The list of
-        officially defined Filter IDs and the formats of their Filter
-        Properties are described in Section 5.3.
-
-        Filter IDs greater than or equal to 0x4000_0000_0000_0000
-        (2^62) are reserved for implementation-specific internal use.
-        These Filter IDs MUST never be used in List of Filter Flags.
-
-
-3.1.6. Header Padding
-
-        This field contains as many null byte as it is needed to make
-        the Block Header have the size specified in Block Header Size.
-        If any of the bytes are not null bytes, the decoder MUST
-        indicate an error. It is possible that there is a new field
-        present which the decoder is not aware of, and can thus parse
-        the Block Header incorrectly.
-
-
-3.1.7. CRC32
-
-        The CRC32 is calculated over everything in the Block Header
-        field except the CRC32 field itself. It is stored as an
-        unsigned 32-bit little endian integer. If the calculated
-        value does not match the stored one, the decoder MUST indicate
-        an error.
-
-        By verifying the CRC32 of the Block Header before parsing the
-        actual contents allows the decoder to distinguish between
-        corrupt and unsupported files.
-
-
-3.2. Compressed Data
-
-        The format of Compressed Data depends on Block Flags and List
-        of Filter Flags. Excluding the descriptions of the simplest
-        filters in Section 5.3, the format of the filter-specific
-        encoded data is out of scope of this document.
-
-
-3.3. Block Padding
-
-        Block Padding MUST contain 0-3 null bytes to make the size of
-        the Block a multiple of four bytes. This can be needed when
-        the size of Compressed Data is not a multiple of four. If any
-        of the bytes in Block Padding are not null bytes, the decoder
-        MUST indicate an error.
-
-
-3.4. Check
-
-        The type and size of the Check field depends on which bits
-        are set in the Stream Flags field (see Section 2.1.1.2).
-
-        The Check, when used, is calculated from the original
-        uncompressed data. If the calculated Check does not match the
-        stored one, the decoder MUST indicate an error. If the selected
-        type of Check is not supported by the decoder, it SHOULD
-        indicate a warning or error.
-
-
-4. Index
-
-        +-----------------+===================+
-        | Index Indicator | Number of Records |
-        +-----------------+===================+
-
-             +=================+===============+-+-+-+-+
-        ---> | List of Records | Index Padding | CRC32 |
-             +=================+===============+-+-+-+-+
-
-        Index serves several purposes. Using it, one can
-          - verify that all Blocks in a Stream have been processed;
-          - find out the uncompressed size of a Stream; and
-          - quickly access the beginning of any Block (random access).
-
-
-4.1. Index Indicator
-
-        This field overlaps with the Block Header Size field (see
-        Section 3.1.1). The value of Index Indicator is always 0x00.
-
-
-4.2. Number of Records
-
-        This field indicates how many Records there are in the List
-        of Records field, and thus how many Blocks there are in the
-        Stream. The value is stored using the encoding described in
-        Section 1.2. If the decoder has decoded all the Blocks of the
-        Stream, and then notices that the Number of Records doesn't
-        match the real number of Blocks, the decoder MUST indicate an
-        error.
-
-
-4.3. List of Records
-
-        List of Records consists of as many Records as indicated by the
-        Number of Records field:
-
-            +========+========+
-            | Record | Record | ...
-            +========+========+
-
-        Each Record contains information about one Block:
-
-            +===============+===================+
-            | Unpadded Size | Uncompressed Size |
-            +===============+===================+
-
-        If the decoder has decoded all the Blocks of the Stream, it
-        MUST verify that the contents of the Records match the real
-        Unpadded Size and Uncompressed Size of the respective Blocks.
-
-        Implementation hint: It is possible to verify the Index with
-        constant memory usage by calculating for example SHA-256 of
-        both the real size values and the List of Records, then
-        comparing the hash values. Implementing this using
-        non-cryptographic hash like CRC32 SHOULD be avoided unless
-        small code size is important.
-
-        If the decoder supports random-access reading, it MUST verify
-        that Unpadded Size and Uncompressed Size of every completely
-        decoded Block match the sizes stored in the Index. If only
-        partial Block is decoded, the decoder MUST verify that the
-        processed sizes don't exceed the sizes stored in the Index.
-
-
-4.3.1. Unpadded Size
-
-        This field indicates the size of the Block excluding the Block
-        Padding field. That is, Unpadded Size is the size of the Block
-        Header, Compressed Data, and Check fields. Unpadded Size is
-        stored using the encoding described in Section 1.2. The value
-        MUST never be zero; with the current structure of Blocks, the
-        actual minimum value for Unpadded Size is five.
-
-        Implementation note: Because the size of the Block Padding
-        field is not included in Unpadded Size, calculating the total
-        size of a Stream or doing random-access reading requires
-        calculating the actual size of the Blocks by rounding Unpadded
-        Sizes up to the next multiple of four.
-
-        The reason to exclude Block Padding from Unpadded Size is to
-        ease making a raw copy of Compressed Data without Block
-        Padding. This can be useful, for example, if someone wants
-        to convert Streams to some other file format quickly.
-
-
-4.3.2. Uncompressed Size
-
-        This field indicates the Uncompressed Size of the respective
-        Block as bytes. The value is stored using the encoding
-        described in Section 1.2.
-
-
-4.4. Index Padding
-
-        This field MUST contain 0-3 null bytes to pad the Index to
-        a multiple of four bytes. If any of the bytes are not null
-        bytes, the decoder MUST indicate an error.
-
-
-4.5. CRC32
-
-        The CRC32 is calculated over everything in the Index field
-        except the CRC32 field itself. The CRC32 is stored as an
-        unsigned 32-bit little endian integer. If the calculated
-        value does not match the stored one, the decoder MUST indicate
-        an error.
-
-
-5. Filter Chains
-
-        The Block Flags field defines how many filters are used. When
-        more than one filter is used, the filters are chained; that is,
-        the output of one filter is the input of another filter. The
-        following figure illustrates the direction of data flow.
-
-                    v   Uncompressed Data   ^
-                    |       Filter 0        |
-            Encoder |       Filter 1        | Decoder
-                    |       Filter n        |
-                    v    Compressed Data    ^
-
-
-5.1. Alignment
-
-        Alignment of uncompressed input data is usually the job of
-        the application producing the data. For example, to get the
-        best results, an archiver tool should make sure that all
-        PowerPC executable files in the archive stream start at
-        offsets that are multiples of four bytes.
-
-        Some filters, for example LZMA2, can be configured to take
-        advantage of specified alignment of input data. Note that
-        taking advantage of aligned input can be beneficial also when
-        a filter is not the first filter in the chain. For example,
-        if you compress PowerPC executables, you may want to use the
-        PowerPC filter and chain that with the LZMA2 filter. Because
-        not only the input but also the output alignment of the PowerPC
-        filter is four bytes, it is now beneficial to set LZMA2
-        settings so that the LZMA2 encoder can take advantage of its
-        four-byte-aligned input data.
-
-        The output of the last filter in the chain is stored to the
-        Compressed Data field, which is is guaranteed to be aligned
-        to a multiple of four bytes relative to the beginning of the
-        Stream. This can increase
-          - speed, if the filtered data is handled multiple bytes at
-            a time by the filter-specific encoder and decoder,
-            because accessing aligned data in computer memory is
-            usually faster; and
-          - compression ratio, if the output data is later compressed
-            with an external compression tool.
-
-
-5.2. Security
-
-        If filters would be allowed to be chained freely, it would be
-        possible to create malicious files, that would be very slow to
-        decode. Such files could be used to create denial of service
-        attacks.
-
-        Slow files could occur when multiple filters are chained:
-
-            v   Compressed input data
-            |   Filter 1 decoder (last filter)
-            |   Filter 0 decoder (non-last filter)
-            v   Uncompressed output data
-
-        The decoder of the last filter in the chain produces a lot of
-        output from little input. Another filter in the chain takes the
-        output of the last filter, and produces very little output
-        while consuming a lot of input. As a result, a lot of data is
-        moved inside the filter chain, but the filter chain as a whole
-        gets very little work done.
-
-        To prevent this kind of slow files, there are restrictions on
-        how the filters can be chained. These restrictions MUST be
-        taken into account when designing new filters.
-
-        The maximum number of filters in the chain has been limited to
-        four, thus there can be at maximum of three non-last filters.
-        Of these three non-last filters, only two are allowed to change
-        the size of the data.
-
-        The non-last filters, that change the size of the data, MUST
-        have a limit how much the decoder can compress the data: the
-        decoder SHOULD produce at least n bytes of output when the
-        filter is given 2n bytes of input. This  limit is not
-        absolute, but significant deviations MUST be avoided.
-
-        The above limitations guarantee that if the last filter in the
-        chain produces 4n bytes of output, the chain as a whole will
-        produce at least n bytes of output.
-
-
-5.3. Filters
-
-5.3.1. LZMA2
-
-        LZMA (Lempel-Ziv-Markov chain-Algorithm) is a general-purpose
-        compression algorithm with high compression ratio and fast
-        decompression. LZMA is based on LZ77 and range coding
-        algorithms.
-
-        LZMA2 is an extension on top of the original LZMA. LZMA2 uses
-        LZMA internally, but adds support for flushing the encoder,
-        uncompressed chunks, eases stateful decoder implementations,
-        and improves support for multithreading. Thus, the plain LZMA
-        will not be supported in this file format.
-
-            Filter ID:                  0x21
-            Size of Filter Properties:  1 byte
-            Changes size of data:       Yes
-            Allow as a non-last filter: No
-            Allow as the last filter:   Yes
-
-            Preferred alignment:
-                Input data:             Adjustable to 1/2/4/8/16 byte(s)
-                Output data:            1 byte
-
-        The format of the one-byte Filter Properties field is as
-        follows:
-
-            Bits   Mask   Description
-            0-5    0x3F   Dictionary Size
-            6-7    0xC0   Reserved for future use; MUST be zero for now.
-
-        Dictionary Size is encoded with one-bit mantissa and five-bit
-        exponent. The smallest dictionary size is 4 KiB and the biggest
-        is 4 GiB.
-
-            Raw value   Mantissa   Exponent   Dictionary size
-                0           2         11         4 KiB
-                1           3         11         6 KiB
-                2           2         12         8 KiB
-                3           3         12        12 KiB
-                4           2         13        16 KiB
-                5           3         13        24 KiB
-                6           2         14        32 KiB
-              ...         ...        ...      ...
-               35           3         27       768 MiB
-               36           2         28      1024 MiB
-               37           3         29      1536 MiB
-               38           2         30      2048 MiB
-               39           3         30      3072 MiB
-               40           2         31      4096 MiB - 1 B
-
-        Instead of having a table in the decoder, the dictionary size
-        can be decoded using the following C code:
-
-            const uint8_t bits = get_dictionary_flags() & 0x3F;
-            if (bits > 40)
-                return DICTIONARY_TOO_BIG; // Bigger than 4 GiB
-
-            uint32_t dictionary_size;
-            if (bits == 40) {
-                dictionary_size = UINT32_MAX;
-            } else {
-                dictionary_size = 2 | (bits & 1);
-                dictionary_size <<= bits / 2 + 11;
-            }
-
-
-5.3.2. Branch/Call/Jump Filters for Executables
-
-        These filters convert relative branch, call, and jump
-        instructions to their absolute counterparts in executable
-        files. This conversion increases redundancy and thus
-        compression ratio.
-
-            Size of Filter Properties:  0 or 4 bytes
-            Changes size of data:       No
-            Allow as a non-last filter: Yes
-            Allow as the last filter:   No
-
-        Below is the list of filters in this category. The alignment
-        is the same for both input and output data.
-
-            Filter ID   Alignment   Description
-              0x04       1 byte     x86 filter (BCJ)
-              0x05       4 bytes    PowerPC (big endian) filter
-              0x06      16 bytes    IA64 filter
-              0x07       4 bytes    ARM (little endian) filter
-              0x08       2 bytes    ARM Thumb (little endian) filter
-              0x09       4 bytes    SPARC filter
-
-        If the size of Filter Properties is four bytes, the Filter
-        Properties field contains the start offset used for address
-        conversions. It is stored as an unsigned 32-bit little endian
-        integer. The start offset MUST be a multiple of the alignment
-        of the filter as listed in the table above; if it isn't, the
-        decoder MUST indicate an error. If the size of Filter
-        Properties is zero, the start offset is zero.
-
-        Setting the start offset may be useful if an executable has
-        multiple sections, and there are many cross-section calls.
-        Taking advantage of this feature usually requires usage of
-        the Subblock filter, whose design is not complete yet.
-
-
-5.3.3. Delta
-
-        The Delta filter may increase compression ratio when the value
-        of the next byte correlates with the value of an earlier byte
-        at specified distance.
-
-            Filter ID:                  0x03
-            Size of Filter Properties:  1 byte
-            Changes size of data:       No
-            Allow as a non-last filter: Yes
-            Allow as the last filter:   No
-
-            Preferred alignment:
-                Input data:             1 byte
-                Output data:            Same as the original input data
-
-        The Properties byte indicates the delta distance, which can be
-        1-256 bytes backwards from the current byte: 0x00 indicates
-        distance of 1 byte and 0xFF distance of 256 bytes.
-
-
-5.3.3.1. Format of the Encoded Output
-
-        The code below illustrates both encoding and decoding with
-        the Delta filter.
-
-            // Distance is in the range [1, 256].
-            const unsigned int distance = get_properties_byte() + 1;
-            uint8_t pos = 0;
-            uint8_t delta[256];
-
-            memset(delta, 0, sizeof(delta));
-
-            while (1) {
-                const int byte = read_byte();
-                if (byte == EOF)
-                    break;
-
-                uint8_t tmp = delta[(uint8_t)(distance + pos)];
-                if (is_encoder) {
-                    tmp = (uint8_t)(byte) - tmp;
-                    delta[pos] = (uint8_t)(byte);
-                } else {
-                    tmp = (uint8_t)(byte) + tmp;
-                    delta[pos] = tmp;
-                }
-
-                write_byte(tmp);
-                --pos;
-            }
-
-
-5.4. Custom Filter IDs
-
-        If a developer wants to use custom Filter IDs, he has two
-        choices. The first choice is to contact Lasse Collin and ask
-        him to allocate a range of IDs for the developer.
-
-        The second choice is to generate a 40-bit random integer,
-        which the developer can use as his personal Developer ID.
-        To minimize the risk of collisions, Developer ID has to be
-        a randomly generated integer, not manually selected "hex word".
-        The following command, which works on many free operating
-        systems, can be used to generate Developer ID:
-
-            dd if=/dev/urandom bs=5 count=1 | hexdump
-
-        The developer can then use his Developer ID to create unique
-        (well, hopefully unique) Filter IDs.
-
-            Bits    Mask                    Description
-             0-15   0x0000_0000_0000_FFFF   Filter ID
-            16-55   0x00FF_FFFF_FFFF_0000   Developer ID
-            56-62   0x3F00_0000_0000_0000   Static prefix: 0x3F
-
-        The resulting 63-bit integer will use 9 bytes of space when
-        stored using the encoding described in Section 1.2. To get
-        a shorter ID, see the beginning of this Section how to
-        request a custom ID range.
-
-
-5.4.1. Reserved Custom Filter ID Ranges
-
-        Range                       Description
-        0x0000_0300 - 0x0000_04FF   Reserved to ease .7z compatibility
-        0x0002_0000 - 0x0007_FFFF   Reserved to ease .7z compatibility
-        0x0200_0000 - 0x07FF_FFFF   Reserved to ease .7z compatibility
-
-
-6. Cyclic Redundancy Checks
-
-        There are several incompatible variations to calculate CRC32
-        and CRC64. For simplicity and clarity, complete examples are
-        provided to calculate the checks as they are used in this file
-        format. Implementations MAY use different code as long as it
-        gives identical results.
-
-        The program below reads data from standard input, calculates
-        the CRC32 and CRC64 values, and prints the calculated values
-        as big endian hexadecimal strings to standard output.
-
-            #include <stddef.h>
-            #include <inttypes.h>
-            #include <stdio.h>
-
-            uint32_t crc32_table[256];
-            uint64_t crc64_table[256];
-
-            void
-            init(void)
-            {
-                static const uint32_t poly32 = UINT32_C(0xEDB88320);
-                static const uint64_t poly64
-                        = UINT64_C(0xC96C5795D7870F42);
-
-                for (size_t i = 0; i < 256; ++i) {
-                    uint32_t crc32 = i;
-                    uint64_t crc64 = i;
-
-                    for (size_t j = 0; j < 8; ++j) {
-                        if (crc32 & 1)
-                            crc32 = (crc32 >> 1) ^ poly32;
-                        else
-                            crc32 >>= 1;
-
-                        if (crc64 & 1)
-                            crc64 = (crc64 >> 1) ^ poly64;
-                        else
-                            crc64 >>= 1;
-                    }
-
-                    crc32_table[i] = crc32;
-                    crc64_table[i] = crc64;
-                }
-            }
-
-            uint32_t
-            crc32(const uint8_t *buf, size_t size, uint32_t crc)
-            {
-                crc = ~crc;
-                for (size_t i = 0; i < size; ++i)
-                    crc = crc32_table[buf[i] ^ (crc & 0xFF)]
-                            ^ (crc >> 8);
-                return ~crc;
-            }
-
-            uint64_t
-            crc64(const uint8_t *buf, size_t size, uint64_t crc)
-            {
-                crc = ~crc;
-                for (size_t i = 0; i < size; ++i)
-                    crc = crc64_table[buf[i] ^ (crc & 0xFF)]
-                            ^ (crc >> 8);
-                return ~crc;
-            }
-
-            int
-            main()
-            {
-                init();
-
-                uint32_t value32 = 0;
-                uint64_t value64 = 0;
-                uint64_t total_size = 0;
-                uint8_t buf[8192];
-
-                while (1) {
-                    const size_t buf_size
-                            = fread(buf, 1, sizeof(buf), stdin);
-                    if (buf_size == 0)
-                        break;
-
-                    total_size += buf_size;
-                    value32 = crc32(buf, buf_size, value32);
-                    value64 = crc64(buf, buf_size, value64);
-                }
-
-                printf("Bytes:  %" PRIu64 "\n", total_size);
-                printf("CRC-32: 0x%08" PRIX32 "\n", value32);
-                printf("CRC-64: 0x%016" PRIX64 "\n", value64);
-
-                return 0;
-            }
-
-
-7. References
-
-        LZMA SDK - The original LZMA implementation
-        http://7-zip.org/sdk.html
-
-        LZMA Utils - LZMA adapted to POSIX-like systems
-        http://tukaani.org/lzma/
-
-        XZ Utils - The next generation of LZMA Utils
-        http://tukaani.org/xz/
-
-        [RFC-1952]
-        GZIP file format specification version 4.3
-        http://www.ietf.org/rfc/rfc1952.txt
-          - Notation of byte boxes in section "2.1. Overall conventions"
-
-        [RFC-2119]
-        Key words for use in RFCs to Indicate Requirement Levels
-        http://www.ietf.org/rfc/rfc2119.txt
-
-        [GNU-tar]
-        GNU tar 1.21 manual
-        http://www.gnu.org/software/tar/manual/html_node/Blocking-Factor.html
-          - Node 9.4.2 "Blocking Factor", paragraph that begins
-            "gzip will complain about trailing garbage"
-          - Note that this URL points to the latest version of the
-            manual, and may some day not contain the note which is in
-            1.21. For the exact version of the manual, download GNU
-            tar 1.21: ftp://ftp.gnu.org/pub/gnu/tar/tar-1.21.tar.gz
-