zipremove 0.5.0__tar.gz → 0.6.0__tar.gz

{zipremove-0.5.0/src/zipremove.egg-info → zipremove-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: zipremove
-Version: 0.5.0
+Version: 0.6.0
 Summary: Extend `zipfile` with `remove`-related functionalities
 Home-page: https://github.com/danny0838/zipremove
 Author: Danny Lin
@@ -45,14 +45,10 @@ This package extends `zipfile` with `remove`-related functionalities.
 
 * `ZipFile.remove(zinfo_or_arcname)`
 
-   Removes a member from the archive.  *zinfo_or_arcname* may be the full path
-   of the member or a `ZipInfo` instance.
-
-   If multiple members share the same full path, only one is removed when
-   a path is provided.
-
-   This does not physically remove the local file entry from the archive.
-   Call `repack` afterwards to reclaim space.
+   Removes a member entry from the archive's central directory.
+   *zinfo_or_arcname* may be the full path of the member or a `ZipInfo`
+   instance.  If multiple members share the same full path and the path is
+   provided, only one of them is removed.
 
    The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'``.
 
@@ -60,42 +56,49 @@ This package extends `zipfile` with `remove`-related functionalities.
 
    Calling `remove` on a closed ZipFile will raise a `ValueError`.
 
+   > **Note:** 
+   > This method only removes the member's entry from the central directory,
+   > making it inaccessible to most tools.  The member's local file entry,
+   > including content and metadata, remains in the archive and is still
+   > recoverable using forensic tools.  Call `repack` afterwards to completely
+   > remove the member and reclaim space.
+
 * `ZipFile.repack(removed=None, *, strict_descriptor=False[, chunk_size])`
 
-   Rewrites the archive to remove stale local file entries, shrinking its file
-   size.
+   Rewrites the archive to remove unreferenced local file entries, shrinking
+   its file size.  The archive must be opened with mode ``'a'``.
 
    If *removed* is provided, it must be a sequence of `ZipInfo` objects
-   representing removed entries; only their corresponding local file entries
-   will be removed.
-
-   If *removed* is not provided, the archive is scanned to identify and remove
-   local file entries that are no longer referenced in the central directory.
-   The algorithm assumes that local file entries (and the central directory,
-   which is mostly treated as the "last entry") are stored consecutively:
-
-   1. Data before the first referenced entry is removed only when it appears to
-      be a sequence of consecutive entries with no extra following bytes; extra
-      preceding bytes are preserved.
-   2. Data between referenced entries is removed only when it appears to
-      be a sequence of consecutive entries with no extra preceding bytes; extra
-      following bytes are preserved.
-   3. Entries must not overlap. If any entry's data overlaps with another, a
-      `BadZipFile` error is raised and no changes are made.
-
-   When scanning, setting `strict_descriptor=True` disables detection of any
-   entry using an unsigned data descriptor (deprecated in the ZIP specification
-   since version 6.3.0, released on 2006-09-29, and used only by some legacy
-   tools). This improves performance, but may cause some stale entries to be
-   preserved.
+   representing the recently removed members, and only their corresponding
+   local file entries will be removed.  Otherwise, the archive is scanned to
+   locate and remove local file entries that are no longer referenced in the
+   central directory.
+
+   When scanning, setting ``strict_descriptor=True`` disables detection of any
+   entry using an unsigned data descriptor (a format deprecated by the ZIP
+   specification since version 6.3.0, released on 2006-09-29, and used only by
+   some legacy tools), which is significantly slower to scan—around 100 to
+   1000 times in the worst case. This does not affect performance on entries
+   without such feature.
 
    *chunk_size* may be specified to control the buffer size when moving
    entry data (default is 1 MiB).
 
-   The archive must be opened with mode ``'a'``.
-
    Calling `repack` on a closed ZipFile will raise a `ValueError`.
 
+   > **Note:** 
+   > The scanning algorithm is heuristic-based and assumes that the ZIP file
+   > is normally structured—for example, with local file entries stored
+   > consecutively, without overlap or interleaved binary data.  Prepended
+   > binary data, such as a self-extractor stub, is recognized and preserved
+   > unless it happens to contain bytes that coincidentally resemble a valid
+   > local file entry in multiple respects—an extremely rare case. Embedded
+   > ZIP payloads are also handled correctly, as long as they follow normal
+   > structure.  However, the algorithm does not guarantee correctness or
+   > safety on untrusted or intentionally crafted input.  It is generally
+   > recommended to provide the *removed* argument for better reliability and
+   > performance.
+
 * `ZipFile.copy(zinfo_or_arcname, new_arcname[, chunk_size])`
 
    Copies a member *zinfo_or_arcname* to *new_arcname* in the archive.

{zipremove-0.5.0 → zipremove-0.6.0}/README.md

@@ -11,14 +11,10 @@ This package extends `zipfile` with `remove`-related functionalities.
 
 * `ZipFile.remove(zinfo_or_arcname)`
 
-   Removes a member from the archive.  *zinfo_or_arcname* may be the full path
-   of the member or a `ZipInfo` instance.
-
-   If multiple members share the same full path, only one is removed when
-   a path is provided.
-
-   This does not physically remove the local file entry from the archive.
-   Call `repack` afterwards to reclaim space.
+   Removes a member entry from the archive's central directory.
+   *zinfo_or_arcname* may be the full path of the member or a `ZipInfo`
+   instance.  If multiple members share the same full path and the path is
+   provided, only one of them is removed.
 
    The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'``.
 
@@ -26,42 +22,49 @@ This package extends `zipfile` with `remove`-related functionalities.
 
    Calling `remove` on a closed ZipFile will raise a `ValueError`.
 
+   > **Note:** 
+   > This method only removes the member's entry from the central directory,
+   > making it inaccessible to most tools.  The member's local file entry,
+   > including content and metadata, remains in the archive and is still
+   > recoverable using forensic tools.  Call `repack` afterwards to completely
+   > remove the member and reclaim space.
+
 * `ZipFile.repack(removed=None, *, strict_descriptor=False[, chunk_size])`
 
-   Rewrites the archive to remove stale local file entries, shrinking its file
-   size.
+   Rewrites the archive to remove unreferenced local file entries, shrinking
+   its file size.  The archive must be opened with mode ``'a'``.
 
    If *removed* is provided, it must be a sequence of `ZipInfo` objects
-   representing removed entries; only their corresponding local file entries
-   will be removed.
-
-   If *removed* is not provided, the archive is scanned to identify and remove
-   local file entries that are no longer referenced in the central directory.
-   The algorithm assumes that local file entries (and the central directory,
-   which is mostly treated as the "last entry") are stored consecutively:
-
-   1. Data before the first referenced entry is removed only when it appears to
-      be a sequence of consecutive entries with no extra following bytes; extra
-      preceding bytes are preserved.
-   2. Data between referenced entries is removed only when it appears to
-      be a sequence of consecutive entries with no extra preceding bytes; extra
-      following bytes are preserved.
-   3. Entries must not overlap. If any entry's data overlaps with another, a
-      `BadZipFile` error is raised and no changes are made.
-
-   When scanning, setting `strict_descriptor=True` disables detection of any
-   entry using an unsigned data descriptor (deprecated in the ZIP specification
-   since version 6.3.0, released on 2006-09-29, and used only by some legacy
-   tools). This improves performance, but may cause some stale entries to be
-   preserved.
+   representing the recently removed members, and only their corresponding
+   local file entries will be removed.  Otherwise, the archive is scanned to
+   locate and remove local file entries that are no longer referenced in the
+   central directory.
+
+   When scanning, setting ``strict_descriptor=True`` disables detection of any
+   entry using an unsigned data descriptor (a format deprecated by the ZIP
+   specification since version 6.3.0, released on 2006-09-29, and used only by
+   some legacy tools), which is significantly slower to scan—around 100 to
+   1000 times in the worst case. This does not affect performance on entries
+   without such feature.
 
    *chunk_size* may be specified to control the buffer size when moving
    entry data (default is 1 MiB).
 
-   The archive must be opened with mode ``'a'``.
-
    Calling `repack` on a closed ZipFile will raise a `ValueError`.
 
+   > **Note:** 
+   > The scanning algorithm is heuristic-based and assumes that the ZIP file
+   > is normally structured—for example, with local file entries stored
+   > consecutively, without overlap or interleaved binary data.  Prepended
+   > binary data, such as a self-extractor stub, is recognized and preserved
+   > unless it happens to contain bytes that coincidentally resemble a valid
+   > local file entry in multiple respects—an extremely rare case. Embedded
+   > ZIP payloads are also handled correctly, as long as they follow normal
+   > structure.  However, the algorithm does not guarantee correctness or
+   > safety on untrusted or intentionally crafted input.  It is generally
+   > recommended to provide the *removed* argument for better reliability and
+   > performance.
+
 * `ZipFile.copy(zinfo_or_arcname, new_arcname[, chunk_size])`
 
    Copies a member *zinfo_or_arcname* to *new_arcname* in the archive.

{zipremove-0.5.0 → zipremove-0.6.0}/setup.cfg

@@ -1,6 +1,6 @@
 [metadata]
 name = zipremove
-version = 0.5.0
+version = 0.6.0
 author = Danny Lin
 author_email = danny0838@gmail.com
 url = https://github.com/danny0838/zipremove

{zipremove-0.5.0 → zipremove-0.6.0}/src/zipremove/__init__.py

@@ -177,7 +177,8 @@ class _ZipRepacker:
 
         Side effects:
             - Modifies the ZIP file in place.
-            - Updates zfile.start_dir to account for removed data.
+            - Updates zfile.start_dir and zfile.data_offset to account for
+              removed data.
             - Sets zfile._didModify to True.
             - Updates header_offset and clears _end_offset of referenced
               ZipInfo instances.
@@ -282,6 +283,15 @@ class _ZipRepacker:
         zfile.start_dir -= entry_offset
         zfile._didModify = True
 
+        # polyfill: update _data_offset if exists
+        if getattr(zfile, '_data_offset', None):
+            try:
+                offset = filelist[0].header_offset
+            except IndexError:
+                offset = zfile.start_dir
+            if offset < zfile._data_offset:
+                zfile._data_offset = offset
+
         # polyfill: clear ZipInfo._end_offset if exists
         # (Python >= 3.8 with fix #109858)
         if hasattr(ZipInfo, '_end_offset'):
@@ -300,7 +310,8 @@ class _ZipRepacker:
                     return entry_size
         return 0
 
-    def _iter_scan_signature(self, fp, signature, start_offset, end_offset, chunk_size=4096):
+    def _iter_scan_signature(self, fp, signature, start_offset, end_offset,
+                             chunk_size=io.DEFAULT_BUFFER_SIZE):
         sig_len = len(signature)
         remainder = b''
         pos = start_offset
@@ -506,7 +517,8 @@ class _ZipRepacker:
 
         return crc, compress_size, file_size, dd_size
 
-    def _trace_compressed_block_end(self, fp, offset, end_offset, decompressor, chunk_size=4096):
+    def _trace_compressed_block_end(self, fp, offset, end_offset, decompressor,
+                                    chunk_size=io.DEFAULT_BUFFER_SIZE):
         fp.seek(offset)
         read_size = 0
         while True:

{zipremove-0.5.0 → zipremove-0.6.0/src/zipremove.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: zipremove
-Version: 0.5.0
+Version: 0.6.0
 Summary: Extend `zipfile` with `remove`-related functionalities
 Home-page: https://github.com/danny0838/zipremove
 Author: Danny Lin
@@ -45,14 +45,10 @@ This package extends `zipfile` with `remove`-related functionalities.
 
 * `ZipFile.remove(zinfo_or_arcname)`
 
-   Removes a member from the archive.  *zinfo_or_arcname* may be the full path
-   of the member or a `ZipInfo` instance.
-
-   If multiple members share the same full path, only one is removed when
-   a path is provided.
-
-   This does not physically remove the local file entry from the archive.
-   Call `repack` afterwards to reclaim space.
+   Removes a member entry from the archive's central directory.
+   *zinfo_or_arcname* may be the full path of the member or a `ZipInfo`
+   instance.  If multiple members share the same full path and the path is
+   provided, only one of them is removed.
 
    The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'``.
 
@@ -60,42 +56,49 @@ This package extends `zipfile` with `remove`-related functionalities.
 
    Calling `remove` on a closed ZipFile will raise a `ValueError`.
 
+   > **Note:** 
+   > This method only removes the member's entry from the central directory,
+   > making it inaccessible to most tools.  The member's local file entry,
+   > including content and metadata, remains in the archive and is still
+   > recoverable using forensic tools.  Call `repack` afterwards to completely
+   > remove the member and reclaim space.
+
 * `ZipFile.repack(removed=None, *, strict_descriptor=False[, chunk_size])`
 
-   Rewrites the archive to remove stale local file entries, shrinking its file
-   size.
+   Rewrites the archive to remove unreferenced local file entries, shrinking
+   its file size.  The archive must be opened with mode ``'a'``.
 
    If *removed* is provided, it must be a sequence of `ZipInfo` objects
-   representing removed entries; only their corresponding local file entries
-   will be removed.
-
-   If *removed* is not provided, the archive is scanned to identify and remove
-   local file entries that are no longer referenced in the central directory.
-   The algorithm assumes that local file entries (and the central directory,
-   which is mostly treated as the "last entry") are stored consecutively:
-
-   1. Data before the first referenced entry is removed only when it appears to
-      be a sequence of consecutive entries with no extra following bytes; extra
-      preceding bytes are preserved.
-   2. Data between referenced entries is removed only when it appears to
-      be a sequence of consecutive entries with no extra preceding bytes; extra
-      following bytes are preserved.
-   3. Entries must not overlap. If any entry's data overlaps with another, a
-      `BadZipFile` error is raised and no changes are made.
-
-   When scanning, setting `strict_descriptor=True` disables detection of any
-   entry using an unsigned data descriptor (deprecated in the ZIP specification
-   since version 6.3.0, released on 2006-09-29, and used only by some legacy
-   tools). This improves performance, but may cause some stale entries to be
-   preserved.
+   representing the recently removed members, and only their corresponding
+   local file entries will be removed.  Otherwise, the archive is scanned to
+   locate and remove local file entries that are no longer referenced in the
+   central directory.
+
+   When scanning, setting ``strict_descriptor=True`` disables detection of any
+   entry using an unsigned data descriptor (a format deprecated by the ZIP
+   specification since version 6.3.0, released on 2006-09-29, and used only by
+   some legacy tools), which is significantly slower to scan—around 100 to
+   1000 times in the worst case. This does not affect performance on entries
+   without such feature.
 
    *chunk_size* may be specified to control the buffer size when moving
    entry data (default is 1 MiB).
 
-   The archive must be opened with mode ``'a'``.
-
    Calling `repack` on a closed ZipFile will raise a `ValueError`.
 
+   > **Note:** 
+   > The scanning algorithm is heuristic-based and assumes that the ZIP file
+   > is normally structured—for example, with local file entries stored
+   > consecutively, without overlap or interleaved binary data.  Prepended
+   > binary data, such as a self-extractor stub, is recognized and preserved
+   > unless it happens to contain bytes that coincidentally resemble a valid
+   > local file entry in multiple respects—an extremely rare case. Embedded
+   > ZIP payloads are also handled correctly, as long as they follow normal
+   > structure.  However, the algorithm does not guarantee correctness or
+   > safety on untrusted or intentionally crafted input.  It is generally
+   > recommended to provide the *removed* argument for better reliability and
+   > performance.
+
 * `ZipFile.copy(zinfo_or_arcname, new_arcname[, chunk_size])`
 
    Copies a member *zinfo_or_arcname* to *new_arcname* in the archive.

{zipremove-0.5.0 → zipremove-0.6.0}/tests/test_zipfile.py

@@ -436,45 +436,49 @@ class AbstractRemoveTests(RepackHelperMixin):
         # suppress duplicated name warning
         with warnings.catch_warnings():
             warnings.simplefilter("ignore")
-
             zinfos = self._prepare_zip_from_test_files(TESTFN, test_files)
-            with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
-                zh.remove('file.txt')
 
-                # check infolist
-                self.assertEqual(
-                    [ComparableZipInfo(zi) for zi in zh.infolist()],
-                    [ComparableZipInfo(zi) for zi in [zinfos[0], zinfos[2]]],
-                )
+        with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
+            zh.remove('file.txt')
+
+            # check infolist
+            self.assertEqual(
+                [ComparableZipInfo(zi) for zi in zh.infolist()],
+                [ComparableZipInfo(zi) for zi in [zinfos[0], zinfos[2]]],
+            )
 
-                # check NameToInfo cache
-                self.assertEqual(
-                    ComparableZipInfo(zh.getinfo('file.txt')),
-                    ComparableZipInfo(zinfos[0]),
-                )
+            # check NameToInfo cache
+            self.assertEqual(
+                ComparableZipInfo(zh.getinfo('file.txt')),
+                ComparableZipInfo(zinfos[0]),
+            )
 
-            # make sure the zip file is still valid
-            with zipfile.ZipFile(TESTFN) as zh:
-                self.assertIsNone(zh.testzip())
+        # make sure the zip file is still valid
+        with zipfile.ZipFile(TESTFN) as zh:
+            self.assertIsNone(zh.testzip())
 
+        # suppress duplicated name warning
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore")
             zinfos = self._prepare_zip_from_test_files(TESTFN, test_files)
-            with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
-                zh.remove('file.txt')
-                zh.remove('file.txt')
 
-                # check infolist
-                self.assertEqual(
-                    [ComparableZipInfo(zi) for zi in zh.infolist()],
-                    [ComparableZipInfo(zi) for zi in [zinfos[2]]],
-                )
+        with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
+            zh.remove('file.txt')
+            zh.remove('file.txt')
+
+            # check infolist
+            self.assertEqual(
+                [ComparableZipInfo(zi) for zi in zh.infolist()],
+                [ComparableZipInfo(zi) for zi in [zinfos[2]]],
+            )
 
-                # check NameToInfo cache
-                with self.assertRaises(KeyError):
-                    zh.getinfo('file.txt')
+            # check NameToInfo cache
+            with self.assertRaises(KeyError):
+                zh.getinfo('file.txt')
 
-            # make sure the zip file is still valid
-            with zipfile.ZipFile(TESTFN) as zh:
-                self.assertIsNone(zh.testzip())
+        # make sure the zip file is still valid
+        with zipfile.ZipFile(TESTFN) as zh:
+            self.assertIsNone(zh.testzip())
 
     def test_remove_by_zinfo_duplicated(self):
         test_files = [
@@ -486,66 +490,74 @@ class AbstractRemoveTests(RepackHelperMixin):
         # suppress duplicated name warning
         with warnings.catch_warnings():
             warnings.simplefilter("ignore")
-
             zinfos = self._prepare_zip_from_test_files(TESTFN, test_files)
-            with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
-                zh.remove(zh.infolist()[0])
 
-                # check infolist
-                self.assertEqual(
-                    [ComparableZipInfo(zi) for zi in zh.infolist()],
-                    [ComparableZipInfo(zi) for zi in [zinfos[1], zinfos[2]]],
-                )
+        with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
+            zh.remove(zh.infolist()[0])
 
-                # check NameToInfo cache
-                self.assertEqual(
-                    ComparableZipInfo(zh.getinfo('file.txt')),
-                    ComparableZipInfo(zinfos[1]),
-                )
+            # check infolist
+            self.assertEqual(
+                [ComparableZipInfo(zi) for zi in zh.infolist()],
+                [ComparableZipInfo(zi) for zi in [zinfos[1], zinfos[2]]],
+            )
 
-            # make sure the zip file is still valid
-            with zipfile.ZipFile(TESTFN) as zh:
-                self.assertIsNone(zh.testzip())
+            # check NameToInfo cache
+            self.assertEqual(
+                ComparableZipInfo(zh.getinfo('file.txt')),
+                ComparableZipInfo(zinfos[1]),
+            )
 
+        # make sure the zip file is still valid
+        with zipfile.ZipFile(TESTFN) as zh:
+            self.assertIsNone(zh.testzip())
+
+        # suppress duplicated name warning
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore")
             zinfos = self._prepare_zip_from_test_files(TESTFN, test_files)
-            with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
-                zh.remove(zh.infolist()[1])
 
-                # check infolist
-                self.assertEqual(
-                    [ComparableZipInfo(zi) for zi in zh.infolist()],
-                    [ComparableZipInfo(zi) for zi in [zinfos[0], zinfos[2]]],
-                )
+        with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
+            zh.remove(zh.infolist()[1])
+
+            # check infolist
+            self.assertEqual(
+                [ComparableZipInfo(zi) for zi in zh.infolist()],
+                [ComparableZipInfo(zi) for zi in [zinfos[0], zinfos[2]]],
+            )
 
-                # check NameToInfo cache
-                self.assertEqual(
-                    ComparableZipInfo(zh.getinfo('file.txt')),
-                    ComparableZipInfo(zinfos[0]),
-                )
+            # check NameToInfo cache
+            self.assertEqual(
+                ComparableZipInfo(zh.getinfo('file.txt')),
+                ComparableZipInfo(zinfos[0]),
+            )
 
-            # make sure the zip file is still valid
-            with zipfile.ZipFile(TESTFN) as zh:
-                self.assertIsNone(zh.testzip())
+        # make sure the zip file is still valid
+        with zipfile.ZipFile(TESTFN) as zh:
+            self.assertIsNone(zh.testzip())
 
+        # suppress duplicated name warning
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore")
             zinfos = self._prepare_zip_from_test_files(TESTFN, test_files)
-            with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
-                infolist = zh.infolist().copy()
-                zh.remove(infolist[0])
-                zh.remove(infolist[1])
 
-                # check infolist
-                self.assertEqual(
-                    [ComparableZipInfo(zi) for zi in zh.infolist()],
-                    [ComparableZipInfo(zi) for zi in [zinfos[2]]],
-                )
+        with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
+            infolist = zh.infolist().copy()
+            zh.remove(infolist[0])
+            zh.remove(infolist[1])
 
-                # check NameToInfo cache
-                with self.assertRaises(KeyError):
-                    zh.getinfo('file.txt')
+            # check infolist
+            self.assertEqual(
+                [ComparableZipInfo(zi) for zi in zh.infolist()],
+                [ComparableZipInfo(zi) for zi in [zinfos[2]]],
+            )
+
+            # check NameToInfo cache
+            with self.assertRaises(KeyError):
+                zh.getinfo('file.txt')
 
-            # make sure the zip file is still valid
-            with zipfile.ZipFile(TESTFN) as zh:
-                self.assertIsNone(zh.testzip())
+        # make sure the zip file is still valid
+        with zipfile.ZipFile(TESTFN) as zh:
+            self.assertIsNone(zh.testzip())
 
     @requires_zip64fix()
     def test_remove_zip64(self):
@@ -823,7 +835,7 @@ class AbstractRepackTests(RepackHelperMixin):
                 with zipfile.ZipFile(TESTFN) as zh:
                     self.assertIsNone(zh.testzip())
 
-    @mock.patch.object(time, 'time', new=lambda: 315504000)  # fix time for ZipFile.writestr()
+    @mock.patch.object(time, 'time', new=lambda: 315590400)  # fix time for ZipFile.writestr()
     def test_repack_bytes_before_removed_files(self):
         """Should preserve if there are bytes before stale local file entries."""
         for ii in ([1], [1, 2], [2]):
@@ -867,7 +879,7 @@ class AbstractRepackTests(RepackHelperMixin):
                 with zipfile.ZipFile(TESTFN) as zh:
                     self.assertIsNone(zh.testzip())
 
-    @mock.patch.object(time, 'time', new=lambda: 315504000)  # fix time for ZipFile.writestr()
+    @mock.patch.object(time, 'time', new=lambda: 315590400)  # fix time for ZipFile.writestr()
     def test_repack_bytes_after_removed_files(self):
         """Should keep extra bytes if there are bytes after stale local file entries."""
         for ii in ([1], [1, 2], [2]):
@@ -910,7 +922,7 @@ class AbstractRepackTests(RepackHelperMixin):
                 with zipfile.ZipFile(TESTFN) as zh:
                     self.assertIsNone(zh.testzip())
 
-    @mock.patch.object(time, 'time', new=lambda: 315504000)  # fix time for ZipFile.writestr()
+    @mock.patch.object(time, 'time', new=lambda: 315590400)  # fix time for ZipFile.writestr()
     def test_repack_bytes_between_removed_files(self):
         """Should strip only local file entries before random bytes."""
         # calculate the expected results
@@ -954,8 +966,8 @@ class AbstractRepackTests(RepackHelperMixin):
         for ii in ([], [0], [0, 1], [1], [2]):
             with self.subTest(remove=ii):
                 # calculate the expected results
-                test_files = [data for j, data in enumerate(self.test_files) if j not in ii]
                 fz = io.BytesIO()
+                test_files = [data for j, data in enumerate(self.test_files) if j not in ii]
                 self._prepare_zip_from_test_files(fz, test_files)
                 fz.seek(0)
                 with open(TESTFN, 'wb') as fh:
@@ -973,9 +985,67 @@ class AbstractRepackTests(RepackHelperMixin):
                     fh.write(b'dummy ')
                     fh.write(fz.read())
                 with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
+                    if hasattr(zh, 'data_offset'):
+                        self.assertEqual(zh.data_offset, 6)
                     for i in ii:
                         zh.remove(self.test_files[i][0])
                     zh.repack()
+                    if hasattr(zh, 'data_offset'):
+                        self.assertEqual(zh.data_offset, 6)
+
+                    # check infolist
+                    self.assertEqual(
+                        [ComparableZipInfo(zi) for zi in zh.infolist()],
+                        [ComparableZipInfo(zi) for zi in expected_zinfos],
+                    )
+
+                # check file size
+                self.assertEqual(os.path.getsize(TESTFN), expected_size)
+
+                # make sure the zip file is still valid
+                with zipfile.ZipFile(TESTFN) as zh:
+                    self.assertIsNone(zh.testzip())
+
+    def test_repack_prepended_file_entry(self):
+        for ii in ([0], [0, 1], [0, 1, 2]):
+            with self.subTest(remove=ii):
+                # calculate the expected results
+                fz = io.BytesIO()
+                test_files = [data for j, data in enumerate(self.test_files) if j not in ii]
+                self._prepare_zip_from_test_files(fz, test_files)
+                fz.seek(0)
+                with open(TESTFN, 'wb') as fh:
+                    fh.write(b'dummy ')
+                    fh.write(fz.read())
+                with zipfile.ZipFile(TESTFN) as zh:
+                    expected_zinfos = list(zh.infolist())
+                expected_size = os.path.getsize(TESTFN)
+
+                # do the removal and check the result
+                fz = io.BytesIO()
+                with zipfile.ZipFile(fz, 'w') as zh:
+                    for j, (file, data) in enumerate(self.test_files):
+                        if j in ii:
+                            zh.writestr(file, data)
+                    fz.seek(0)
+                    prefix = fz.read()
+
+                fz = io.BytesIO()
+                test_files = [data for j, data in enumerate(self.test_files) if j not in ii]
+                self._prepare_zip_from_test_files(fz, test_files)
+                fz.seek(0)
+
+                with open(TESTFN, 'wb') as fh:
+                    fh.write(b'dummy ')
+                    fh.write(prefix)
+                    fh.write(fz.read())
+
+                with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
+                    if hasattr(zh, 'data_offset'):
+                        self.assertEqual(zh.data_offset, 6 + len(prefix))
+                    zh.repack()
+                    if hasattr(zh, 'data_offset'):
+                        self.assertEqual(zh.data_offset, 6)
 
                     # check infolist
                     self.assertEqual(
@@ -1068,7 +1138,7 @@ class AbstractRepackTests(RepackHelperMixin):
                 with zipfile.ZipFile(TESTFN) as zh:
                     self.assertIsNone(zh.testzip())
 
-    @mock.patch.object(time, 'time', new=lambda: 315504000)  # fix time for ZipFile.writestr()
+    @mock.patch.object(time, 'time', new=lambda: 315590400)  # fix time for ZipFile.writestr()
     def test_repack_removed_bytes_between_files(self):
         """Should not remove bytes between local file entries."""
         for ii in ([0], [1], [2]):
@@ -1165,8 +1235,12 @@ class AbstractRepackTests(RepackHelperMixin):
                     fh.write(b'dummy ')
                     fh.write(fz.read())
                 with zipfile.ZipFile(TESTFN, 'a', self.compression) as zh:
+                    if hasattr(zh, 'data_offset'):
+                        self.assertEqual(zh.data_offset, 6)
                     zinfos = [zh.remove(self.test_files[i][0]) for i in ii]
                     zh.repack(zinfos)
+                    if hasattr(zh, 'data_offset'):
+                        self.assertEqual(zh.data_offset, 6)
 
                     # check infolist
                     self.assertEqual(
@@ -1468,11 +1542,14 @@ class ZipRepackerTests(unittest.TestCase):
         fz = io.BytesIO()
         f = Unseekable(fz) if dd else fz
         cm = (mock.patch.object(struct, 'pack', side_effect=struct_pack_no_dd_sig)
-              if not dd_sig else contextlib.nullcontext())
+              if dd and not dd_sig else contextlib.nullcontext())
         with zipfile.ZipFile(f, 'w', compression=compression) as zh:
-            with cm:
-                with zh.open(arcname, 'w', force_zip64=force_zip64) as fh:
-                    fh.write(raw_bytes)
+            with cm, zh.open(arcname, 'w', force_zip64=force_zip64) as fh:
+                fh.write(raw_bytes)
+            if dd:
+                zi = zh.infolist()[0]
+                self.assertTrue(zi.flag_bits & zipfile._MASK_USE_DATA_DESCRIPTOR,
+                                f'data descriptor flag not set: {zi.filename}')
             fz.seek(0)
             return fz.read()
 
@@ -1578,10 +1655,10 @@ class ZipRepackerTests(unittest.TestCase):
         m_sddnsbd.assert_not_called()
         m_sddns.assert_not_called()
 
-        # return None if no sufficient header length
+        # return None if truncated local file header
         bytes_ = self._generate_local_file_entry(
             'file.txt', b'dummy', compression=method)
-        bytes_ = bytes_[:29]
+        bytes_ = bytes_[:zipfile.sizeFileHeader - 1]
         fz = io.BytesIO(bytes_)
         with mock.patch.object(repacker, '_scan_data_descriptor',
                                wraps=repacker._scan_data_descriptor) as m_sdd, \
@@ -2125,6 +2202,10 @@ class ZipRepackerTests(unittest.TestCase):
     def test_trace_compressed_block_end_bz2(self):
         self._test_trace_compressed_block_end(zipfile.ZIP_BZIP2, OSError)
 
+    @requires_lzma()
+    def test_trace_compressed_block_end_lzma(self):
+        self._test_trace_compressed_block_end(zipfile.ZIP_LZMA, EOFError)
+
     @requires_zstd()
     def test_trace_compressed_block_end_zstd(self):
         import compression.zstd
@@ -2193,6 +2274,87 @@ class ZipRepackerTests(unittest.TestCase):
             comp_len,
         )
 
+    def test_calc_local_file_entry_size(self):
+        repacker = zipfile._ZipRepacker()
+
+        # basic
+        fz = io.BytesIO()
+        with zipfile.ZipFile(fz, 'w') as zh:
+            with zh.open('file.txt', 'w') as fh:
+                fh.write(b'dummy')
+            zi = zh.infolist()[-1]
+
+        self.assertEqual(
+            repacker._calc_local_file_entry_size(fz, zi),
+            (30, 8, 0, 5, 0),
+        )
+
+        # data descriptor
+        fz = io.BytesIO()
+        with zipfile.ZipFile(Unseekable(fz), 'w') as zh:
+            with zh.open('file.txt', 'w') as fh:
+                fh.write(b'dummy')
+            zi = zh.infolist()[-1]
+
+        self.assertEqual(
+            repacker._calc_local_file_entry_size(fz, zi),
+            (30, 8, 0, 5, 16),
+        )
+
+        # data descriptor (unsigned)
+        fz = io.BytesIO()
+        with zipfile.ZipFile(Unseekable(fz), 'w') as zh:
+            with mock.patch.object(struct, 'pack', side_effect=struct_pack_no_dd_sig), \
+                 zh.open('file.txt', 'w') as fh:
+                fh.write(b'dummy')
+            zi = zh.infolist()[-1]
+
+        self.assertEqual(
+            repacker._calc_local_file_entry_size(fz, zi),
+            (30, 8, 0, 5, 12),
+        )
+
+    @requires_zip64fix()
+    def test_calc_local_file_entry_size_zip64(self):
+        repacker = zipfile._ZipRepacker()
+
+        # zip64
+        fz = io.BytesIO()
+        with zipfile.ZipFile(fz, 'w') as zh:
+            with zh.open('file.txt', 'w', force_zip64=True) as fh:
+                fh.write(b'dummy')
+            zi = zh.infolist()[-1]
+
+        self.assertEqual(
+            repacker._calc_local_file_entry_size(fz, zi),
+            (30, 8, 20, 5, 0),
+        )
+
+        # data descriptor + zip64
+        fz = io.BytesIO()
+        with zipfile.ZipFile(Unseekable(fz), 'w') as zh:
+            with zh.open('file.txt', 'w', force_zip64=True) as fh:
+                fh.write(b'dummy')
+            zi = zh.infolist()[-1]
+
+        self.assertEqual(
+            repacker._calc_local_file_entry_size(fz, zi),
+            (30, 8, 20, 5, 24),
+        )
+
+        # data descriptor (unsigned) + zip64
+        fz = io.BytesIO()
+        with zipfile.ZipFile(Unseekable(fz), 'w') as zh:
+            with mock.patch.object(struct, 'pack', side_effect=struct_pack_no_dd_sig), \
+                 zh.open('file.txt', 'w', force_zip64=True) as fh:
+                fh.write(b'dummy')
+            zi = zh.infolist()[-1]
+
+        self.assertEqual(
+            repacker._calc_local_file_entry_size(fz, zi),
+            (30, 8, 20, 5, 20),
+        )
+
     def test_copy_bytes(self):
         repacker = zipfile._ZipRepacker()
 

{zipremove-0.5.0 → zipremove-0.6.0}/tests/test_zipfile64.py

@@ -132,10 +132,6 @@ class TestRepack(unittest.TestCase):
             zh.writestr(file, data)
 
         with zipfile.ZipFile(f, 'a') as zh:
-            # make sure data descriptor bit is really set (by making zip file unseekable)
-            for zi in zh.infolist():
-                self.assertTrue(zi.flag_bits & 8, f'data descriptor flag not set: {zi.filename}')
-
             zh.remove(file1)
             zh.repack()
             self.assertIsNone(zh.testzip())
@@ -143,6 +139,10 @@ class TestRepack(unittest.TestCase):
     def test_strip_removed_large_file_with_dd_no_sig(self):
         """Should scan for the data descriptor (without signature) of a removed
         large file without causing a memory issue."""
+        # Reduce data scale for this test, as it's especially slow...
+        self.datacount = 30*1024**2 // len(self.data)
+        self.allowed_memory = 200*1024
+
         # Try the temp file.  If we do TESTFN2, then it hogs
         # gigabytes of disk space for the duration of the test.
         with TemporaryFile() as f:
@@ -154,9 +154,6 @@ class TestRepack(unittest.TestCase):
             self.assertLess(peak, self.allowed_memory)
 
     def _test_strip_removed_large_file_with_dd_no_sig(self, f):
-        # Reduce data to 400 MiB for this test, as it's especially slow...
-        self.datacount = 400*1024**2 // len(self.data)
-
         file = 'file.txt'
         file1 = 'largefile.txt'
         data = b'Sed ut perspiciatis unde omnis iste natus error sit voluptatem'
@@ -167,10 +164,6 @@ class TestRepack(unittest.TestCase):
                 zh.writestr(file, data)
 
         with zipfile.ZipFile(f, 'a') as zh:
-            # make sure data descriptor bit is really set (by making zip file unseekable)
-            for zi in zh.infolist():
-                self.assertTrue(zi.flag_bits & 8, f'data descriptor flag not set: {zi.filename}')
-
             zh.remove(file1)
             zh.repack()
             self.assertIsNone(zh.testzip())
@@ -201,10 +194,6 @@ class TestRepack(unittest.TestCase):
                 zh.writestr(file, data)
 
         with zipfile.ZipFile(f, 'a') as zh:
-            # make sure data descriptor bit is really set (by making zip file unseekable)
-            for zi in zh.infolist():
-                self.assertTrue(zi.flag_bits & 8, f'data descriptor flag not set: {zi.filename}')
-
             zh.remove(file1)
             zh.repack()
             self.assertIsNone(zh.testzip())