FastWARC 1.0.1__tar.gz → 1.0.2__tar.gz

fastwarc-1.0.2/FastWARC.egg-info/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.4
+Name: FastWARC
+Version: 1.0.2
+Summary: A high-performance WARC parsing library written in Rust with bindings for Python.
+Author: Janek Bevendorff
+License-Expression: Apache-2.0
+Project-URL: repository, https://github.com/chatnoir-eu/chatnoir-resiliparse
+Project-URL: homepage, https://resiliparse.chatnoir.eu
+Project-URL: documentation, https://resiliparse.chatnoir.eu
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: brotli
+Requires-Dist: click
+Requires-Dist: tqdm
+Provides-Extra: all
+Requires-Dist: fastwarc[fsspec]; extra == "all"
+Provides-Extra: fsspec
+Requires-Dist: fsspec; extra == "fsspec"
+Requires-Dist: s3fs; extra == "fsspec"
+Provides-Extra: test
+Requires-Dist: fastwarc[all]; extra == "test"
+Requires-Dist: pytest>=7.2.2; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+
+# FastWARC (Python)
+
+FastWARC is a high-performance WARC parsing library written in Rust with bindings for Python.
+FastWARC supports compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip,
+Zstd, and LZ4.
+
+This package contains the Python bindings for FastWARC.
+
+FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
+and robust web data processing.
+
+## Installing FastWARC
+
+Pre-built FastWARC binaries for Python can be installed from [PyPi](https://pypi.org/project/FastWARC/):
+
+```bash
+pip install fastwarc
+```
+
+## Building Python Bindings From Source
+
+To be able to build FastWARC, you need the Rust toolchain installed (see above). Then to build directly from the GitHub
+repository, follow these steps:
+
+```bash
+# Clone repository
+
+git clone https://github.com/chatnoir-eu/chatnoir-resiliparse.git
+cd chatnoir-resiliparse
+
+# Install Rust toolchain (if not already installed)
+
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+
+# Create a fresh venv first (recommended)
+
+python3 -m venv venv && source venv/bin/activate
+
+# Option 1: Build and install in editable mode (best for development)
+
+python3 -m pip install -e ./fastwarc-py
+
+# Option 2 (alternative): Build and install wheels in separate steps (best for redistribution)
+
+python3 -m pip wheel -w build ./fastwarc-py
+ls ./build/*.whl | xargs python3 -m pip install
+```
+
+## Usage Instructions
+
+For detailed usage instructions, please consult
+the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
+
+## Cite Us
+
+If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
+
+```bibtex
+@InProceedings{bevendorff:2021,
+  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
+  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
+  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
+  month =                 oct,
+  publisher =             {International Open Search Symposium},
+  site =                  {CERN, Geneva, Switzerland},
+  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
+  year =                  2021
+}
+```
+
+You can also cite our [ECIR 2018 demo paper](https://webis.de/downloads/publications/papers/bevendorff_2018.pdf):
+
+```bibtex
+@InProceedings{bevendorff:2018,
+  address =             {Berlin Heidelberg New York},
+  author =              {Janek Bevendorff and Benno Stein and Matthias Hagen and Martin Potthast},
+  booktitle =           {Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018)},
+  editor =              {Leif Azzopardi and Allan Hanbury and Gabriella Pasi and Benjamin Piwowarski},
+  month =               mar,
+  publisher =           {Springer},
+  series =              {Lecture Notes in Computer Science},
+  site =                {Grenoble, France},
+  title =               {{Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl}},
+  year =                2018
+}
+```

{fastwarc-1.0.1 → fastwarc-1.0.2}/FastWARC.egg-info/SOURCES.txt

@@ -16,6 +16,7 @@ fastwarc/stream_io.pyi
 fastwarc/warc.pyi
 fastwarc/legacy/__init__.pxd
 fastwarc/legacy/__init__.py
+fastwarc/legacy/__init__.pyi
 fastwarc/legacy/_shims.py
 fastwarc/legacy/stream_io.pxd
 fastwarc/legacy/stream_io.pyi

{fastwarc-1.0.1 → fastwarc-1.0.2}/FastWARC.egg-info/requires.txt

@@ -7,9 +7,9 @@ fastwarc[fsspec]
 
 [fsspec]
 fsspec
+s3fs
 
 [test]
 fastwarc[all]
 pytest>=7.2.2
 pytest-cov>=4.0.0
-lz4

fastwarc-1.0.2/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.4
+Name: FastWARC
+Version: 1.0.2
+Summary: A high-performance WARC parsing library written in Rust with bindings for Python.
+Author: Janek Bevendorff
+License-Expression: Apache-2.0
+Project-URL: repository, https://github.com/chatnoir-eu/chatnoir-resiliparse
+Project-URL: homepage, https://resiliparse.chatnoir.eu
+Project-URL: documentation, https://resiliparse.chatnoir.eu
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: brotli
+Requires-Dist: click
+Requires-Dist: tqdm
+Provides-Extra: all
+Requires-Dist: fastwarc[fsspec]; extra == "all"
+Provides-Extra: fsspec
+Requires-Dist: fsspec; extra == "fsspec"
+Requires-Dist: s3fs; extra == "fsspec"
+Provides-Extra: test
+Requires-Dist: fastwarc[all]; extra == "test"
+Requires-Dist: pytest>=7.2.2; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+
+# FastWARC (Python)
+
+FastWARC is a high-performance WARC parsing library written in Rust with bindings for Python.
+FastWARC supports compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip,
+Zstd, and LZ4.
+
+This package contains the Python bindings for FastWARC.
+
+FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
+and robust web data processing.
+
+## Installing FastWARC
+
+Pre-built FastWARC binaries for Python can be installed from [PyPi](https://pypi.org/project/FastWARC/):
+
+```bash
+pip install fastwarc
+```
+
+## Building Python Bindings From Source
+
+To be able to build FastWARC, you need the Rust toolchain installed (see above). Then to build directly from the GitHub
+repository, follow these steps:
+
+```bash
+# Clone repository
+
+git clone https://github.com/chatnoir-eu/chatnoir-resiliparse.git
+cd chatnoir-resiliparse
+
+# Install Rust toolchain (if not already installed)
+
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+
+# Create a fresh venv first (recommended)
+
+python3 -m venv venv && source venv/bin/activate
+
+# Option 1: Build and install in editable mode (best for development)
+
+python3 -m pip install -e ./fastwarc-py
+
+# Option 2 (alternative): Build and install wheels in separate steps (best for redistribution)
+
+python3 -m pip wheel -w build ./fastwarc-py
+ls ./build/*.whl | xargs python3 -m pip install
+```
+
+## Usage Instructions
+
+For detailed usage instructions, please consult
+the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
+
+## Cite Us
+
+If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
+
+```bibtex
+@InProceedings{bevendorff:2021,
+  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
+  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
+  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
+  month =                 oct,
+  publisher =             {International Open Search Symposium},
+  site =                  {CERN, Geneva, Switzerland},
+  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
+  year =                  2021
+}
+```
+
+You can also cite our [ECIR 2018 demo paper](https://webis.de/downloads/publications/papers/bevendorff_2018.pdf):
+
+```bibtex
+@InProceedings{bevendorff:2018,
+  address =             {Berlin Heidelberg New York},
+  author =              {Janek Bevendorff and Benno Stein and Matthias Hagen and Martin Potthast},
+  booktitle =           {Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018)},
+  editor =              {Leif Azzopardi and Allan Hanbury and Gabriella Pasi and Benjamin Piwowarski},
+  month =               mar,
+  publisher =           {Springer},
+  series =              {Lecture Notes in Computer Science},
+  site =                {Grenoble, France},
+  title =               {{Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl}},
+  year =                2018
+}
+```

fastwarc-1.0.2/README.md

@@ -0,0 +1,86 @@
+# FastWARC (Python)
+
+FastWARC is a high-performance WARC parsing library written in Rust with bindings for Python.
+FastWARC supports compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip,
+Zstd, and LZ4.
+
+This package contains the Python bindings for FastWARC.
+
+FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
+and robust web data processing.
+
+## Installing FastWARC
+
+Pre-built FastWARC binaries for Python can be installed from [PyPi](https://pypi.org/project/FastWARC/):
+
+```bash
+pip install fastwarc
+```
+
+## Building Python Bindings From Source
+
+To be able to build FastWARC, you need the Rust toolchain installed (see above). Then to build directly from the GitHub
+repository, follow these steps:
+
+```bash
+# Clone repository
+
+git clone https://github.com/chatnoir-eu/chatnoir-resiliparse.git
+cd chatnoir-resiliparse
+
+# Install Rust toolchain (if not already installed)
+
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+
+# Create a fresh venv first (recommended)
+
+python3 -m venv venv && source venv/bin/activate
+
+# Option 1: Build and install in editable mode (best for development)
+
+python3 -m pip install -e ./fastwarc-py
+
+# Option 2 (alternative): Build and install wheels in separate steps (best for redistribution)
+
+python3 -m pip wheel -w build ./fastwarc-py
+ls ./build/*.whl | xargs python3 -m pip install
+```
+
+## Usage Instructions
+
+For detailed usage instructions, please consult
+the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
+
+## Cite Us
+
+If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
+
+```bibtex
+@InProceedings{bevendorff:2021,
+  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
+  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
+  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
+  month =                 oct,
+  publisher =             {International Open Search Symposium},
+  site =                  {CERN, Geneva, Switzerland},
+  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
+  year =                  2021
+}
+```
+
+You can also cite our [ECIR 2018 demo paper](https://webis.de/downloads/publications/papers/bevendorff_2018.pdf):
+
+```bibtex
+@InProceedings{bevendorff:2018,
+  address =             {Berlin Heidelberg New York},
+  author =              {Janek Bevendorff and Benno Stein and Matthias Hagen and Martin Potthast},
+  booktitle =           {Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018)},
+  editor =              {Leif Azzopardi and Allan Hanbury and Gabriella Pasi and Benjamin Piwowarski},
+  month =               mar,
+  publisher =           {Springer},
+  series =              {Lecture Notes in Computer Science},
+  site =                {Grenoble, France},
+  title =               {{Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl}},
+  year =                2018
+}
+```

{fastwarc-1.0.1 → fastwarc-1.0.2}/fastwarc/cli.py

@@ -14,6 +14,7 @@
 
 import getpass
 import importlib
+from enum import IntFlag
 from itertools import chain
 import json
 import os
@@ -24,14 +25,13 @@ import urllib.request
 import click
 from tqdm import tqdm
 
-from fastwarc.legacy.stream_io import FileStream, StreamError, FastWARCError, PythonIOStreamAdapter
-from fastwarc.legacy.warc import ArchiveIterator, WarcRecordType
-from fastwarc.legacy.tools import CompressionAlg, detect_compression_algorithm, wrap_warc_stream, \
-    recompress_warc_interactive, verify_digests
+from fastwarc.stream_io import *
+from fastwarc.warc import ArchiveIterator, WarcRecordType
+from fastwarc.legacy.tools import CompressionAlg, detect_compression_algorithm, wrap_warc_stream, verify_digests
 
 
 def exception_handler(exctype, value, _):
-    if exctype == FastWARCError:
+    if exctype == OSError:
         click.echo(str(value), err=True)
         sys.exit(1)
 
@@ -56,46 +56,132 @@ def _human_readable_bytes(byte_num):
         byte_num /= 1024
 
 
+class CompressionAlg(IntFlag):
+    UNCOMPRESSED = 0
+    GZIP = 1
+    ZSTD = 2
+    LZ4 = 3
+    AUTO = 65535
+
+
+def detect_compression_algorithm(infile):
+    """
+    Try to detect the used compression algorithm from the given filename.
+
+    :param infile: filename or file object
+    :return: compression algorithm
+    """
+
+    if os.fspath(infile):
+        infile = str(infile)
+    elif hasattr(infile, 'name'):
+        infile = infile.name
+    else:
+        raise ValueError('Invalid infile type.')
+
+    if type(infile) is str:
+        if infile.endswith('.gz'):
+            return CompressionAlg.GZIP
+        if infile.endswith('.zst'):
+            return CompressionAlg.ZSTD
+        if infile.endswith('.lz4'):
+            return CompressionAlg.LZ4
+        if infile.endswith('.warc'):
+            return CompressionAlg.UNCOMPRESSED
+
+    if not os.path.isfile(infile):
+        raise click.UsageError('Could not determine compression algorithm.')
+
+    with open(infile, 'rb') as infile:
+        magic_bytes = infile.read(5)
+
+    if magic_bytes[:3] == b'\x1F\x8B\x08':
+        return CompressionAlg.GZIP
+    elif magic_bytes[:4] == b'\x28\xB5\x2F\xFD' or magic_bytes[:4] == b'\x5D\x2A\x4D\x18':
+        return CompressionAlg.ZSTD
+    elif magic_bytes[:4] == b'\x04\x22\x4D\x18':
+        return CompressionAlg.LZ4
+    elif magic_bytes == b'WARC/':
+        return CompressionAlg.UNCOMPRESSED
+
+    raise click.FileError('No compression algorithm detected.')
+
+
+def get_compressor(filename, alg, level):
+    if alg == CompressionAlg.AUTO:
+        alg = detect_compression_algorithm(filename)
+    if alg == CompressionAlg.UNCOMPRESSED:
+        return open(filename, 'wb')
+    elif alg == CompressionAlg.GZIP:
+        return GzipWriter(filename, compression_level=level or 9)
+    elif alg == CompressionAlg.ZSTD:
+        return ZstdWriter(filename, compression_level=level or 3)
+    elif alg == CompressionAlg.LZ4:
+        return Lz4Writer(filename)  # compression_level currently unsupported
+    else:
+        raise ValueError(f'Unknown compression algorithm: {alg}')
+
+
+def get_decompressor(filename, decompress_alg):
+    if os.fspath(filename) and not decompress_alg:
+        # Use plain filename to avoid Python IO stack
+        return str(filename)
+
+    if decompress_alg:
+        alg = decompress_alg
+    else:
+        alg = detect_compression_algorithm(filename)
+    if alg == CompressionAlg.UNCOMPRESSED:
+        return open(filename, 'rb')
+    if alg == CompressionAlg.GZIP:
+        return GzipReader(filename)
+    if alg == CompressionAlg.ZSTD:
+        return ZstdReader(filename)
+    if alg == CompressionAlg.LZ4:
+        return Lz4Reader(filename)  # compression_level currently unsupported
+
+    raise ValueError(f'Unknown compression algorithm: {alg}')
+
+
 @main.command()
 @click.argument('infile', type=click.Path(dir_okay=False, exists=True))
 @click.argument('outfile', type=click.Path(dir_okay=False, exists=False))
-@click.option('-c', '--compress-alg', type=click.Choice(['gzip', 'lz4', 'uncompressed', 'auto']),
+@click.option('-c', '--compress-alg', type=click.Choice(['uncompressed', 'gzip', 'zstd', 'lz4', 'auto']),
               default='auto', show_default=True, help='Compression algorithm to use for output file')
-@click.option('-d', '--decompress-alg', type=click.Choice(['gzip', 'lz4', 'uncompressed', 'auto']),
+@click.option('-d', '--decompress-alg', type=click.Choice(['uncompressed', 'gzip', 'zstd', 'lz4', 'auto']),
               default='auto', show_default=True,
               help='Decompression algorithm for decoding input file (auto tries to detect based on file extension)')
-@click.option('-l', '--compress-level', type=int, default=None, help='Compression level (defaults to max)')
+@click.option('-l', '--compress-level', type=int, default=None, help='Compression level')
 @click.option('-q', '--quiet', is_flag=True, help='Do not print progress information')
 def recompress(infile, outfile, compress_alg, decompress_alg, compress_level, quiet):
     """
     Recompress a WARC file.
 
-    This command allows you to recompress a WARC file if it is uncompressed or not compressed
-    properly at the record-level if or you want to recompress a GZip WARC as LZ4 or vice versa.
+    This command allows you to recompress a WARC file if it is uncompressed, not compressed
+    properly at the record-level, or if you want to recompress a Gzip WARC as Zstd or LZ4 or vice versa.
     """
 
-    compress_alg = getattr(CompressionAlg, compress_alg)
-    decompress_alg = getattr(CompressionAlg, decompress_alg)
+    compress_alg = getattr(CompressionAlg, compress_alg.upper())
+    decompress_alg = getattr(CompressionAlg, decompress_alg.upper())
 
-    comp_args = {}
-    if compress_level is None:
-        if compress_alg == CompressionAlg.gzip:
-            comp_args['compression_level'] = 9
-        elif compress_alg == CompressionAlg.lz4:
-            comp_args['compression_level'] = 12
+    infile = get_decompressor(infile, decompress_alg if decompress_alg != CompressionAlg.AUTO else None)
+    outfile_name = outfile
+    outfile = get_compressor(outfile, compress_alg, compress_level)
 
-    bytes_written = 0
     num = 0
-    start = time.time()
+    start = time.monotonic()
     try:
-        for _, b in tqdm(recompress_warc_interactive(infile, outfile, decompress_alg, compress_alg, **comp_args),
-                         desc='Recompressing WARC file', unit=' record(s)', leave=False,
-                         disable=quiet, mininterval=0.2):
-            num += 1
-            bytes_written += b
+        with outfile as out:
+            for r in tqdm(ArchiveIterator(infile),
+                          desc='Recompressing WARC file', unit=' record(s)', leave=False,
+                          disable=quiet, mininterval=0.3):
+                num += 1
+                r.write(out)
+                if compress_alg != CompressionAlg.UNCOMPRESSED:
+                    out.finish()
 
-        if not quiet:
-            click.echo('Recompression completed.')
+            if not quiet:
+                click.echo('Recompression completed.')
     except KeyboardInterrupt:
         if not quiet:
             click.echo('Recompression aborted.')
@@ -103,8 +189,8 @@ def recompress(infile, outfile, compress_alg, decompress_alg, compress_level, qu
     finally:
         if not quiet and num > 0:
             click.echo(f'  - Records recompressed: {num}')
-            click.echo(f'  - Bytes written: {_human_readable_bytes(bytes_written)}')
-            click.echo(f'  - Completed in: {time.time() - start:.02f} seconds')
+            click.echo(f'  - Bytes written: {_human_readable_bytes(os.stat(outfile_name).st_size)}')
+            click.echo(f'  - Completed in: {time.monotonic() - start:.02f} seconds')
 
 
 @main.command()

fastwarc-1.0.2/fastwarc/legacy/__init__.pyi

@@ -0,0 +1,19 @@
+from .stream_io import (
+    FileStream,
+    GZipStream,
+    LZ4Stream,
+    FastWARCError,
+    StreamError,
+)
+from .warc import ArchiveIterator, WarcRecord, WarcRecordType
+
+__all__ = [
+    "FileStream",
+    "GZipStream",
+    "LZ4Stream",
+    "FastWARCError",
+    "StreamError",
+    "ArchiveIterator",
+    "WarcRecord",
+    "WarcRecordType",
+]

{fastwarc-1.0.1 → fastwarc-1.0.2}/fastwarc/stream_io.pyi

@@ -128,54 +128,55 @@ _WriterInput = WarcWriter | BinaryIO | _GenericWriter | PathLike[str] | str
 
 @disjoint_base
 class GzipReader(WarcReader):
-    def __new__(cls, inner: _ReaderInput, buffer_size=4096, zlib=False, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _ReaderInput, buffer_size: int = 4096, zlib: bool = False, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class GzipWriter(WarcWriter):
-    def __new__(cls, inner: _WriterInput, compression_level=9, buffer_size=8192, zlib=False,
+    def __new__(cls, inner: _WriterInput, compression_level: int = 9, buffer_size=8192, zlib=False,
                 fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class ZstdReader(WarcReader):
-    def __new__(cls, inner: _ReaderInput, buffer_size=4096, fsspec_args=None, dictionary=None) -> Self: ...
+    def __new__(cls, inner: _ReaderInput, buffer_size: int = 4096, fsspec_args=None,
+                dictionary: bytes | None = None) -> Self: ...
 
 
 @disjoint_base
 class ZstdWriter(WarcWriter):
-    def __new__(cls, inner: _WriterInput, buffer_size=8192, fsspec_args=None, dictionary=None,
-                compress_dictionary_frame=False) -> Self: ...
+    def __new__(cls, inner: _WriterInput, buffer_size: int = 8192, compression_level: int = 3, fsspec_args=None,
+                dictionary: bytes | None = None, compress_dictionary_frame=False) -> Self: ...
 
 
 @disjoint_base
 class Lz4Reader(WarcReader):
-    def __new__(cls, inner: _ReaderInput, buffer_size=4096, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _ReaderInput, buffer_size: int = 4096, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class Lz4Writer(WarcWriter):
-    def __new__(cls, inner: _WriterInput, buffer_size=8192, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _WriterInput, buffer_size: int = 8192, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class BrotliReader(WarcReader):
-    def __new__(cls, inner: _ReaderInput, buffer_size=4096, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _ReaderInput, buffer_size: int = 4096, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class BrotliWriter(WarcWriter):
-    def __new__(cls, inner: _WriterInput, buffer_size=8192, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _WriterInput, buffer_size: int = 8192, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class ChunkedReader(WarcReader):
-    def __new__(cls, inner: _ReaderInput, buffer_size=1024, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _ReaderInput, buffer_size: int = 1024, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class ChunkedWriter(WarcWriter):
-    def __new__(cls, inner: _WriterInput, min_chunk_size=512, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _WriterInput, min_chunk_size: int = 512, fsspec_args=None) -> Self: ...
 
 
 def zstd_train_dictionary_from_continuous(sample_data: bytes, sample_sizes: list[int], max_size: int) -> bytes: ...

{fastwarc-1.0.1 → fastwarc-1.0.2}/fastwarc/warc.pyi

@@ -213,6 +213,12 @@ class WarcRecordPayloadReader(WarcReader):
 class WarcRecord:
     def __new__(cls) -> Self: ...
 
+    @classmethod
+    def from_bytes(cls, payload: bytes) -> Self: ...
+
+    @classmethod
+    def from_reader(cls, reader: _ReaderInput, quirks_mode: bool = False) -> Self: ...
+
     @property
     def record_id(self) -> str | None: ...
 
@@ -298,6 +304,10 @@ class WarcRecord:
             chunk_size: int = 16384
     ) -> int: ...
 
+    def __eq__(self, other, /) -> bool: ...
+
+    def __ne__(self, other, /) -> bool: ...
+
     def __str__(self) -> str: ...
 
     def __repr__(self) -> str: ...

{fastwarc-1.0.1 → fastwarc-1.0.2}/pyproject.toml

@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "FastWARC"
-version = "1.0.1"
-description = "A high-performance WARC parsing library for Python written in C++/Cython."
+version = "1.0.2"
+description = "A high-performance WARC parsing library written in Rust with bindings for Python."
 authors = [
     { name = "Janek Bevendorff" }
 ]
 license = "Apache-2.0"
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.10"
 dependencies = [
     "brotli",
     "click",
@@ -28,20 +28,20 @@ all = [
     "fastwarc[fsspec]"
 ]
 fsspec = [
-    "fsspec"
+    "fsspec",
+    "s3fs",
 ]
 test = [
     "fastwarc[all]",
     "pytest>=7.2.2",
     "pytest-cov>=4.0.0",
-    "lz4",
 ]
 
 [project.scripts]
 fastwarc = "fastwarc.cli:main"
 
 [tool.setuptools.packages.find]
-include = ["fastwarc*"]
+include = ["fastwarc", "fastwarc.legacy"]
 
 [tool.setuptools.package-data]
 "*" = ["*.pyi"]

fastwarc-1.0.1/FastWARC.egg-info/PKG-INFO

@@ -1,97 +0,0 @@
-Metadata-Version: 2.4
-Name: FastWARC
-Version: 1.0.1
-Summary: A high-performance WARC parsing library for Python written in C++/Cython.
-Author: Janek Bevendorff
-License-Expression: Apache-2.0
-Project-URL: repository, https://github.com/chatnoir-eu/chatnoir-resiliparse
-Project-URL: homepage, https://resiliparse.chatnoir.eu
-Project-URL: documentation, https://resiliparse.chatnoir.eu
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-Requires-Dist: brotli
-Requires-Dist: click
-Requires-Dist: tqdm
-Provides-Extra: all
-Requires-Dist: fastwarc[fsspec]; extra == "all"
-Provides-Extra: fsspec
-Requires-Dist: fsspec; extra == "fsspec"
-Provides-Extra: test
-Requires-Dist: fastwarc[all]; extra == "test"
-Requires-Dist: pytest>=7.2.2; extra == "test"
-Requires-Dist: pytest-cov>=4.0.0; extra == "test"
-Requires-Dist: lz4; extra == "test"
-
-# FastWARC (Python)
-
-FastWARC is a high-performance WARC parsing library for Python written in Rust with Python bindings. FastWARC supports
-compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip and LZ4.
-
-This crate contains the Python bindings for FastWARC.
-
-FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
-and robust web data processing.
-
-## Installing FastWARC
-
-Pre-built FastWARC binaries for most Linux platforms can be installed from PyPi:
-
-```bash
-pip install fastwarc
-```
-
-**However:** the Linux binaries are provided *solely for your convenience*. Since they are built on the very old
-`manylinux` base system for better compatibility, their performance isn't optimal (though still better than WARCIO). For
-best performance, see the next section on how to build FastWARC yourself.
-
-## Building FastWARC From Source
-
-You can compile FastWARC either from the PyPi source package or directly from this repository, though in any case, you
-need to install all required build-time dependencies first. On Ubuntu, this is done as follows:
-
-```bash
-sudo apt install build-essential python3-dev zlib1g-dev liblz4-dev
-```
-
-To build and install FastWARC from PyPi, run
-
-```bash
-pip install --no-binary fastwarc fastwarc
-```
-
-That's it. If you prefer to build and install directly from this repository instead, run:
-
-```bash
-pip install -e fastwarc
-```
-
-To build the wheels without installing them, run:
-
-```bash
-pip wheel -e fastwarc
-
-# Or:
-pip install build && python -m build --wheel fastwarc
-```
-
-## Usage Instructions
-
-For detailed usage instructions, please consult
-the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
-
-## Cite Us
-
-If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
-
-```bibtex
-@InProceedings{bevendorff:2021,
-  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
-  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
-  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
-  month =                 oct,
-  publisher =             {International Open Search Symposium},
-  site =                  {CERN, Geneva, Switzerland},
-  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
-  year =                  2021
-}
-```

fastwarc-1.0.1/PKG-INFO

@@ -1,97 +0,0 @@
-Metadata-Version: 2.4
-Name: FastWARC
-Version: 1.0.1
-Summary: A high-performance WARC parsing library for Python written in C++/Cython.
-Author: Janek Bevendorff
-License-Expression: Apache-2.0
-Project-URL: repository, https://github.com/chatnoir-eu/chatnoir-resiliparse
-Project-URL: homepage, https://resiliparse.chatnoir.eu
-Project-URL: documentation, https://resiliparse.chatnoir.eu
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-Requires-Dist: brotli
-Requires-Dist: click
-Requires-Dist: tqdm
-Provides-Extra: all
-Requires-Dist: fastwarc[fsspec]; extra == "all"
-Provides-Extra: fsspec
-Requires-Dist: fsspec; extra == "fsspec"
-Provides-Extra: test
-Requires-Dist: fastwarc[all]; extra == "test"
-Requires-Dist: pytest>=7.2.2; extra == "test"
-Requires-Dist: pytest-cov>=4.0.0; extra == "test"
-Requires-Dist: lz4; extra == "test"
-
-# FastWARC (Python)
-
-FastWARC is a high-performance WARC parsing library for Python written in Rust with Python bindings. FastWARC supports
-compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip and LZ4.
-
-This crate contains the Python bindings for FastWARC.
-
-FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
-and robust web data processing.
-
-## Installing FastWARC
-
-Pre-built FastWARC binaries for most Linux platforms can be installed from PyPi:
-
-```bash
-pip install fastwarc
-```
-
-**However:** the Linux binaries are provided *solely for your convenience*. Since they are built on the very old
-`manylinux` base system for better compatibility, their performance isn't optimal (though still better than WARCIO). For
-best performance, see the next section on how to build FastWARC yourself.
-
-## Building FastWARC From Source
-
-You can compile FastWARC either from the PyPi source package or directly from this repository, though in any case, you
-need to install all required build-time dependencies first. On Ubuntu, this is done as follows:
-
-```bash
-sudo apt install build-essential python3-dev zlib1g-dev liblz4-dev
-```
-
-To build and install FastWARC from PyPi, run
-
-```bash
-pip install --no-binary fastwarc fastwarc
-```
-
-That's it. If you prefer to build and install directly from this repository instead, run:
-
-```bash
-pip install -e fastwarc
-```
-
-To build the wheels without installing them, run:
-
-```bash
-pip wheel -e fastwarc
-
-# Or:
-pip install build && python -m build --wheel fastwarc
-```
-
-## Usage Instructions
-
-For detailed usage instructions, please consult
-the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
-
-## Cite Us
-
-If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
-
-```bibtex
-@InProceedings{bevendorff:2021,
-  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
-  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
-  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
-  month =                 oct,
-  publisher =             {International Open Search Symposium},
-  site =                  {CERN, Geneva, Switzerland},
-  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
-  year =                  2021
-}
-```

fastwarc-1.0.1/README.md

@@ -1,73 +0,0 @@
-# FastWARC (Python)
-
-FastWARC is a high-performance WARC parsing library for Python written in Rust with Python bindings. FastWARC supports
-compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip and LZ4.
-
-This crate contains the Python bindings for FastWARC.
-
-FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
-and robust web data processing.
-
-## Installing FastWARC
-
-Pre-built FastWARC binaries for most Linux platforms can be installed from PyPi:
-
-```bash
-pip install fastwarc
-```
-
-**However:** the Linux binaries are provided *solely for your convenience*. Since they are built on the very old
-`manylinux` base system for better compatibility, their performance isn't optimal (though still better than WARCIO). For
-best performance, see the next section on how to build FastWARC yourself.
-
-## Building FastWARC From Source
-
-You can compile FastWARC either from the PyPi source package or directly from this repository, though in any case, you
-need to install all required build-time dependencies first. On Ubuntu, this is done as follows:
-
-```bash
-sudo apt install build-essential python3-dev zlib1g-dev liblz4-dev
-```
-
-To build and install FastWARC from PyPi, run
-
-```bash
-pip install --no-binary fastwarc fastwarc
-```
-
-That's it. If you prefer to build and install directly from this repository instead, run:
-
-```bash
-pip install -e fastwarc
-```
-
-To build the wheels without installing them, run:
-
-```bash
-pip wheel -e fastwarc
-
-# Or:
-pip install build && python -m build --wheel fastwarc
-```
-
-## Usage Instructions
-
-For detailed usage instructions, please consult
-the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
-
-## Cite Us
-
-If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
-
-```bibtex
-@InProceedings{bevendorff:2021,
-  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
-  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
-  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
-  month =                 oct,
-  publisher =             {International Open Search Symposium},
-  site =                  {CERN, Geneva, Switzerland},
-  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
-  year =                  2021
-}
-```