FastWARC 1.0.1__tar.gz → 1.0.3__tar.gz

fastwarc-1.0.3/FastWARC.egg-info/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.4
+Name: FastWARC
+Version: 1.0.3
+Summary: A high-performance WARC parsing library written in Rust with bindings for Python.
+Author: Janek Bevendorff
+License-Expression: Apache-2.0
+Project-URL: repository, https://github.com/chatnoir-eu/chatnoir-resiliparse
+Project-URL: homepage, https://resiliparse.chatnoir.eu
+Project-URL: documentation, https://resiliparse.chatnoir.eu
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: brotli
+Requires-Dist: click
+Requires-Dist: tqdm
+Provides-Extra: all
+Requires-Dist: fastwarc[fsspec]; extra == "all"
+Provides-Extra: fsspec
+Requires-Dist: fsspec; extra == "fsspec"
+Requires-Dist: s3fs; extra == "fsspec"
+Provides-Extra: test
+Requires-Dist: fastwarc[all]; extra == "test"
+Requires-Dist: pytest>=7.2.2; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+
+# FastWARC (Python)
+
+FastWARC is a high-performance WARC parsing library written in Rust with bindings for Python.
+FastWARC supports compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip,
+Zstd, and LZ4.
+
+This package contains the Python bindings for FastWARC.
+
+FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
+and robust web data processing.
+
+## Installing FastWARC
+
+Pre-built FastWARC binaries for Python can be installed from [PyPi](https://pypi.org/project/FastWARC/):
+
+```bash
+pip install fastwarc
+```
+
+## Building Python Bindings From Source
+
+To be able to build FastWARC, you need the Rust toolchain installed (see above). Then to build directly from the GitHub
+repository, follow these steps:
+
+```bash
+# Clone repository
+
+git clone https://github.com/chatnoir-eu/chatnoir-resiliparse.git
+cd chatnoir-resiliparse
+
+# Install Rust toolchain (if not already installed)
+
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+
+# Create a fresh venv first (recommended)
+
+python3 -m venv venv && source venv/bin/activate
+
+# Option 1: Build and install in editable mode (best for development)
+
+python3 -m pip install -e ./fastwarc-py
+
+# Option 2 (alternative): Build and install wheels in separate steps (best for redistribution)
+
+python3 -m pip wheel -w build ./fastwarc-py
+ls ./build/*.whl | xargs python3 -m pip install
+```
+
+## Usage Instructions
+
+For detailed usage instructions, please consult
+the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
+
+## Cite Us
+
+If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
+
+```bibtex
+@InProceedings{bevendorff:2021,
+  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
+  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
+  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
+  month =                 oct,
+  publisher =             {International Open Search Symposium},
+  site =                  {CERN, Geneva, Switzerland},
+  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
+  year =                  2021
+}
+```
+
+You can also cite our [ECIR 2018 demo paper](https://webis.de/downloads/publications/papers/bevendorff_2018.pdf):
+
+```bibtex
+@InProceedings{bevendorff:2018,
+  address =             {Berlin Heidelberg New York},
+  author =              {Janek Bevendorff and Benno Stein and Matthias Hagen and Martin Potthast},
+  booktitle =           {Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018)},
+  editor =              {Leif Azzopardi and Allan Hanbury and Gabriella Pasi and Benjamin Piwowarski},
+  month =               mar,
+  publisher =           {Springer},
+  series =              {Lecture Notes in Computer Science},
+  site =                {Grenoble, France},
+  title =               {{Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl}},
+  year =                2018
+}
+```

{fastwarc-1.0.1 → fastwarc-1.0.3}/FastWARC.egg-info/SOURCES.txt

@@ -16,6 +16,7 @@ fastwarc/stream_io.pyi
 fastwarc/warc.pyi
 fastwarc/legacy/__init__.pxd
 fastwarc/legacy/__init__.py
+fastwarc/legacy/__init__.pyi
 fastwarc/legacy/_shims.py
 fastwarc/legacy/stream_io.pxd
 fastwarc/legacy/stream_io.pyi

{fastwarc-1.0.1 → fastwarc-1.0.3}/FastWARC.egg-info/requires.txt

@@ -7,9 +7,9 @@ fastwarc[fsspec]
 
 [fsspec]
 fsspec
+s3fs
 
 [test]
 fastwarc[all]
 pytest>=7.2.2
 pytest-cov>=4.0.0
-lz4

fastwarc-1.0.3/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.4
+Name: FastWARC
+Version: 1.0.3
+Summary: A high-performance WARC parsing library written in Rust with bindings for Python.
+Author: Janek Bevendorff
+License-Expression: Apache-2.0
+Project-URL: repository, https://github.com/chatnoir-eu/chatnoir-resiliparse
+Project-URL: homepage, https://resiliparse.chatnoir.eu
+Project-URL: documentation, https://resiliparse.chatnoir.eu
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: brotli
+Requires-Dist: click
+Requires-Dist: tqdm
+Provides-Extra: all
+Requires-Dist: fastwarc[fsspec]; extra == "all"
+Provides-Extra: fsspec
+Requires-Dist: fsspec; extra == "fsspec"
+Requires-Dist: s3fs; extra == "fsspec"
+Provides-Extra: test
+Requires-Dist: fastwarc[all]; extra == "test"
+Requires-Dist: pytest>=7.2.2; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+
+# FastWARC (Python)
+
+FastWARC is a high-performance WARC parsing library written in Rust with bindings for Python.
+FastWARC supports compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip,
+Zstd, and LZ4.
+
+This package contains the Python bindings for FastWARC.
+
+FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
+and robust web data processing.
+
+## Installing FastWARC
+
+Pre-built FastWARC binaries for Python can be installed from [PyPi](https://pypi.org/project/FastWARC/):
+
+```bash
+pip install fastwarc
+```
+
+## Building Python Bindings From Source
+
+To be able to build FastWARC, you need the Rust toolchain installed (see above). Then to build directly from the GitHub
+repository, follow these steps:
+
+```bash
+# Clone repository
+
+git clone https://github.com/chatnoir-eu/chatnoir-resiliparse.git
+cd chatnoir-resiliparse
+
+# Install Rust toolchain (if not already installed)
+
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+
+# Create a fresh venv first (recommended)
+
+python3 -m venv venv && source venv/bin/activate
+
+# Option 1: Build and install in editable mode (best for development)
+
+python3 -m pip install -e ./fastwarc-py
+
+# Option 2 (alternative): Build and install wheels in separate steps (best for redistribution)
+
+python3 -m pip wheel -w build ./fastwarc-py
+ls ./build/*.whl | xargs python3 -m pip install
+```
+
+## Usage Instructions
+
+For detailed usage instructions, please consult
+the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
+
+## Cite Us
+
+If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
+
+```bibtex
+@InProceedings{bevendorff:2021,
+  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
+  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
+  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
+  month =                 oct,
+  publisher =             {International Open Search Symposium},
+  site =                  {CERN, Geneva, Switzerland},
+  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
+  year =                  2021
+}
+```
+
+You can also cite our [ECIR 2018 demo paper](https://webis.de/downloads/publications/papers/bevendorff_2018.pdf):
+
+```bibtex
+@InProceedings{bevendorff:2018,
+  address =             {Berlin Heidelberg New York},
+  author =              {Janek Bevendorff and Benno Stein and Matthias Hagen and Martin Potthast},
+  booktitle =           {Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018)},
+  editor =              {Leif Azzopardi and Allan Hanbury and Gabriella Pasi and Benjamin Piwowarski},
+  month =               mar,
+  publisher =           {Springer},
+  series =              {Lecture Notes in Computer Science},
+  site =                {Grenoble, France},
+  title =               {{Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl}},
+  year =                2018
+}
+```

fastwarc-1.0.3/README.md

@@ -0,0 +1,86 @@
+# FastWARC (Python)
+
+FastWARC is a high-performance WARC parsing library written in Rust with bindings for Python.
+FastWARC supports compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip,
+Zstd, and LZ4.
+
+This package contains the Python bindings for FastWARC.
+
+FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
+and robust web data processing.
+
+## Installing FastWARC
+
+Pre-built FastWARC binaries for Python can be installed from [PyPi](https://pypi.org/project/FastWARC/):
+
+```bash
+pip install fastwarc
+```
+
+## Building Python Bindings From Source
+
+To be able to build FastWARC, you need the Rust toolchain installed (see above). Then to build directly from the GitHub
+repository, follow these steps:
+
+```bash
+# Clone repository
+
+git clone https://github.com/chatnoir-eu/chatnoir-resiliparse.git
+cd chatnoir-resiliparse
+
+# Install Rust toolchain (if not already installed)
+
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+
+# Create a fresh venv first (recommended)
+
+python3 -m venv venv && source venv/bin/activate
+
+# Option 1: Build and install in editable mode (best for development)
+
+python3 -m pip install -e ./fastwarc-py
+
+# Option 2 (alternative): Build and install wheels in separate steps (best for redistribution)
+
+python3 -m pip wheel -w build ./fastwarc-py
+ls ./build/*.whl | xargs python3 -m pip install
+```
+
+## Usage Instructions
+
+For detailed usage instructions, please consult
+the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
+
+## Cite Us
+
+If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
+
+```bibtex
+@InProceedings{bevendorff:2021,
+  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
+  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
+  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
+  month =                 oct,
+  publisher =             {International Open Search Symposium},
+  site =                  {CERN, Geneva, Switzerland},
+  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
+  year =                  2021
+}
+```
+
+You can also cite our [ECIR 2018 demo paper](https://webis.de/downloads/publications/papers/bevendorff_2018.pdf):
+
+```bibtex
+@InProceedings{bevendorff:2018,
+  address =             {Berlin Heidelberg New York},
+  author =              {Janek Bevendorff and Benno Stein and Matthias Hagen and Martin Potthast},
+  booktitle =           {Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018)},
+  editor =              {Leif Azzopardi and Allan Hanbury and Gabriella Pasi and Benjamin Piwowarski},
+  month =               mar,
+  publisher =           {Springer},
+  series =              {Lecture Notes in Computer Science},
+  site =                {Grenoble, France},
+  title =               {{Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl}},
+  year =                2018
+}
+```

{fastwarc-1.0.1 → fastwarc-1.0.3}/fastwarc/cli.py

@@ -24,8 +24,8 @@ import urllib.request
 import click
 from tqdm import tqdm
 
-from fastwarc.legacy.stream_io import FileStream, StreamError, FastWARCError, PythonIOStreamAdapter
-from fastwarc.legacy.warc import ArchiveIterator, WarcRecordType
+from fastwarc.stream_io import FileStream, StreamError, FastWARCError, PythonIOStreamAdapter
+from fastwarc.warc import ArchiveIterator, WarcRecordType
 from fastwarc.legacy.tools import CompressionAlg, detect_compression_algorithm, wrap_warc_stream, \
     recompress_warc_interactive, verify_digests
 
@@ -439,8 +439,9 @@ def benchmark(input_url, decompress_alg, endpoint_url, aws_access_key, aws_secre
         return num, time.monotonic() - start, interrupted
 
     def _fastwarc_iterator(f):
-        s = _get_raw_stream_from_url(f, use_python_stream)
-        s = wrap_warc_stream(s, 'rb', decompress_alg)
+        # s = _get_raw_stream_from_url(f, use_python_stream)
+        # s = wrap_warc_stream(s, 'rb', decompress_alg)
+        s = '/home/roce3528/code-in-progress/code-research/web-search/chatnoir/chatnoir-resiliparse/tmpfs/CC-MAIN-20231005012006-20231005042006-00899.warc'
         return ArchiveIterator(s, rec_type_filter, parse_http=parse_http, verify_digests=verify_digests)
 
     n, t_fastwarc, interrupted = _bench(input_url, _fastwarc_iterator, 'FastWARC')

fastwarc-1.0.3/fastwarc/legacy/__init__.pyi

@@ -0,0 +1,19 @@
+from .stream_io import (
+    FileStream,
+    GZipStream,
+    LZ4Stream,
+    FastWARCError,
+    StreamError,
+)
+from .warc import ArchiveIterator, WarcRecord, WarcRecordType
+
+__all__ = [
+    "FileStream",
+    "GZipStream",
+    "LZ4Stream",
+    "FastWARCError",
+    "StreamError",
+    "ArchiveIterator",
+    "WarcRecord",
+    "WarcRecordType",
+]

{fastwarc-1.0.1 → fastwarc-1.0.3}/fastwarc/stream_io.pyi

@@ -128,54 +128,56 @@ _WriterInput = WarcWriter | BinaryIO | _GenericWriter | PathLike[str] | str
 
 @disjoint_base
 class GzipReader(WarcReader):
-    def __new__(cls, inner: _ReaderInput, buffer_size=4096, zlib=False, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _ReaderInput, buffer_size: int = 64 << 10, zlib: bool = False,
+                fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class GzipWriter(WarcWriter):
-    def __new__(cls, inner: _WriterInput, compression_level=9, buffer_size=8192, zlib=False,
+    def __new__(cls, inner: _WriterInput, compression_level: int = 9, buffer_size: int = 8192, zlib=False,
                 fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class ZstdReader(WarcReader):
-    def __new__(cls, inner: _ReaderInput, buffer_size=4096, fsspec_args=None, dictionary=None) -> Self: ...
+    def __new__(cls, inner: _ReaderInput, buffer_size: int = 64 << 10, fsspec_args=None,
+                dictionary: bytes | None = None) -> Self: ...
 
 
 @disjoint_base
 class ZstdWriter(WarcWriter):
-    def __new__(cls, inner: _WriterInput, buffer_size=8192, fsspec_args=None, dictionary=None,
-                compress_dictionary_frame=False) -> Self: ...
+    def __new__(cls, inner: _WriterInput, buffer_size: int = 8192, compression_level: int = 3, fsspec_args=None,
+                dictionary: bytes | None = None, compress_dictionary_frame=False) -> Self: ...
 
 
 @disjoint_base
 class Lz4Reader(WarcReader):
-    def __new__(cls, inner: _ReaderInput, buffer_size=4096, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _ReaderInput, buffer_size: int = 64 << 10, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class Lz4Writer(WarcWriter):
-    def __new__(cls, inner: _WriterInput, buffer_size=8192, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _WriterInput, buffer_size: int = 8192, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class BrotliReader(WarcReader):
-    def __new__(cls, inner: _ReaderInput, buffer_size=4096, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _ReaderInput, buffer_size: int = 64 << 10, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class BrotliWriter(WarcWriter):
-    def __new__(cls, inner: _WriterInput, buffer_size=8192, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _WriterInput, buffer_size: int = 8192, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class ChunkedReader(WarcReader):
-    def __new__(cls, inner: _ReaderInput, buffer_size=1024, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _ReaderInput, buffer_size: int = 4096, fsspec_args=None) -> Self: ...
 
 
 @disjoint_base
 class ChunkedWriter(WarcWriter):
-    def __new__(cls, inner: _WriterInput, min_chunk_size=512, fsspec_args=None) -> Self: ...
+    def __new__(cls, inner: _WriterInput, min_chunk_size: int = 1024, fsspec_args=None) -> Self: ...
 
 
 def zstd_train_dictionary_from_continuous(sample_data: bytes, sample_sizes: list[int], max_size: int) -> bytes: ...

{fastwarc-1.0.1 → fastwarc-1.0.3}/fastwarc/warc.pyi

@@ -213,6 +213,12 @@ class WarcRecordPayloadReader(WarcReader):
 class WarcRecord:
     def __new__(cls) -> Self: ...
 
+    @classmethod
+    def from_bytes(cls, payload: bytes) -> Self: ...
+
+    @classmethod
+    def from_reader(cls, reader: _ReaderInput, quirks_mode: bool = False) -> Self: ...
+
     @property
     def record_id(self) -> str | None: ...
 
@@ -298,6 +304,10 @@ class WarcRecord:
             chunk_size: int = 16384
     ) -> int: ...
 
+    def __eq__(self, other, /) -> bool: ...
+
+    def __ne__(self, other, /) -> bool: ...
+
     def __str__(self) -> str: ...
 
     def __repr__(self) -> str: ...
@@ -323,6 +333,7 @@ class ArchiveIterator(Iterable[WarcRecord]):
             quirks_mode: bool = False,
             auto_decode: Literal['none', 'content', 'transfer', 'all'] = 'none',
             stream_detect: bool = True,
+            buffer_size: int = 64 << 10,
             fsspec_args=None,
             *,
             strict_mode: bool = True

{fastwarc-1.0.1 → fastwarc-1.0.3}/pyproject.toml

@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "FastWARC"
-version = "1.0.1"
-description = "A high-performance WARC parsing library for Python written in C++/Cython."
+version = "1.0.3"
+description = "A high-performance WARC parsing library written in Rust with bindings for Python."
 authors = [
     { name = "Janek Bevendorff" }
 ]
 license = "Apache-2.0"
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.10"
 dependencies = [
     "brotli",
     "click",
@@ -28,20 +28,20 @@ all = [
     "fastwarc[fsspec]"
 ]
 fsspec = [
-    "fsspec"
+    "fsspec",
+    "s3fs",
 ]
 test = [
     "fastwarc[all]",
     "pytest>=7.2.2",
     "pytest-cov>=4.0.0",
-    "lz4",
 ]
 
 [project.scripts]
 fastwarc = "fastwarc.cli:main"
 
 [tool.setuptools.packages.find]
-include = ["fastwarc*"]
+include = ["fastwarc", "fastwarc.legacy"]
 
 [tool.setuptools.package-data]
 "*" = ["*.pyi"]

fastwarc-1.0.1/FastWARC.egg-info/PKG-INFO

@@ -1,97 +0,0 @@
-Metadata-Version: 2.4
-Name: FastWARC
-Version: 1.0.1
-Summary: A high-performance WARC parsing library for Python written in C++/Cython.
-Author: Janek Bevendorff
-License-Expression: Apache-2.0
-Project-URL: repository, https://github.com/chatnoir-eu/chatnoir-resiliparse
-Project-URL: homepage, https://resiliparse.chatnoir.eu
-Project-URL: documentation, https://resiliparse.chatnoir.eu
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-Requires-Dist: brotli
-Requires-Dist: click
-Requires-Dist: tqdm
-Provides-Extra: all
-Requires-Dist: fastwarc[fsspec]; extra == "all"
-Provides-Extra: fsspec
-Requires-Dist: fsspec; extra == "fsspec"
-Provides-Extra: test
-Requires-Dist: fastwarc[all]; extra == "test"
-Requires-Dist: pytest>=7.2.2; extra == "test"
-Requires-Dist: pytest-cov>=4.0.0; extra == "test"
-Requires-Dist: lz4; extra == "test"
-
-# FastWARC (Python)
-
-FastWARC is a high-performance WARC parsing library for Python written in Rust with Python bindings. FastWARC supports
-compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip and LZ4.
-
-This crate contains the Python bindings for FastWARC.
-
-FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
-and robust web data processing.
-
-## Installing FastWARC
-
-Pre-built FastWARC binaries for most Linux platforms can be installed from PyPi:
-
-```bash
-pip install fastwarc
-```
-
-**However:** the Linux binaries are provided *solely for your convenience*. Since they are built on the very old
-`manylinux` base system for better compatibility, their performance isn't optimal (though still better than WARCIO). For
-best performance, see the next section on how to build FastWARC yourself.
-
-## Building FastWARC From Source
-
-You can compile FastWARC either from the PyPi source package or directly from this repository, though in any case, you
-need to install all required build-time dependencies first. On Ubuntu, this is done as follows:
-
-```bash
-sudo apt install build-essential python3-dev zlib1g-dev liblz4-dev
-```
-
-To build and install FastWARC from PyPi, run
-
-```bash
-pip install --no-binary fastwarc fastwarc
-```
-
-That's it. If you prefer to build and install directly from this repository instead, run:
-
-```bash
-pip install -e fastwarc
-```
-
-To build the wheels without installing them, run:
-
-```bash
-pip wheel -e fastwarc
-
-# Or:
-pip install build && python -m build --wheel fastwarc
-```
-
-## Usage Instructions
-
-For detailed usage instructions, please consult
-the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
-
-## Cite Us
-
-If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
-
-```bibtex
-@InProceedings{bevendorff:2021,
-  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
-  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
-  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
-  month =                 oct,
-  publisher =             {International Open Search Symposium},
-  site =                  {CERN, Geneva, Switzerland},
-  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
-  year =                  2021
-}
-```

fastwarc-1.0.1/PKG-INFO

@@ -1,97 +0,0 @@
-Metadata-Version: 2.4
-Name: FastWARC
-Version: 1.0.1
-Summary: A high-performance WARC parsing library for Python written in C++/Cython.
-Author: Janek Bevendorff
-License-Expression: Apache-2.0
-Project-URL: repository, https://github.com/chatnoir-eu/chatnoir-resiliparse
-Project-URL: homepage, https://resiliparse.chatnoir.eu
-Project-URL: documentation, https://resiliparse.chatnoir.eu
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-Requires-Dist: brotli
-Requires-Dist: click
-Requires-Dist: tqdm
-Provides-Extra: all
-Requires-Dist: fastwarc[fsspec]; extra == "all"
-Provides-Extra: fsspec
-Requires-Dist: fsspec; extra == "fsspec"
-Provides-Extra: test
-Requires-Dist: fastwarc[all]; extra == "test"
-Requires-Dist: pytest>=7.2.2; extra == "test"
-Requires-Dist: pytest-cov>=4.0.0; extra == "test"
-Requires-Dist: lz4; extra == "test"
-
-# FastWARC (Python)
-
-FastWARC is a high-performance WARC parsing library for Python written in Rust with Python bindings. FastWARC supports
-compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip and LZ4.
-
-This crate contains the Python bindings for FastWARC.
-
-FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
-and robust web data processing.
-
-## Installing FastWARC
-
-Pre-built FastWARC binaries for most Linux platforms can be installed from PyPi:
-
-```bash
-pip install fastwarc
-```
-
-**However:** the Linux binaries are provided *solely for your convenience*. Since they are built on the very old
-`manylinux` base system for better compatibility, their performance isn't optimal (though still better than WARCIO). For
-best performance, see the next section on how to build FastWARC yourself.
-
-## Building FastWARC From Source
-
-You can compile FastWARC either from the PyPi source package or directly from this repository, though in any case, you
-need to install all required build-time dependencies first. On Ubuntu, this is done as follows:
-
-```bash
-sudo apt install build-essential python3-dev zlib1g-dev liblz4-dev
-```
-
-To build and install FastWARC from PyPi, run
-
-```bash
-pip install --no-binary fastwarc fastwarc
-```
-
-That's it. If you prefer to build and install directly from this repository instead, run:
-
-```bash
-pip install -e fastwarc
-```
-
-To build the wheels without installing them, run:
-
-```bash
-pip wheel -e fastwarc
-
-# Or:
-pip install build && python -m build --wheel fastwarc
-```
-
-## Usage Instructions
-
-For detailed usage instructions, please consult
-the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
-
-## Cite Us
-
-If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
-
-```bibtex
-@InProceedings{bevendorff:2021,
-  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
-  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
-  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
-  month =                 oct,
-  publisher =             {International Open Search Symposium},
-  site =                  {CERN, Geneva, Switzerland},
-  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
-  year =                  2021
-}
-```

fastwarc-1.0.1/README.md

@@ -1,73 +0,0 @@
-# FastWARC (Python)
-
-FastWARC is a high-performance WARC parsing library for Python written in Rust with Python bindings. FastWARC supports
-compressed and uncompressed WARC/1.0 and WARC/1.1 streams. Supported compression algorithms are Gzip and LZ4.
-
-This crate contains the Python bindings for FastWARC.
-
-FastWARC belongs to the [ChatNoir Resiliparse toolkit](https://github.com/chatnoir-eu/chatnoir-resiliparse/) for fast
-and robust web data processing.
-
-## Installing FastWARC
-
-Pre-built FastWARC binaries for most Linux platforms can be installed from PyPi:
-
-```bash
-pip install fastwarc
-```
-
-**However:** the Linux binaries are provided *solely for your convenience*. Since they are built on the very old
-`manylinux` base system for better compatibility, their performance isn't optimal (though still better than WARCIO). For
-best performance, see the next section on how to build FastWARC yourself.
-
-## Building FastWARC From Source
-
-You can compile FastWARC either from the PyPi source package or directly from this repository, though in any case, you
-need to install all required build-time dependencies first. On Ubuntu, this is done as follows:
-
-```bash
-sudo apt install build-essential python3-dev zlib1g-dev liblz4-dev
-```
-
-To build and install FastWARC from PyPi, run
-
-```bash
-pip install --no-binary fastwarc fastwarc
-```
-
-That's it. If you prefer to build and install directly from this repository instead, run:
-
-```bash
-pip install -e fastwarc
-```
-
-To build the wheels without installing them, run:
-
-```bash
-pip wheel -e fastwarc
-
-# Or:
-pip install build && python -m build --wheel fastwarc
-```
-
-## Usage Instructions
-
-For detailed usage instructions, please consult
-the [FastWARC User Manual](https://resiliparse.chatnoir.eu/en/latest/man/fastwarc.html).
-
-## Cite Us
-
-If you use FastWARC, please consider citing our [OSSYM 2021 abstract paper](https://arxiv.org/abs/2112.03103):
-
-```bibtex
-@InProceedings{bevendorff:2021,
-  author =                {Janek Bevendorff and Martin Potthast and Benno Stein},
-  booktitle =             {3rd International Symposium on Open Search Technology (OSSYM 2021)},
-  editor =                {Andreas Wagner and Christian Guetl and Michael Granitzer and Stefan Voigt},
-  month =                 oct,
-  publisher =             {International Open Search Symposium},
-  site =                  {CERN, Geneva, Switzerland},
-  title =                 {{FastWARC: Optimizing Large-Scale Web Archive Analytics}},
-  year =                  2021
-}
-```