wiki-dump-extractor 0.1.0__tar.gz

wiki_dump_extractor-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Zulko
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE. 

wiki_dump_extractor-0.1.0/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.4
+Name: wiki_dump_extractor
+Version: 0.1.0
+Summary: A Python package for extracting and processing Wikipedia data
+Author: Zulko
+License: MIT
+Project-URL: Homepage, https://github.com/zulko/wiki_extractor
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Text Processing :: Markup
+Classifier: Topic :: Scientific/Engineering :: GIS
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: lxml>=4.9.0
+Requires-Dist: fastavro>=1.0.0
+Requires-Dist: zstandard>=0.23
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: tqdm>=4.65.0
+Requires-Dist: lmdb>=1.4.1
+Requires-Dist: aiostream>=0.5.0
+Requires-Dist: regex>=2023.12.0
+Provides-Extra: llm
+Requires-Dist: google-cloud-aiplatform; extra == "llm"
+Requires-Dist: google-cloud-storage; extra == "llm"
+Requires-Dist: google-genai; extra == "llm"
+Requires-Dist: pydantic; extra == "llm"
+Requires-Dist: pydantic-ai; extra == "llm"
+Requires-Dist: mwparserfromhell; extra == "llm"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+Provides-Extra: docs
+Requires-Dist: sphinx>=7.0.0; extra == "docs"
+Requires-Dist: myst-parser>=2.0.0; extra == "docs"
+Requires-Dist: shibuya==2024.10.15; extra == "docs"
+Dynamic: license-file
+
+# Wiki dump extractor
+
+A python library to extract and analyze pages from a wiki dump.
+
+This library is used in particular in the [Landnotes](https://github.com/Zulko/landnotes) project to extract and analyze pages from the Wikipedia dump.
+
+The project is hosted on [GitHub](https://github.com/zulko/wiki_dump_extractor) an the HTML documentation is available [here](https://zulko.github.io/wiki_dump_extractor/).
+
+## Scope
+
+Make the wikipedia dumps easier to work with:
+
+- Extract pages from a wiki dump
+- Be easy to install and run
+- Be fast (can iterate over 50,000 pages / secong using Avro)
+- Be memory efficient
+- Allow for batch processing and parallel processing
+
+Provide utilities for page analysis:
+
+- Date parsing
+- Section extraction
+- Text cleaning
+- and more.
+
+## Usage
+
+To simply iterate over the pages in the dump:
+
+```python
+from wiki_dump_extractor import WikiDumpExtractor
+
+dump_file = "enwiki-20220301-pages-articles-multistream.xml.bz2"
+extractor = WikiDumpExtractor(file_path=dump_file)
+for page in extractor.iter_pages(limit=1000):
+    print(page.title)
+```
+
+To extract the pages by batches (here we save the pages separate CSV files):
+
+```python
+from wiki_dump_extractor import WikiDumpExtractor
+
+dump_file = "enwiki-20220301-pages-articles-multistream.xml.bz2"
+extractor = WikiDumpExtractor(file_path=dump_file)
+batches = extractor.iter_page_batches(batch_size=1000, limit=10)
+for i, batch in enumerate(batches):
+    df = pandas.DataFrame([page.to_dict() for page in batch])
+    df.to_csv(f"batch_{i}.csv")
+```
+
+### Converting the dump to Avro
+
+There are many reasons why you might want to convert the dump to Avro. The original `xml.bz2` dump is 22Gb but very slow to read from (250/s), the uncompressed dump is 107Gb, relatively fast to read (this library uses lxml which reads thousands of pages per second), however 50% of the pages in there are empty redirect pages.
+
+The following code converts the batch to a 28G avro dump that only contains the 12 million real pages, stores redirects in a fast LMDB database, and creates an index for quick page lookups. The operation takes ~40 minutes depending on your machine.
+
+```python
+from wiki_dump_extractor import WikiXmlDumpExtractor
+
+file_path = "enwiki-20250201-pages-articles-multistream.xml"
+extractor = WikiXmlDumpExtractor(file_path=file_path)
+ignored_fields = ["timestamp", "page_id", "revision_id", "redirect_title"]
+extractor.extract_pages_to_avro(
+    output_file="wiki_dump.avro",
+    redirects_db_path="redirects.lmdb",  # LMDB database for fast redirect lookups
+    ignored_fields=ignored_fields,
+)
+```
+
+Then index the pages for fast lookups:
+
+```python
+from wiki_dump_extractor import WikiAvroDumpExtractor
+
+extractor = WikiAvroDumpExtractor(file_path="wiki_dump.avro")
+extractor.index_pages(page_index_db="page_index.lmdb")
+```
+
+Later on, read the Avro file and use redirects and index as follows (reads the 12 million pages in ~3-4 minutes depending on your machine):
+
+```python
+from wiki_dump_extractor import WikiAvroDumpExtractor
+
+# Create extractor
+extractor = WikiAvroDumpExtractor(
+    file_path="wiki_dump.avro",
+    index_dir="page_index.lmdb"  # Use the index for faster lookups
+)
+
+# Get pages with automatic redirect resolution
+pages = extractor.get_page_batch_by_title(
+    ["Page Title 1", "Page Title 2"]
+)
+```
+
+## Installation
+
+```bash
+pip install wiki-dump-extractor
+```
+
+Or from the source in development mode:
+
+```bash
+pip install -e .
+```
+
+To use the LLM-specific module (that would be mostly if you are on a project like Landnotes), use
+
+```bash
+pip install wiki-dump-extractor[llm]
+```
+
+Or locally:
+```bash
+pip install -e ".[llm]"
+```
+
+To install with tests, use `pip install -e ".[dev]"` then run the tests with `pytest` in the root directory.
+
+### Requirements for running the LLM utils
+
+```bash
+# Add the Cloud SDK distribution URI as a package source
+echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
+
+# Import the Google Cloud public key
+curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
+
+# Update the package list and install the Cloud SDK
+sudo apt-get update && sudo apt-get install google-cloud-sdk
+```

wiki_dump_extractor-0.1.0/README.md

@@ -0,0 +1,133 @@
+# Wiki dump extractor
+
+A python library to extract and analyze pages from a wiki dump.
+
+This library is used in particular in the [Landnotes](https://github.com/Zulko/landnotes) project to extract and analyze pages from the Wikipedia dump.
+
+The project is hosted on [GitHub](https://github.com/zulko/wiki_dump_extractor) an the HTML documentation is available [here](https://zulko.github.io/wiki_dump_extractor/).
+
+## Scope
+
+Make the wikipedia dumps easier to work with:
+
+- Extract pages from a wiki dump
+- Be easy to install and run
+- Be fast (can iterate over 50,000 pages / secong using Avro)
+- Be memory efficient
+- Allow for batch processing and parallel processing
+
+Provide utilities for page analysis:
+
+- Date parsing
+- Section extraction
+- Text cleaning
+- and more.
+
+## Usage
+
+To simply iterate over the pages in the dump:
+
+```python
+from wiki_dump_extractor import WikiDumpExtractor
+
+dump_file = "enwiki-20220301-pages-articles-multistream.xml.bz2"
+extractor = WikiDumpExtractor(file_path=dump_file)
+for page in extractor.iter_pages(limit=1000):
+    print(page.title)
+```
+
+To extract the pages by batches (here we save the pages separate CSV files):
+
+```python
+from wiki_dump_extractor import WikiDumpExtractor
+
+dump_file = "enwiki-20220301-pages-articles-multistream.xml.bz2"
+extractor = WikiDumpExtractor(file_path=dump_file)
+batches = extractor.iter_page_batches(batch_size=1000, limit=10)
+for i, batch in enumerate(batches):
+    df = pandas.DataFrame([page.to_dict() for page in batch])
+    df.to_csv(f"batch_{i}.csv")
+```
+
+### Converting the dump to Avro
+
+There are many reasons why you might want to convert the dump to Avro. The original `xml.bz2` dump is 22Gb but very slow to read from (250/s), the uncompressed dump is 107Gb, relatively fast to read (this library uses lxml which reads thousands of pages per second), however 50% of the pages in there are empty redirect pages.
+
+The following code converts the batch to a 28G avro dump that only contains the 12 million real pages, stores redirects in a fast LMDB database, and creates an index for quick page lookups. The operation takes ~40 minutes depending on your machine.
+
+```python
+from wiki_dump_extractor import WikiXmlDumpExtractor
+
+file_path = "enwiki-20250201-pages-articles-multistream.xml"
+extractor = WikiXmlDumpExtractor(file_path=file_path)
+ignored_fields = ["timestamp", "page_id", "revision_id", "redirect_title"]
+extractor.extract_pages_to_avro(
+    output_file="wiki_dump.avro",
+    redirects_db_path="redirects.lmdb",  # LMDB database for fast redirect lookups
+    ignored_fields=ignored_fields,
+)
+```
+
+Then index the pages for fast lookups:
+
+```python
+from wiki_dump_extractor import WikiAvroDumpExtractor
+
+extractor = WikiAvroDumpExtractor(file_path="wiki_dump.avro")
+extractor.index_pages(page_index_db="page_index.lmdb")
+```
+
+Later on, read the Avro file and use redirects and index as follows (reads the 12 million pages in ~3-4 minutes depending on your machine):
+
+```python
+from wiki_dump_extractor import WikiAvroDumpExtractor
+
+# Create extractor
+extractor = WikiAvroDumpExtractor(
+    file_path="wiki_dump.avro",
+    index_dir="page_index.lmdb"  # Use the index for faster lookups
+)
+
+# Get pages with automatic redirect resolution
+pages = extractor.get_page_batch_by_title(
+    ["Page Title 1", "Page Title 2"]
+)
+```
+
+## Installation
+
+```bash
+pip install wiki-dump-extractor
+```
+
+Or from the source in development mode:
+
+```bash
+pip install -e .
+```
+
+To use the LLM-specific module (that would be mostly if you are on a project like Landnotes), use
+
+```bash
+pip install wiki-dump-extractor[llm]
+```
+
+Or locally:
+```bash
+pip install -e ".[llm]"
+```
+
+To install with tests, use `pip install -e ".[dev]"` then run the tests with `pytest` in the root directory.
+
+### Requirements for running the LLM utils
+
+```bash
+# Add the Cloud SDK distribution URI as a package source
+echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
+
+# Import the Google Cloud public key
+curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
+
+# Update the package list and install the Cloud SDK
+sudo apt-get update && sudo apt-get install google-cloud-sdk
+```

wiki_dump_extractor-0.1.0/pyproject.toml

@@ -0,0 +1,55 @@
+[build-system]
+requires = ["setuptools>=42", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "wiki_dump_extractor"
+version = "0.1.0"
+description = "A Python package for extracting and processing Wikipedia data"
+readme = "README.md"
+authors = [{ name = "Zulko" }]
+license = { text = "MIT" }
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Topic :: Text Processing :: Markup",
+    "Topic :: Scientific/Engineering :: GIS",
+]
+requires-python = ">=3.8"
+dependencies = [
+    "lxml>=4.9.0",
+    "fastavro>=1.0.0",
+    "zstandard>=0.23",
+    "pandas>=2.0.0",
+    "tqdm>=4.65.0",
+    "lmdb>=1.4.1",
+    "aiostream>=0.5.0",
+    "regex>=2023.12.0",
+]
+
+[project.optional-dependencies]
+llm = [
+    "google-cloud-aiplatform",
+    "google-cloud-storage",
+    "google-genai",
+    "pydantic",
+    "pydantic-ai",
+    "mwparserfromhell",
+]
+test = ["pytest>=7.0.0", "pytest-cov>=4.0.0"]
+
+docs = ["sphinx>=7.0.0", "myst-parser>=2.0.0", "shibuya==2024.10.15"]
+
+[project.urls]
+"Homepage" = "https://github.com/zulko/wiki_extractor"
+
+[tool.setuptools]
+package-dir = { "" = "src" }
+packages = ["wiki_dump_extractor"]
+
+[tool.pytest.ini_options]
+testpaths = ["test"]
+python_files = "test_*.py"
+python_functions = "test_*"
+addopts = "--cov=wiki_dump_extractor"

wiki_dump_extractor-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

wiki_dump_extractor-0.1.0/src/wiki_dump_extractor/__init__.py

@@ -0,0 +1,14 @@
+"""My Project package."""
+
+__version__ = "0.1.0"
+
+from .wiki_dump_extractor import WikiXmlDumpExtractor, WikiAvroDumpExtractor
+from .wiki_sql_extractor import WikiSqlExtractor
+from .download_utils import download_file
+
+__all__ = [
+    "WikiXmlDumpExtractor",
+    "WikiAvroDumpExtractor",
+    "WikiSqlExtractor",
+    "download_file",
+]