wiktionary-de-parser 0.12.7__tar.gz

wiktionary_de_parser-0.12.7/CHANGELOG.md

@@ -0,0 +1,254 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.12.7] - 2024-12-31
+### Changed
+- Update dependencies
+- Improved meanings parsing (experimental)
+
+## [0.12.5] - 2024-12-30
+### Changed
+- Update dependencies
+- Improved meanings parsing (experimental)
+
+## [0.12.1] - 2024-12-30
+### Added
+- Parse meanings and add "meanings" field to output, when `include_meanings`-param ist True in `parse_entry`-call.
+
+## [0.12.0] - 2024-07-29
+### Changed
+- Update value for xml property "xsi:schemaLocation" to "http://www.mediawiki.org/xml/export-0.11/"
+
+## [0.11.5] - 2024-02-10
+### Removed
+- "wikicode" field from page model (not used)
+
+### Fixed
+- Performance improvements
+
+## [0.11.4] - 2024-02-09
+### Changed
+- Add dict comprehension to improve performance
+
+## [0.11.3] - 2024-02-09
+### Changed
+- Add None type to language
+- Small improvements
+
+## [0.11.2] - 2024-02-09
+### Changed
+- Rename "syllables" to "hyphenation"
+
+## [0.11.1] - 2024-02-05
+### Changed
+- Allow to specify path to dump file in `WiktionaryDump` class
+
+## [0.11.0] - 2024-02-04
+### Changed
+- Update dependencies
+- Refactor internally to use `pydantic` models
+
+### Removed
+- Remove `Record` class
+- Remove config options
+
+### Added
+- Add pydantic models
+- Add WiktionaryParser and WiktionaryDump classes
+
+## [0.10.1] - 2024-01-29
+### Changed
+- pass parsed wikitext internally to extraction methods
+- update tests
+
+## [0.10.0] - 2024-01-29
+### Changed
+- Update dependencies
+- Use dataclasses instead of dicts internally
+### Added
+- Add "page_id" and "index" field to output (if a page contains multiple entries, the index indicates the position of the word in the page)
+- Add tests for POS and language parsing
+### Removed
+- __BREAKING__: Removed the ability to load custom methods from outside the package. The same can be achieved by setting the "wiki_text" field in the config dict and parsing the Wikitext manually.
+
+## [0.9.5] - 2022-07-26
+### Fixed
+- Make sure "title" is of base string type (not `etree._ElementUnicodeResult`)
+
+## [0.9.4] - 2022-07-18
+### Changed
+- Improve typing
+
+## [0.9.3] - 2022-07-18
+### Fixed
+- Fix type errors
+### Added
+- Add method to parse rhymes
+- Add tests for rhymes parsing
+
+## [0.9.2] - 2022-07-17
+### Fixed
+- Improve lemma parsing
+### Added
+- Add tests for lemma parsing
+
+## [0.9.1] - 2022-07-15
+### Fixed
+- Make config dict keys optional
+
+## [0.9.0] - 2022-07-13
+### Added
+- Add development instructions to README.md
+- Add tests for syllable parsing
+- Add tests for IPA parsing
+- Add VSCode launch.json
+- Add config dict
+- Add config option to optionally include wikitext in output (disabled by default)
+### Changed
+- Update dependencies
+- Replace Autopep with black
+- Ignore inflected forms, regional slang, Austrian/Swiss dialect etc. when parsing IPA-templates from now on
+- `ignored_prefixes` is now part of a config dict
+### Fixed
+- Improve syllable parsing
+- Improve IPA parsing
+### Removed
+- `pyphen` as fallback for syllables parsing
+
+
+## [0.8.9] - 2021-11-13
+### Changed
+- Change repository and package name from `wiktionary_de_parser` to `wiktionary-de-parser`
+
+## [0.8.8] - 2021-11-12
+### Changed
+- Make `lemma` and `inflected` fields required fields
+
+## [0.8.7] - 2021-11-12
+### Changed
+- Removed typing_extensions again
+
+## [0.8.6] - 2021-11-12
+### Added
+- Added typing_extensions
+
+## [0.8.5] - 2021-11-12
+### Added
+- More type hints
+
+## [0.8.4] - 2021-11-12
+### Added
+- Type hint for iterable (Record)
+### Changed
+- removed None type dict entries in flexion parsing result
+### Fixed
+- minor flexion parsing improvements
+
+## [0.8.3] - 2021-11-12
+### Changed
+- Converted repository to [Poetry](https://python-poetry.org/) project
+- Renamed `langCode` to `lang_code`
+### Added
+- Started to implement tests and type hints
+### Fixed
+- Updated regular expression and improved flexion parsing
+
+## [0.8.1] - 2020-07-10
+### Fixed
+- improve dash parsing in table values
+
+## [0.8.0] - 2019-12-01
+### Fixed
+- `MANIFEST.in` added langcode files
+
+## [0.7.9] - 2019-12-01
+### Fixed
+- `syllables.py` improvemed syllables parsing
+### Added
+- `language.py` added field `langCode` (providing ISO639-1 language code)
+### Changed
+- `language.py` renamed field `language` to `lang`
+- `README.md` updated readme
+
+## [0.7.8] - 2019-12-01
+### Fixed
+- `ipa.py` IPA parsing improvement
+
+## [0.7.7] - 2019-07-16
+### Fixed
+- `pos.py` added 'Deklinierte Form' as POS (can be Substantiv, Adjektiv, Artikel, Pronomen)
+
+## [0.7.6] - 2019-07-13
+### Fixed
+- `ipa.py` Match correct paragraph in WikiText for parsing IPA
+
+## [0.7.5] - 2019-07-13
+### Fixed
+- `syllables.py` Improved syllables parsing
+
+## [0.7.4] - 2019-07-13
+### Changed
+- `ipa.py` Make IPA field a `list` (support multiple IPA transcriptions for one word)
+
+### Fixed
+- `ipa.py` Improved IPA parsing
+
+## [0.7.3] - 2019-05-29
+### Fixed
+- `pos.py` Prevent duplicate POS names
+
+## [0.7.2] - 2019-05-29
+### Fixed
+- `pos.py` Toponym was a Dict key, when Template 'Deutsch Toponym Übersicht' was present (should be nested noun value)
+
+## [0.7.1] - 2019-05-27
+### Added
+- [Python package](https://pypi.org) support
+
+### Changed
+- repository structure
+- README.md
+
+## [0.6.6] - 2019-04-14
+### Added
+- allow 'Genus 1' - 'Genus 4' in flexion dictionary
+- added `inflected` field to indicate whether entry is for inflected word
+
+### Changed
+- put 'Genus' back to to flexion dictionary
+
+### Fixed
+- strip values in `lemma.py`, `language.py`, `ipa.py`
+
+## [0.6.5] - 2019-04-14
+### Added
+- accept `Vorlage-Test` in regex pattern in `pos.py` & `language.py`
+- accept `Merkspruch` in `pos.py`
+
+### Fixed
+- improved regex for section splitting
+- improved regex for POS matching
+- fix missing POS names when there is a POS template
+
+### Removed
+- language codes
+
+## [0.6.0] - 2019-04-12
+### Added
+- loading custom methods via `custom_methods` argument in class constructor and `load_methods` function
+- Changelog.md (this file)
+
+### Changed
+- load all files from `methods` folder and initialize them as extraction methods
+- extraction methods must return a `Dict()` now
+- `flexion.py`: returns 'genus' and flexion info separately
+
+### Removed
+- `method_names` in `__init__.py`
+
+## [0.5.0] - 2019-04-11
+### Added
+- initial release

wiktionary_de_parser-0.12.7/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Gregor Weichbrodt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

wiktionary_de_parser-0.12.7/PKG-INFO

@@ -0,0 +1,155 @@
+Metadata-Version: 2.1
+Name: wiktionary-de-parser
+Version: 0.12.7
+Summary: Extracts data from German Wiktionary dump files.
+Home-page: https://github.com/gambolputty/wiktionary-de-parser
+License: MIT
+Keywords: wiktionary,xml,parser,data-extraction,german,nlp
+Author: Gregor Weichbrodt
+Author-email: gregorweichbrodt@gmail.com
+Requires-Python: >=3.11,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: German
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Topic :: Text Processing :: Markup :: XML
+Requires-Dist: black (>=24.1.1,<25.0.0)
+Requires-Dist: lxml (>=5.1.0,<6.0.0)
+Requires-Dist: mwparserfromhell (>=0.6.6,<0.7.0)
+Requires-Dist: pydantic (>=2.6.0,<3.0.0)
+Requires-Dist: requests (>=2.31.0,<3.0.0)
+Requires-Dist: tqdm (>=4.66.1,<5.0.0)
+Requires-Dist: wikitextparser (>=0.56.2,<0.57.0)
+Project-URL: Bug Tracker, https://github.com/gambolputty/wiktionary-de-parser/issues
+Project-URL: Repository, https://github.com/gambolputty/wiktionary-de-parser
+Description-Content-Type: text/markdown
+
+# wiktionary-de-parser
+
+A Python module to extract data from German Wiktionary XML files (for Python 3.11+).
+
+## Features
+
+- Extracts _IPA transcriptions_, _hyphenation_, _language_, _part of speech_ information (basic), _genus_ and _flexion tables_ of a word.
+- Yields per entry, not per page (a page can have multiple entries/ words can have different meanings)
+
+## Installation
+
+`pip install wiktionary-de-parser`
+
+Or with [Poetry](https://python-poetry.org/):
+
+`poetry add wiktionary-de-parser`
+
+## Usage
+
+### Loading the XML dump file
+```python
+from wiktionary_de_parser import WiktionaryParser
+from wiktionary_de_parser.dump_processor import WiktionaryDump
+
+# To download the dump file, specify the directory where the
+# dump file should be stored.
+dump = WiktionaryDump(dump_dir_path="directory-of-dump-file")
+
+# This will download "dewiktionary-latest-pages-articles-multistream.xml.bz2" to
+# the directory specified in `dump_dir_path`.
+dump.download_dump()
+
+# Alternatively you can specify a different dump file to download.
+dump = WiktionaryDump(
+    dump_dir_path="directory-of-dump-file",
+    dump_download_url="url-to-dump-file.xml.bz2",
+)
+dump.download_dump()
+
+# If you already have the dump file locally, specify the path to the file.
+dump = WiktionaryDump(dump_file_path="path-to-dump-file.xml.bz2")
+dump.download_dump()
+```
+
+### Parsing the dump file
+```python
+from pprint import pprint
+from wiktionary_de_parser import WiktionaryParser
+
+# ... (see above)
+
+parser = WiktionaryParser()
+
+for page in dump.pages():
+    # Skip redirects
+    if page.redirect_to:
+        continue
+
+    if page.name == "Abend":
+        # Parse all entries for "Abend"
+        for entry in parser.entries_from_page(page):
+            results = parser.parse_entry(entry)
+            pprint(results)
+        break
+```
+
+## Output
+All page entries for "Abend":
+
+```python
+ParsedWiktionaryPageEntry(
+    name="Abend",
+    hyphenation=["Abend"],
+    flexion={
+        "Genus": "m",
+        "Nominativ Singular": "Abend",
+        "Nominativ Plural": "Abende",
+        "Genitiv Singular": "Abends",
+        "Genitiv Plural": "Abende",
+        "Dativ Singular": "Abend",
+        "Dativ Plural": "Abenden",
+        "Akkusativ Singular": "Abend",
+        "Akkusativ Plural": "Abende",
+    },
+    ipa=["ˈaːbn̩t", "ˈaːbm̩t"],
+    language=Language(lang="Deutsch", lang_code="de"),
+    lemma=Lemma(lemma="Abend", inflected=False),
+    pos={"Substantiv": []},
+    rhymes=["aːbn̩t"],
+)
+ParsedWiktionaryPageEntry(
+    name="Abend",
+    hyphenation=["Abend"],
+    flexion=None,
+    ipa=["ˈaːbn̩t"],
+    language=Language(lang="Deutsch", lang_code="de"),
+    lemma=Lemma(lemma="Abend", inflected=False),
+    pos={"Substantiv": ["Nachname"]},
+    rhymes=["aːbn̩t"],
+)
+ParsedWiktionaryPageEntry(
+    name="Abend",
+    hyphenation=["Abend"],
+    flexion=None,
+    ipa=["ˈaːbn̩t", "ˈaːbm̩t"],
+    language=Language(lang="Deutsch", lang_code="de"),
+    lemma=Lemma(lemma="Abend", inflected=False),
+    pos={"Substantiv": ["Toponym"]},
+    rhymes=["aːbn̩t"],
+)
+
+```
+
+## Development
+This project uses [Poetry](https://python-poetry.org/).
+
+1. Install [Poetry](https://python-poetry.org/).
+2. Clone this repository
+3. Run `poetry install` inside of the project folder to install dependencies.
+4. There is a `notebook.ipynb` to test the parser.
+5. Run `poetry run pytest` to run tests.
+
+## License
+
+[MIT](https://github.com/gambolputty/wiktionary-de-parser/blob/master/LICENSE.md) © Gregor Weichbrodt
+

wiktionary_de_parser-0.12.7/README.md

@@ -0,0 +1,125 @@
+# wiktionary-de-parser
+
+A Python module to extract data from German Wiktionary XML files (for Python 3.11+).
+
+## Features
+
+- Extracts _IPA transcriptions_, _hyphenation_, _language_, _part of speech_ information (basic), _genus_ and _flexion tables_ of a word.
+- Yields per entry, not per page (a page can have multiple entries/ words can have different meanings)
+
+## Installation
+
+`pip install wiktionary-de-parser`
+
+Or with [Poetry](https://python-poetry.org/):
+
+`poetry add wiktionary-de-parser`
+
+## Usage
+
+### Loading the XML dump file
+```python
+from wiktionary_de_parser import WiktionaryParser
+from wiktionary_de_parser.dump_processor import WiktionaryDump
+
+# To download the dump file, specify the directory where the
+# dump file should be stored.
+dump = WiktionaryDump(dump_dir_path="directory-of-dump-file")
+
+# This will download "dewiktionary-latest-pages-articles-multistream.xml.bz2" to
+# the directory specified in `dump_dir_path`.
+dump.download_dump()
+
+# Alternatively you can specify a different dump file to download.
+dump = WiktionaryDump(
+    dump_dir_path="directory-of-dump-file",
+    dump_download_url="url-to-dump-file.xml.bz2",
+)
+dump.download_dump()
+
+# If you already have the dump file locally, specify the path to the file.
+dump = WiktionaryDump(dump_file_path="path-to-dump-file.xml.bz2")
+dump.download_dump()
+```
+
+### Parsing the dump file
+```python
+from pprint import pprint
+from wiktionary_de_parser import WiktionaryParser
+
+# ... (see above)
+
+parser = WiktionaryParser()
+
+for page in dump.pages():
+    # Skip redirects
+    if page.redirect_to:
+        continue
+
+    if page.name == "Abend":
+        # Parse all entries for "Abend"
+        for entry in parser.entries_from_page(page):
+            results = parser.parse_entry(entry)
+            pprint(results)
+        break
+```
+
+## Output
+All page entries for "Abend":
+
+```python
+ParsedWiktionaryPageEntry(
+    name="Abend",
+    hyphenation=["Abend"],
+    flexion={
+        "Genus": "m",
+        "Nominativ Singular": "Abend",
+        "Nominativ Plural": "Abende",
+        "Genitiv Singular": "Abends",
+        "Genitiv Plural": "Abende",
+        "Dativ Singular": "Abend",
+        "Dativ Plural": "Abenden",
+        "Akkusativ Singular": "Abend",
+        "Akkusativ Plural": "Abende",
+    },
+    ipa=["ˈaːbn̩t", "ˈaːbm̩t"],
+    language=Language(lang="Deutsch", lang_code="de"),
+    lemma=Lemma(lemma="Abend", inflected=False),
+    pos={"Substantiv": []},
+    rhymes=["aːbn̩t"],
+)
+ParsedWiktionaryPageEntry(
+    name="Abend",
+    hyphenation=["Abend"],
+    flexion=None,
+    ipa=["ˈaːbn̩t"],
+    language=Language(lang="Deutsch", lang_code="de"),
+    lemma=Lemma(lemma="Abend", inflected=False),
+    pos={"Substantiv": ["Nachname"]},
+    rhymes=["aːbn̩t"],
+)
+ParsedWiktionaryPageEntry(
+    name="Abend",
+    hyphenation=["Abend"],
+    flexion=None,
+    ipa=["ˈaːbn̩t", "ˈaːbm̩t"],
+    language=Language(lang="Deutsch", lang_code="de"),
+    lemma=Lemma(lemma="Abend", inflected=False),
+    pos={"Substantiv": ["Toponym"]},
+    rhymes=["aːbn̩t"],
+)
+
+```
+
+## Development
+This project uses [Poetry](https://python-poetry.org/).
+
+1. Install [Poetry](https://python-poetry.org/).
+2. Clone this repository
+3. Run `poetry install` inside of the project folder to install dependencies.
+4. There is a `notebook.ipynb` to test the parser.
+5. Run `poetry run pytest` to run tests.
+
+## License
+
+[MIT](https://github.com/gambolputty/wiktionary-de-parser/blob/master/LICENSE.md) © Gregor Weichbrodt

wiktionary_de_parser-0.12.7/pyproject.toml

@@ -0,0 +1,46 @@
+[tool.poetry]
+name = "wiktionary-de-parser"
+version = "0.12.7"
+description = "Extracts data from German Wiktionary dump files."
+authors = ["Gregor Weichbrodt <gregorweichbrodt@gmail.com>"]
+license = "MIT"
+readme = "README.md"
+homepage = "https://github.com/gambolputty/wiktionary-de-parser"
+repository = "https://github.com/gambolputty/wiktionary-de-parser"
+keywords = ["wiktionary", "xml", "parser", "data-extraction", "german", "nlp"]
+classifiers = [
+    "Programming Language :: Python :: 3.7",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Natural Language :: German",
+    "Topic :: Text Processing :: Markup :: XML"
+]
+include = [
+    "CHANGELOG.md",
+    "LICENSE.txt",
+]
+
+[tool.poetry.urls]
+"Bug Tracker" = "https://github.com/gambolputty/wiktionary-de-parser/issues"
+
+[tool.poetry.dependencies]
+python = "^3.11"
+lxml = "^5.1.0"
+mwparserfromhell = "^0.6.6"
+black = "^24.1.1"
+pydantic = "^2.6.0"
+requests = "^2.31.0"
+tqdm = "^4.66.1"
+wikitextparser = "^0.56.2"
+
+[tool.poetry.dev-dependencies]
+pytest = "^7.1.2"
+black = "*"
+
+[tool.poetry.group.dev.dependencies]
+ipykernel = "^6.29.0"
+ipywidgets = "^8.1.5"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"

wiktionary_de_parser-0.12.7/wiktionary_de_parser/__init__.py

@@ -0,0 +1,98 @@
+import importlib.util
+import inspect
+import re
+from pathlib import Path
+from typing import Type
+
+from wiktionary_de_parser.models import (
+    ParsedWiktionaryPageEntry,
+    WiktionaryPage,
+    WiktionaryPageEntry,
+)
+from wiktionary_de_parser.parser import Parser
+
+
+class WiktionaryParser:
+    parser_classes: list[Type[Parser]]
+
+    def __init__(self):
+        self.parser_classes = self.find_parser_classes()
+
+    @staticmethod
+    def find_parser_classes():
+        path = Path(__file__).parent / "parser"
+        parent_class = Parser
+        classes: list[Type[Parser]] = []
+
+        for child in path.iterdir():
+            if (
+                child.is_file()
+                and child.name.endswith(".py")
+                and child.name != "__init__.py"
+            ):
+                module_name = child.stem  # Entfernen Sie die .py-Endung
+                spec = importlib.util.spec_from_file_location(
+                    module_name, child
+                )
+
+                if not spec or not spec.loader:
+                    raise Exception(f"Could not load {child}")
+
+                module = importlib.util.module_from_spec(spec)
+                spec.loader.exec_module(module)
+
+                for name, obj in inspect.getmembers(module):
+                    if (
+                        inspect.isclass(obj)
+                        and issubclass(obj, parent_class)
+                        and (obj != parent_class)
+                    ):
+                        classes.append(obj)
+
+        return classes
+
+    def entries_from_page(self, page: WiktionaryPage):
+        """
+        Split page into entries. One page can have multiple word entries, for example:
+            - https://de.wiktionary.org/wiki/instrument
+
+        New entries begin at "==" and "===" (sometimes there is no "==")
+        Compare:
+            - https://de.wiktionary.org/wiki/instrument
+            - https://de.wiktionary.org/wiki/Becken
+        """
+        if not page.wikitext:
+            return
+
+        entries: list[str] = re.findall(
+            r"(=== {{Wortart(?:[\w\W](?!^===? ))+)", page.wikitext, re.MULTILINE
+        )
+
+        for index, entry in enumerate(entries):
+            yield WiktionaryPageEntry(
+                page=page,
+                index=index,
+                wikitext=entry,
+            )
+
+    def parse_entry(
+        self,
+        wiktionary_entry: WiktionaryPageEntry,
+        include_meanings: bool = False,
+    ):
+        """
+        Parses an entry of a page.
+        """
+
+        # Instantiate all subclasses and run them
+        results = {
+            instance.name: instance.run()
+            for subclass in self.parser_classes
+            if (instance := subclass(wiktionary_entry))
+            and (include_meanings or instance.name != "meanings")
+        }
+
+        # Add the page name
+        results["name"] = wiktionary_entry.page.name
+
+        return ParsedWiktionaryPageEntry(**results)