jp-stopword-filter 0.1.0__tar.gz

jp_stopword_filter-0.1.0/LICENSE

@@ -0,0 +1,23 @@
+
+MIT License
+
+Copyright (c) 2024 Xu Liang
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

jp_stopword_filter-0.1.0/PKG-INFO

@@ -0,0 +1,112 @@
+Metadata-Version: 2.1
+Name: jp-stopword-filter
+Version: 0.1.0
+Summary: A stopword filter for Japanese
+Author-Email: BrambleXu <liangxu006@gmail.com>
+License: MIT
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.26.4
+Description-Content-Type: text/markdown
+
+# JaStopwordFilter
+
+`JaStopwordFilter` is a lightweight Python library designed to filter stopwords from Japanese text based on customizable rules. It provides an efficient way to preprocess Japanese text for natural language processing (NLP) tasks, with support for common stopword removal techniques and user-defined customization.
+
+
+## Features
+
+- **Preloaded Stopwords**: Includes a comprehensive list of Japanese stopwords from SlothLib.
+- **Customizable Rules**:
+  - Remove tokens based on **length**.
+  - Filter **dates** in common Japanese formats (e.g., `2024年11月`).
+  - Exclude **numbers**, **symbols**, **spaces**, and **emojis**.
+- **Custom Wordlist**: Add your own stopwords to the filter.
+- **Flexible Usage**: Use only the rules you need by enabling or disabling them during initialization.
+
+
+## Installation
+
+Clone the repository and install the dependencies:
+
+```bash
+git clone https://github.com/your-username/ja-stopword-filter.git
+cd ja-stopword-filter
+pip install -r requirements.txt
+```
+
+
+## Usage
+
+### Example Code
+
+```python
+from ja_stopword_filter import JaStopwordFilter
+
+# Example token list
+tokens = ["2024年11月", "こんにちは", "123", "！", "😊", "スペース", "短い", "custom"]
+
+# Custom wordlist
+custom_wordlist = ["custom", "スペース"]
+
+# Initialize the filter
+filter = JaStopwordFilter(
+    use_slothlib=True,      # Use SlothLib stopwords
+    use_length=True,        # Filter tokens with length <= 1
+    use_date=True,          # Filter Japanese date formats
+    use_numbers=True,       # Filter numeric tokens
+    use_symbols=True,       # Filter symbolic tokens
+    use_spaces=True,        # Filter whitespace-only tokens
+    use_emojis=True,        # Filter emoji tokens
+    custom_wordlist=custom_wordlist  # Add custom stopwords
+)
+
+# Filter tokens
+filtered_tokens = filter.remove(tokens)
+print(filtered_tokens)  # Output: ['こんにちは', '短い']
+```
+
+
+## Parameters
+
+The `JaStopwordFilter` class supports the following parameters during initialization:
+
+
+| Parameter         | Type   | Default | Description                                             |
+|--- |--- |--- |---|
+| `use_slothlib`    | `bool` | `True`  | Whether to use the SlothLib stopword list.              |
+| `use_length`      | `bool` | `False` | Remove tokens with a length of 1 character or less.     |
+| `use_date`        | `bool` | `False` | Remove tokens that match Japanese date formats.         |
+| `use_numbers`     | `bool` | `False` | Remove numeric tokens.                                  |
+| `use_symbols`     | `bool` | `False` | Remove symbolic tokens (e.g., `!`, `@`).                |
+| `use_spaces`      | `bool` | `False` | Remove tokens that are empty or consist only of spaces. |
+| `use_emojis`      | `bool` | `False` | Remove tokens containing emojis.                        |
+| `custom_wordlist` | `list` | `None`  | A list of user-defined stopwords to remove.             |
+
+
+## Stopword Sources
+
+### SlothLib Stopwords
+If `use_slothlib` is set to `True`, the filter loads stopwords from a `slothlib.txt` file. Ensure this file is in the same directory as the script or adjust the file path in the `get_stopwords` function.
+
+### Custom Wordlist
+You can pass a list of custom stopwords using the `custom_wordlist` parameter. These will be merged with the SlothLib stopwords if enabled.
+
+
+## Rules
+
+The filter applies the following rules if they are enabled:
+
+1. **Length Filtering**: Tokens with one or fewer characters are removed.
+2. **Date Filtering**: Matches Japanese date patterns like:
+   - `YYYY年MM月`
+   - `MM月DD日`
+   - `YYYY年MM月DD日`
+3. **Number Filtering**: Removes numeric tokens (`123`, `2024`).
+4. **Symbol Filtering**: Removes punctuation and special symbols.
+5. **Space Filtering**: Removes tokens that are empty or consist only of spaces.
+6. **Emoji Filtering**: Detects and removes tokens containing emojis.
+
+
+## Contributing
+
+Contributions are welcome! If you find a bug or have a feature request, feel free to open an issue or submit a pull request.

jp_stopword_filter-0.1.0/README.md

@@ -0,0 +1,102 @@
+# JaStopwordFilter
+
+`JaStopwordFilter` is a lightweight Python library designed to filter stopwords from Japanese text based on customizable rules. It provides an efficient way to preprocess Japanese text for natural language processing (NLP) tasks, with support for common stopword removal techniques and user-defined customization.
+
+
+## Features
+
+- **Preloaded Stopwords**: Includes a comprehensive list of Japanese stopwords from SlothLib.
+- **Customizable Rules**:
+  - Remove tokens based on **length**.
+  - Filter **dates** in common Japanese formats (e.g., `2024年11月`).
+  - Exclude **numbers**, **symbols**, **spaces**, and **emojis**.
+- **Custom Wordlist**: Add your own stopwords to the filter.
+- **Flexible Usage**: Use only the rules you need by enabling or disabling them during initialization.
+
+
+## Installation
+
+Clone the repository and install the dependencies:
+
+```bash
+git clone https://github.com/your-username/ja-stopword-filter.git
+cd ja-stopword-filter
+pip install -r requirements.txt
+```
+
+
+## Usage
+
+### Example Code
+
+```python
+from ja_stopword_filter import JaStopwordFilter
+
+# Example token list
+tokens = ["2024年11月", "こんにちは", "123", "！", "😊", "スペース", "短い", "custom"]
+
+# Custom wordlist
+custom_wordlist = ["custom", "スペース"]
+
+# Initialize the filter
+filter = JaStopwordFilter(
+    use_slothlib=True,      # Use SlothLib stopwords
+    use_length=True,        # Filter tokens with length <= 1
+    use_date=True,          # Filter Japanese date formats
+    use_numbers=True,       # Filter numeric tokens
+    use_symbols=True,       # Filter symbolic tokens
+    use_spaces=True,        # Filter whitespace-only tokens
+    use_emojis=True,        # Filter emoji tokens
+    custom_wordlist=custom_wordlist  # Add custom stopwords
+)
+
+# Filter tokens
+filtered_tokens = filter.remove(tokens)
+print(filtered_tokens)  # Output: ['こんにちは', '短い']
+```
+
+
+## Parameters
+
+The `JaStopwordFilter` class supports the following parameters during initialization:
+
+
+| Parameter         | Type   | Default | Description                                             |
+|--- |--- |--- |---|
+| `use_slothlib`    | `bool` | `True`  | Whether to use the SlothLib stopword list.              |
+| `use_length`      | `bool` | `False` | Remove tokens with a length of 1 character or less.     |
+| `use_date`        | `bool` | `False` | Remove tokens that match Japanese date formats.         |
+| `use_numbers`     | `bool` | `False` | Remove numeric tokens.                                  |
+| `use_symbols`     | `bool` | `False` | Remove symbolic tokens (e.g., `!`, `@`).                |
+| `use_spaces`      | `bool` | `False` | Remove tokens that are empty or consist only of spaces. |
+| `use_emojis`      | `bool` | `False` | Remove tokens containing emojis.                        |
+| `custom_wordlist` | `list` | `None`  | A list of user-defined stopwords to remove.             |
+
+
+## Stopword Sources
+
+### SlothLib Stopwords
+If `use_slothlib` is set to `True`, the filter loads stopwords from a `slothlib.txt` file. Ensure this file is in the same directory as the script or adjust the file path in the `get_stopwords` function.
+
+### Custom Wordlist
+You can pass a list of custom stopwords using the `custom_wordlist` parameter. These will be merged with the SlothLib stopwords if enabled.
+
+
+## Rules
+
+The filter applies the following rules if they are enabled:
+
+1. **Length Filtering**: Tokens with one or fewer characters are removed.
+2. **Date Filtering**: Matches Japanese date patterns like:
+   - `YYYY年MM月`
+   - `MM月DD日`
+   - `YYYY年MM月DD日`
+3. **Number Filtering**: Removes numeric tokens (`123`, `2024`).
+4. **Symbol Filtering**: Removes punctuation and special symbols.
+5. **Space Filtering**: Removes tokens that are empty or consist only of spaces.
+6. **Emoji Filtering**: Detects and removes tokens containing emojis.
+
+
+## Contributing
+
+Contributions are welcome! If you find a bug or have a feature request, feel free to open an issue or submit a pull request.

jp_stopword_filter-0.1.0/pyproject.toml

@@ -0,0 +1,33 @@
+[project]
+name = "jp-stopword-filter"
+version = "0.1.0"
+description = "A stopword filter for Japanese"
+authors = [
+    { name = "BrambleXu", email = "liangxu006@gmail.com" },
+]
+dependencies = [
+    "numpy>=1.26.4",
+]
+requires-python = ">=3.9"
+readme = "README.md"
+
+[project.license]
+text = "MIT"
+
+[build-system]
+requires = [
+    "pdm-backend",
+]
+build-backend = "pdm.backend"
+
+[tool.pdm]
+distribution = true
+
+[tool.pdm.dev-dependencies]
+dev = [
+    "ruff>=0.4.4",
+    "pytest>=8.2.0",
+    "mypy>=1.10.0",
+    "pre-commit>=3.7.1",
+    "codespell>=2.2.6",
+]

jp_stopword_filter-0.1.0/src/JaStopwordFilter.py

@@ -0,0 +1,182 @@
+import re
+import unicodedata
+from typing import List, Optional, Set
+
+
+def get_stopwords() -> List[str]:
+    """
+    Reads the SlothLib stopwords from a file and returns them as a list.
+
+    Returns:
+        List[str]: A list of Japanese stopwords from SlothLib.
+    """
+    file_path = "./src/slothlib.txt"
+
+    # Reading the file content and converting it into a list
+    with open(file_path, "r", encoding="utf-8") as file:
+        content = file.read()
+
+    # Splitting the content by lines to get a list of stop words
+    japanese_stop_words_list = content.splitlines()
+
+    return japanese_stop_words_list
+
+
+def convert_to_halfwidth(text: str) -> str:
+    """
+    Converts a string from full-width to half-width characters.
+
+    Args:
+        text (str): The input string.
+
+    Returns:
+        str: The converted string with half-width characters.
+    """
+    return unicodedata.normalize("NFKC", text)
+
+
+class JaStopwordFilter:
+    """
+    A filter class to remove Japanese stopwords and other undesired tokens
+    based on customizable rules.
+    """
+
+    def __init__(
+        self,
+        convert_full_to_half: bool = True,
+        use_slothlib: bool = True,
+        filter_length: int = 0,
+        use_date: bool = False,
+        use_numbers: bool = False,
+        use_symbols: bool = False,
+        use_spaces: bool = False,
+        use_emojis: bool = False,
+        custom_wordlist: Optional[List[str]] = None,
+    ) -> None:
+        """
+        Initializes the JaStopwordFilter with the specified filtering rules.
+
+        Args:
+            convert_full_to_half (bool): Whether to convert full-width characters to half-width. Defaults to True.
+            use_slothlib (bool): Whether to use the SlothLib stopword list. Defaults to True.
+            filter_length (int): Remove tokens with a length less than or equal to this value.
+                                 Defaults to 0 (no filtering).
+            use_date (bool): Whether to remove tokens that match Japanese date patterns. Defaults to False.
+            use_numbers (bool): Whether to remove numeric tokens. Defaults to False.
+            use_symbols (bool): Whether to remove tokens consisting of symbols. Defaults to False.
+            use_spaces (bool): Whether to remove tokens that are empty or contain only spaces. Defaults to False.
+            use_emojis (bool): Whether to remove tokens containing emojis. Defaults to False.
+            custom_wordlist (Optional[List[str]]): A list of user-defined stopwords to remove. Defaults to None.
+        """
+        self.stopwords: Set[str] = set()
+        self.filter_length = filter_length
+        self.use_date = use_date
+        self.use_numbers = use_numbers
+        self.use_symbols = use_symbols
+        self.use_spaces = use_spaces
+        self.use_emojis = use_emojis
+        self.convert_full_to_half = convert_full_to_half
+
+        # Load SlothLib stopwords
+        if use_slothlib:
+            self.stopwords.update(
+                convert_to_halfwidth(word) if self.convert_full_to_half else word for word in get_stopwords()
+            )
+
+        # Add custom wordlist to stopwords
+        if custom_wordlist:
+            self.stopwords.update(
+                convert_to_halfwidth(word) if self.convert_full_to_half else word for word in custom_wordlist
+            )
+
+    def remove(self, tokens: List[str]) -> List[str]:
+        """
+        Removes tokens based on the filtering rules.
+
+        Args:
+            tokens (List[str]): A list of input tokens to filter.
+
+        Returns:
+            List[str]: A list of filtered tokens.
+        """
+        # Convert tokens to half-width if the option is enabled
+        if self.convert_full_to_half:
+            tokens = [convert_to_halfwidth(token) for token in tokens]
+
+        filtered_tokens: List[str] = []
+        for token in tokens:
+            if token in self.stopwords:
+                continue
+            if self.filter_length > 0 and len(token) <= self.filter_length:
+                continue
+            if self.use_date and self._is_date(token):
+                continue
+            if self.use_numbers and self._is_number(token):
+                continue
+            if self.use_symbols and self._is_symbol(token):
+                continue
+            if self.use_spaces and token.strip() == "":
+                continue
+            if self.use_emojis and self._is_emoji(token):
+                continue
+            filtered_tokens.append(token)
+        return filtered_tokens
+
+    def _is_date(self, token: str) -> bool:
+        """
+        Checks if a token matches common Japanese date patterns.
+
+        Args:
+            token (str): The token to check.
+
+        Returns:
+            bool: True if the token matches a date pattern, otherwise False.
+        """
+        date_patterns = [
+            r"\d{4}年\d{1,2}月",  # YYYY年MM月
+            r"\d{1,2}月\d{1,2}日",  # MM月DD日
+            r"\d{4}年\d{1,2}月\d{1,2}日",  # YYYY年MM月DD日
+        ]
+        return any(re.match(pattern, token) for pattern in date_patterns)
+
+    def _is_number(self, token: str) -> bool:
+        """
+        Checks if a token is numeric.
+
+        Args:
+            token (str): The token to check.
+
+        Returns:
+            bool: True if the token is numeric, otherwise False.
+        """
+        return token.isdigit()
+
+    def _is_symbol(self, token: str) -> bool:
+        """
+        Checks if a token is a symbol.
+
+        Args:
+            token (str): The token to check.
+
+        Returns:
+            bool: True if the token is a symbol, otherwise False.
+        """
+        return re.fullmatch(r"[!-/:-@[-`{-~]", token) is not None
+
+    def _is_emoji(self, token: str) -> bool:
+        """
+        Checks if a token contains emojis.
+
+        Args:
+            token (str): The token to check.
+
+        Returns:
+            bool: True if the token contains emojis, otherwise False.
+        """
+        return any(
+            "\U0001f600" <= char <= "\U0001f64f"  # Emoticons
+            or "\U0001f300" <= char <= "\U0001f5ff"  # Symbols & Pictographs
+            or "\U0001f680" <= char <= "\U0001f6ff"  # Transport & Map Symbols
+            or "\U0001f1e0" <= char <= "\U0001f1ff"  # Flags
+            for char in token
+        )

jp_stopword_filter-0.1.0/tests/test_JaStopwordFilter.py

@@ -0,0 +1,177 @@
+import pytest
+
+from src.JaStopwordFilter import JaStopwordFilter, convert_to_halfwidth, get_stopwords
+
+
+@pytest.fixture
+def tokens():
+    """
+    A sample token list for testing.
+    """
+    return [
+        "２０２４年１１月",  # Full-width date
+        "2024年11月",  # Half-width date
+        "１２３",  # Full-width number
+        "123",  # Half-width number
+        "！",  # Full-width symbol
+        "!",  # Half-width symbol
+        "😊",  # Emoji
+        "短",  # Short token
+        "長い単語",  # Long token
+        "custom",  # Custom word
+    ]
+
+
+def test_get_stopwords():
+    """
+    Test if get_stopwords correctly loads stopwords from a file.
+    """
+    stopwords = get_stopwords()
+    assert isinstance(stopwords, list), "Stopwords should be returned as a list."
+    assert len(stopwords) > 0, "Stopwords list should not be empty."
+
+
+def test_convert_to_halfwidth():
+    """
+    Test if convert_to_halfwidth correctly converts full-width characters to half-width.
+    """
+    assert convert_to_halfwidth("１２３") == "123", "Full-width numbers should convert to half-width."
+    assert convert_to_halfwidth("ＡＢＣ") == "ABC", "Full-width letters should convert to half-width."
+    assert convert_to_halfwidth("！＠＃") == "!@#", "Full-width symbols should convert to half-width."
+
+
+def test_init_with_full_to_half(tokens):
+    """
+    Test if the JaStopwordFilter correctly handles full-to-half conversion during initialization.
+    """
+    filter = JaStopwordFilter(convert_full_to_half=True)
+    assert filter.convert_full_to_half, "convert_full_to_half should be True when enabled."
+
+
+def test_remove_with_full_to_half(tokens):
+    """
+    Test if tokens are correctly filtered after converting full-width to half-width.
+    """
+    custom_wordlist = ["123", "custom"]
+    filter = JaStopwordFilter(convert_full_to_half=True, custom_wordlist=custom_wordlist)
+    filtered = filter.remove(tokens)
+    assert "１２３" not in filtered, "Full-width '１２３' should be converted and removed."
+    assert "custom" not in filtered, "'custom' should be removed."
+    assert "２０２４年１１月" not in filtered, "Full-width date should be converted and removed."
+
+
+def test_remove_without_full_to_half(tokens):
+    """
+    Test if tokens are filtered without converting full-width to half-width.
+    """
+    custom_wordlist = ["１２３", "custom"]
+    filter = JaStopwordFilter(convert_full_to_half=False, custom_wordlist=custom_wordlist)
+    filtered = filter.remove(tokens)
+    assert "１２３" not in filtered, "Full-width '１２３' should be removed."
+    assert "123" in filtered, "Half-width '123' should not be removed."
+    assert "custom" not in filtered, "'custom' should be removed."
+
+
+def test_filter_length(tokens):
+    """
+    Test if tokens with length less than or equal to filter_length are correctly removed.
+    """
+    filter = JaStopwordFilter(filter_length=2, convert_full_to_half=True)
+    filtered = filter.remove(tokens)
+    assert "短" not in filtered, "'短' should be removed because its length is <= 2."
+    assert "長い単語" in filtered, "'長い単語' should remain because its length is > 2."
+
+
+def test_remove_with_date(tokens):
+    """
+    Test if tokens matching date patterns are correctly removed.
+    """
+    filter = JaStopwordFilter(use_date=True, convert_full_to_half=True)
+    filtered = filter.remove(tokens)
+    assert "２０２４年１１月" not in filtered, "Full-width date should be converted and removed."
+    assert "2024年11月" not in filtered, "Half-width date should be removed."
+
+
+def test_remove_with_numbers(tokens):
+    """
+    Test if numeric tokens are correctly removed.
+    """
+    filter = JaStopwordFilter(use_numbers=True, convert_full_to_half=True)
+    filtered = filter.remove(tokens)
+    assert "１２３" not in filtered, "Full-width number should be converted and removed."
+    assert "123" not in filtered, "Half-width number should be removed."
+
+
+def test_remove_with_symbols(tokens):
+    """
+    Test if symbolic tokens are correctly removed.
+    """
+    filter = JaStopwordFilter(use_symbols=True, convert_full_to_half=True)
+    filtered = filter.remove(tokens)
+    assert "！" not in filtered, "Full-width symbol should be converted and removed."
+    assert "!" not in filtered, "Half-width symbol should be removed."
+
+
+def test_remove_with_spaces():
+    """
+    Test if empty or whitespace-only tokens are correctly removed.
+    """
+    tokens_with_spaces = [" ", "　", "custom"]
+    filter = JaStopwordFilter(use_spaces=True, convert_full_to_half=True)
+    filtered = filter.remove(tokens_with_spaces)
+    assert " " not in filtered, "Half-width space should be removed."
+    assert "　" not in filtered, "Full-width space should be removed."
+    assert "custom" in filtered, "'custom' should remain."
+
+
+def test_remove_with_emojis(tokens):
+    """
+    Test if tokens containing emojis are correctly removed.
+    """
+    filter = JaStopwordFilter(use_emojis=True, convert_full_to_half=True)
+    filtered = filter.remove(tokens)
+    assert "😊" not in filtered, "Emoji '😊' should be removed."
+
+
+def test_combined_rules(tokens):
+    """
+    Test if the filter works correctly with combined rules and full-to-half conversion.
+    """
+    custom_wordlist = ["123", "custom"]
+    filter = JaStopwordFilter(
+        filter_length=2,
+        use_date=True,
+        use_numbers=True,
+        use_symbols=True,
+        use_emojis=True,
+        convert_full_to_half=True,
+        custom_wordlist=custom_wordlist,
+    )
+    filtered = filter.remove(tokens)
+    assert "２０２４年１１月" not in filtered, "Full-width date should be converted and removed."
+    assert "123" not in filtered, "Half-width number should be removed."
+    assert "１２３" not in filtered, "Full-width number should be converted and removed."
+    assert "！" not in filtered, "Full-width symbol should be converted and removed."
+    assert "😊" not in filtered, "Emoji '😊' should be removed."
+    assert "短" not in filtered, "Tokens with length <= 2 should be removed."
+    assert "長い単語" in filtered, "'長い単語' should remain in the filtered list."
+
+
+def test_no_rules(tokens):
+    """
+    Test if all tokens are retained when no rules are enabled.
+    """
+    filter = JaStopwordFilter(convert_full_to_half=False, use_slothlib=False)
+    filtered = filter.remove(tokens)
+    assert filtered == [
+        "２０２４年１１月",
+        "2024年11月",
+        "１２３",
+        "123",
+        "！",
+        "!",
+        "😊",
+        "短",
+        "長い単語",
+        "custom",
+    ], "All tokens should remain when no rules are enabled."