sonatoki 0.2.2__tar.gz → 0.3.1__tar.gz

{sonatoki-0.2.2 → sonatoki-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: sonatoki
-Version: 0.2.2
+Version: 0.3.1
 Summary: ilo li moku e toki li pana e sona ni: ni li toki ala toki pona?
 Author-Email: "jan Kekan San (@gregdan3)" <gregory.danielson3@gmail.com>
 License: AGPL-3.0-or-later
@@ -12,15 +12,22 @@ Description-Content-Type: text/markdown
 
 # sona toki
 
+<div align="center">
+
+![Test workflow for this library](https://github.com/gregdan3/sona-toki/workflows/Tests/badge.svg)
+[![Version number for this library](https://img.shields.io/pypi/v/sonatoki?logo=python&logoColor=%23cccccc)](https://pypi.org/project/sonatoki)
+
+</div>
+
 ## What is **sona toki**?
 
-This library, "Language Knowledge," helps you identify whether a message is in Toki Pona. No grammar checking, yet, which means this more checks whether a given message has enough Toki Pona words.
+This library, "Language Knowledge," helps you identify whether a message is in Toki Pona. It does so by determining whether a large enough number of words in a statement are "in Toki Pona". No grammar checking, yet.
 
-I wrote it with a variety of scraps and lessons learned from a prior project, [ilo pi toki pona taso, "toki-pona-only tool"](https://github.com/gregdan3/ilo-pi-toki-pona-taso). That tool will be rewritten to use this library shortly.
+I wrote this library with a variety of scraps and lessons learned from a prior project, [ilo pi toki pona taso, "toki-pona-only tool"](https://github.com/gregdan3/ilo-pi-toki-pona-taso). That tool now uses this library to great success!
 
-If you've ever worked on a similar project, you know the question "is this message in [language]" is not a consistent one- the environment, time, preferences of the speaker, and much more, can all alter whether a given message is "in" any specific language, and this question applies to Toki Pona too.
+If you've ever worked on a similar project, you know the question "is this message in [language]" is not a consistent one- the environment, time, preferences of the speaker, and much more, can all alter whether a given message is "in" any specific language. This complexity applies to Toki Pona too.
 
-This project "solves" that complex problem by offering a highly configurable parser, so you can tune it to your preferences and goals.
+So, this project "solves" that complex problem by offering an opinionated tokenizer and a configurable parser, allowing you to tune its output to your preferences and goals. [Even silly ones.](https://sona.pona.la/wiki/isipin_epiku).
 
 ## Quick Start
 
@@ -53,12 +60,12 @@ Or if you'd prefer to configure on your own:
 from copy import deepcopy
 from sonatoki.ilo import Ilo
 from sonatoki.Configs import BaseConfig
-from sonatoki.Filters import NimiPuAle, Phonotactic, ProperName
+from sonatoki.Filters import NimiLinkuCore, Phonotactic, ProperName
 from sonatoki.Scorers import SoftPassFail
 
 def main():
     config = deepcopy(BaseConfig)
-    config["scoring_filters"].extend([NimiPuAle, Phonotactic, ProperName])
+    config["scoring_filters"].extend([NimiLinkuCore, Phonotactic, ProperName])
     config["scorer"] = SoftPassFail
 
     ilo = Ilo(**config)
@@ -88,24 +95,28 @@ After our proposal has been examined and a result given by the committee, I will
 
 ### What's the deal with the tokenizers?
 
-The Toki Pona tokenizer `word_tokenize_tok` is very specific in always separating writing characters from punctuation, and leaving contiguous punctuation as contiguous- this is a level of precision that NLTK's English tokenizer does not want for several reasons, such as that English words can have "punctuation" characters in them.
-
-Toki Pona doesn't have any mid-word symbols when rendered in the Latin alphabet, so a more aggressive tokenizer is highly desirable.
+The Toki Pona tokenizer `sonatoki.Tokenizers.WordTokenizer` has the goal of tokenizing statements such that every token either represents a word candidate ("toki", "mumumu") or a complete non-candidate ("..!", "123").
+This design is highly undesirable for NLTK's English tokenizer because English words can have "punctuation" characters in them.
+But Toki Pona doesn't have any mid-word symbols when rendered in the Latin alphabet or in [Private Use Area Unicode characters](https://www.kreativekorp.com/ucsur/), so a more aggressive tokenizer is highly desirable.
 
-The other tokenizers are provided as a comparison case more than anything. I do not recommend their use.
+The goal of splitting into word candidates and non-candidates is important, because any [encoding of Toki Pona's logographic script](https://www.kreativekorp.com/ucsur/charts/sitelen.html) will require each character be split into its own token, where the default behavior would be to leave consecutive non-punctuation together.
 
 ### Aren't there a lot of false positives?
 
-Yes. It's up to you to use this tool responsibly on input you've done your best to clean, and better, use stronger filters before weaker ones. For now though, here's a list of relevant false positives:
+Yes, depending on the filter you choose and how you apply it.
+It's up to you to use this tool responsibly on input you've done your best to clean, such as by using stronger filters before weaker ones.
+For now though, here's a list of relevant false positives:
 
-- `ProperName` will errantly match text in languages without a capital/lowercase distinction, artificially inflating the scores.
-- `Alphabetic` will match a _lot_ of undesirable text- it essentially allows 14 letters of the English alphabet.
+- `ProperName` will errantly match text in languages without a capital/lowercase distinction, artificially increasing scores.
+- `Alphabetic` will match a _lot_ of undesirable text- it essentially allows 14 letters of the English alphabet. For example, "I'm well" would match as _three_ words: "i", "m", "well".
+- `NimiPu` and other sets containing `a`, `mute`, `open`, and others will unavoidably match those words in English text too.
 
 ### Don't some of the cleaners/filters conflict?
 
-Yes. Some do so
+Yes, though not terribly much.
 
 - `ConsecutiveDuplicates` may errantly change a word's validity. For example, "manna" is phonotactically invalid in Toki Pona, but would become "mana" which is valid.
-- `ConsecutiveDuplicates` will not work correctly with syllabaries (alphabets, but representing a pair of consonant and vowel).
+- `ConsecutiveDuplicates` will not work correctly with syllabaries, though this should not change the validity of the analyzed word unless you attempt to dictionary match these words.
+- If you build your own `MemberFilter` with words that have capital letters or consecutive duplicates, they will never match unless you use `prep_dictionary`.
 
-You'll notice a _lot_ of these are troubles regarding the application of latin alphabet filters to non-latin text. Working on it!
+You'll notice these are mostly casued by applying latin alphabet filters to non-latin text. Working on it!

{sonatoki-0.2.2 → sonatoki-0.3.1}/README.md

@@ -1,14 +1,21 @@
 # sona toki
 
+<div align="center">
+
+![Test workflow for this library](https://github.com/gregdan3/sona-toki/workflows/Tests/badge.svg)
+[![Version number for this library](https://img.shields.io/pypi/v/sonatoki?logo=python&logoColor=%23cccccc)](https://pypi.org/project/sonatoki)
+
+</div>
+
 ## What is **sona toki**?
 
-This library, "Language Knowledge," helps you identify whether a message is in Toki Pona. No grammar checking, yet, which means this more checks whether a given message has enough Toki Pona words.
+This library, "Language Knowledge," helps you identify whether a message is in Toki Pona. It does so by determining whether a large enough number of words in a statement are "in Toki Pona". No grammar checking, yet.
 
-I wrote it with a variety of scraps and lessons learned from a prior project, [ilo pi toki pona taso, "toki-pona-only tool"](https://github.com/gregdan3/ilo-pi-toki-pona-taso). That tool will be rewritten to use this library shortly.
+I wrote this library with a variety of scraps and lessons learned from a prior project, [ilo pi toki pona taso, "toki-pona-only tool"](https://github.com/gregdan3/ilo-pi-toki-pona-taso). That tool now uses this library to great success!
 
-If you've ever worked on a similar project, you know the question "is this message in [language]" is not a consistent one- the environment, time, preferences of the speaker, and much more, can all alter whether a given message is "in" any specific language, and this question applies to Toki Pona too.
+If you've ever worked on a similar project, you know the question "is this message in [language]" is not a consistent one- the environment, time, preferences of the speaker, and much more, can all alter whether a given message is "in" any specific language. This complexity applies to Toki Pona too.
 
-This project "solves" that complex problem by offering a highly configurable parser, so you can tune it to your preferences and goals.
+So, this project "solves" that complex problem by offering an opinionated tokenizer and a configurable parser, allowing you to tune its output to your preferences and goals. [Even silly ones.](https://sona.pona.la/wiki/isipin_epiku).
 
 ## Quick Start
 
@@ -41,12 +48,12 @@ Or if you'd prefer to configure on your own:
 from copy import deepcopy
 from sonatoki.ilo import Ilo
 from sonatoki.Configs import BaseConfig
-from sonatoki.Filters import NimiPuAle, Phonotactic, ProperName
+from sonatoki.Filters import NimiLinkuCore, Phonotactic, ProperName
 from sonatoki.Scorers import SoftPassFail
 
 def main():
     config = deepcopy(BaseConfig)
-    config["scoring_filters"].extend([NimiPuAle, Phonotactic, ProperName])
+    config["scoring_filters"].extend([NimiLinkuCore, Phonotactic, ProperName])
     config["scorer"] = SoftPassFail
 
     ilo = Ilo(**config)
@@ -76,24 +83,28 @@ After our proposal has been examined and a result given by the committee, I will
 
 ### What's the deal with the tokenizers?
 
-The Toki Pona tokenizer `word_tokenize_tok` is very specific in always separating writing characters from punctuation, and leaving contiguous punctuation as contiguous- this is a level of precision that NLTK's English tokenizer does not want for several reasons, such as that English words can have "punctuation" characters in them.
-
-Toki Pona doesn't have any mid-word symbols when rendered in the Latin alphabet, so a more aggressive tokenizer is highly desirable.
+The Toki Pona tokenizer `sonatoki.Tokenizers.WordTokenizer` has the goal of tokenizing statements such that every token either represents a word candidate ("toki", "mumumu") or a complete non-candidate ("..!", "123").
+This design is highly undesirable for NLTK's English tokenizer because English words can have "punctuation" characters in them.
+But Toki Pona doesn't have any mid-word symbols when rendered in the Latin alphabet or in [Private Use Area Unicode characters](https://www.kreativekorp.com/ucsur/), so a more aggressive tokenizer is highly desirable.
 
-The other tokenizers are provided as a comparison case more than anything. I do not recommend their use.
+The goal of splitting into word candidates and non-candidates is important, because any [encoding of Toki Pona's logographic script](https://www.kreativekorp.com/ucsur/charts/sitelen.html) will require each character be split into its own token, where the default behavior would be to leave consecutive non-punctuation together.
 
 ### Aren't there a lot of false positives?
 
-Yes. It's up to you to use this tool responsibly on input you've done your best to clean, and better, use stronger filters before weaker ones. For now though, here's a list of relevant false positives:
+Yes, depending on the filter you choose and how you apply it.
+It's up to you to use this tool responsibly on input you've done your best to clean, such as by using stronger filters before weaker ones.
+For now though, here's a list of relevant false positives:
 
-- `ProperName` will errantly match text in languages without a capital/lowercase distinction, artificially inflating the scores.
-- `Alphabetic` will match a _lot_ of undesirable text- it essentially allows 14 letters of the English alphabet.
+- `ProperName` will errantly match text in languages without a capital/lowercase distinction, artificially increasing scores.
+- `Alphabetic` will match a _lot_ of undesirable text- it essentially allows 14 letters of the English alphabet. For example, "I'm well" would match as _three_ words: "i", "m", "well".
+- `NimiPu` and other sets containing `a`, `mute`, `open`, and others will unavoidably match those words in English text too.
 
 ### Don't some of the cleaners/filters conflict?
 
-Yes. Some do so
+Yes, though not terribly much.
 
 - `ConsecutiveDuplicates` may errantly change a word's validity. For example, "manna" is phonotactically invalid in Toki Pona, but would become "mana" which is valid.
-- `ConsecutiveDuplicates` will not work correctly with syllabaries (alphabets, but representing a pair of consonant and vowel).
+- `ConsecutiveDuplicates` will not work correctly with syllabaries, though this should not change the validity of the analyzed word unless you attempt to dictionary match these words.
+- If you build your own `MemberFilter` with words that have capital letters or consecutive duplicates, they will never match unless you use `prep_dictionary`.
 
-You'll notice a _lot_ of these are troubles regarding the application of latin alphabet filters to non-latin text. Working on it!
+You'll notice these are mostly casued by applying latin alphabet filters to non-latin text. Working on it!

{sonatoki-0.2.2 → sonatoki-0.3.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "sonatoki"
-version = "0.2.2"
+version = "0.3.1"
 description = "ilo li moku e toki li pana e sona ni: ni li toki ala toki pona?"
 authors = [
     { name = "jan Kekan San (@gregdan3)", email = "gregory.danielson3@gmail.com" },

{sonatoki-0.2.2 → sonatoki-0.3.1}/src/sonatoki/Cleaners.py

@@ -60,6 +60,13 @@ class ConsecutiveDuplicatesRe(RegexCleaner):
     replace = r"\1"
 
 
+class Lowercase(Cleaner):
+    @classmethod
+    @override
+    def clean(cls, token: str) -> str:
+        return token.lower()
+
+
 __all__ = [
     "ConsecutiveDuplicates",
 ]

sonatoki-0.3.1/src/sonatoki/Configs.py

@@ -0,0 +1,139 @@
+# STL
+from copy import deepcopy
+from typing import List, Type, TypedDict
+
+# LOCAL
+from sonatoki.Filters import (
+    Filter,
+    NimiPu,
+    Numeric,
+    OrFilter,
+    Syllabic,
+    NimiUCSUR,
+    Alphabetic,
+    ProperName,
+    Phonotactic,
+    Punctuation,
+    NimiLinkuCore,
+    NimiPuSynonyms,
+    OrMemberFilter,
+    NimiLinkuCommon,
+    NimiLinkuObscure,
+    NimiLinkuSandbox,
+    EnglishIgnorables,
+    NimiLinkuUncommon,
+)
+from sonatoki.Scorers import Number, Scorer, PassFail, SoftScaling, SoftPassFail
+from sonatoki.Cleaners import Cleaner, ConsecutiveDuplicates
+from sonatoki.Tokenizers import Tokenizer, WordTokenizer
+from sonatoki.Preprocessors import (
+    URLs,
+    Reference,
+    Preprocessor,
+    DiscordEmotes,
+    DiscordSpecial,
+    DiscordChannels,
+    DiscordMentions,
+    AngleBracketObject,
+)
+
+
+class IloConfig(TypedDict):
+    preprocessors: List[Type[Preprocessor]]
+    word_tokenizer: Type[Tokenizer]
+    cleaners: List[Type[Cleaner]]
+    ignoring_filters: List[Type[Filter]]
+    scoring_filters: List[Type[Filter]]
+    scorer: Type[Scorer]
+    passing_score: Number
+
+
+# TODO: branching configs?
+
+BaseConfig: IloConfig = {
+    "preprocessors": [URLs],
+    "cleaners": [ConsecutiveDuplicates],
+    "ignoring_filters": [Numeric, Punctuation],
+    "scoring_filters": [],
+    "scorer": PassFail,
+    "passing_score": 0.8,
+    "word_tokenizer": WordTokenizer,
+}
+
+
+PrefConfig: IloConfig = {
+    "preprocessors": [URLs, Reference],
+    "cleaners": [ConsecutiveDuplicates],
+    "ignoring_filters": [Numeric, Punctuation, EnglishIgnorables],
+    "scoring_filters": [
+        OrMemberFilter(NimiLinkuCore, NimiLinkuCommon, NimiUCSUR),
+        Syllabic,
+        ProperName,
+        Alphabetic,
+    ],
+    "scorer": SoftScaling,
+    "passing_score": 0.8,
+    "word_tokenizer": WordTokenizer,
+}
+
+CorpusConfig: IloConfig = {
+    "preprocessors": [URLs, AngleBracketObject, Reference],
+    "cleaners": [ConsecutiveDuplicates],
+    "ignoring_filters": [Numeric, Punctuation, EnglishIgnorables],
+    "scoring_filters": [
+        OrMemberFilter(
+            NimiLinkuCore,
+            NimiLinkuCommon,
+            NimiLinkuUncommon,
+            NimiLinkuObscure,
+            NimiLinkuSandbox,
+            NimiUCSUR,
+        ),
+        Syllabic,
+        ProperName,
+        Alphabetic,
+    ],
+    "scorer": SoftScaling,
+    "passing_score": 0.8,
+    "word_tokenizer": WordTokenizer,
+}
+
+
+LazyConfig: IloConfig = {
+    "preprocessors": [URLs],
+    "cleaners": [ConsecutiveDuplicates],
+    "ignoring_filters": [Numeric, Punctuation],
+    "scoring_filters": [Alphabetic, NimiUCSUR, ProperName],
+    "scorer": SoftPassFail,
+    "passing_score": 0.8,
+    "word_tokenizer": WordTokenizer,
+}
+
+DiscordConfig: IloConfig = {
+    "preprocessors": [URLs, AngleBracketObject, Reference],
+    "cleaners": [ConsecutiveDuplicates],
+    "ignoring_filters": [Numeric, Punctuation, EnglishIgnorables],
+    "scoring_filters": [
+        OrMemberFilter(NimiLinkuCore, NimiLinkuCommon, NimiUCSUR),
+        Syllabic,
+        ProperName,
+        Alphabetic,
+    ],
+    "scorer": SoftScaling,
+    "passing_score": 0.8,
+    "word_tokenizer": WordTokenizer,
+}
+
+TelegramConfig: IloConfig = deepcopy(PrefConfig)
+ForumConfig: IloConfig = deepcopy(PrefConfig)
+
+__all__ = [
+    "BaseConfig",
+    "CorpusConfig",
+    "DiscordConfig",
+    "ForumConfig",
+    "IloConfig",
+    "LazyConfig",
+    "PrefConfig",
+    "TelegramConfig",
+]

{sonatoki-0.2.2 → sonatoki-0.3.1}/src/sonatoki/Filters.py

@@ -1,7 +1,7 @@
 # STL
 import re
 from abc import ABC, abstractmethod
-from typing import Set
+from typing import Set, List, Type
 from functools import lru_cache as cache  # cache comes in 3.9
 
 # PDM
@@ -9,19 +9,26 @@ import regex
 from typing_extensions import override
 
 # LOCAL
+from sonatoki.utils import prep_dictionary
 from sonatoki.constants import (
     VOWELS,
     NIMI_PU,
     ALPHABET,
+    ALL_PUNCT,
     ALLOWABLES,
     CONSONANTS,
-    NIMI_LINKU,
-    POSIX_PUNCT,
-    UNICODE_PUNCT,
-    NIMI_LINKU_LILI,
+    IGNORABLES,
+    NIMI_UCSUR,
+    NIMI_KU_LILI,
+    NIMI_KU_SULI,
+    NIMI_LINKU_CORE,
     ALL_PUNCT_RANGES,
     NIMI_PU_SYNONYMS,
+    NIMI_LINKU_COMMON,
+    NIMI_LINKU_OBSCURE,
     NIMI_LINKU_SANDBOX,
+    UCSUR_PUNCT_RANGES,
+    NIMI_LINKU_UNCOMMON,
 )
 
 regex.DEFAULT_VERSION = regex.VERSION1
@@ -79,6 +86,10 @@ class Miscellaneous(MemberFilter):
     tokens = set(ALLOWABLES)
 
 
+class EnglishIgnorables(MemberFilter):
+    tokens = set(IGNORABLES)
+
+
 class ProperName(Filter):
     """Determines if a given token is a valid name (also called a loan word).
     When Toki Pona is written with the Latin alphabet, names are generally
@@ -99,23 +110,43 @@ class ProperName(Filter):
 
 
 class NimiPu(MemberFilter):
-    tokens = set(NIMI_PU)
+    tokens = prep_dictionary(NIMI_PU)
+
+
+class NimiPuSynonyms(MemberFilter):
+    tokens = prep_dictionary(NIMI_PU_SYNONYMS)
+
+
+class NimiKuSuli(MemberFilter):
+    tokens = prep_dictionary(NIMI_KU_SULI)
+
+
+class NimiKuLili(MemberFilter):
+    tokens = prep_dictionary(NIMI_KU_LILI)
+
+
+class NimiLinkuCore(MemberFilter):
+    tokens = prep_dictionary(NIMI_LINKU_CORE)
 
 
-class NimiPuAle(MemberFilter):
-    tokens = set(NIMI_PU + NIMI_PU_SYNONYMS)
+class NimiLinkuCommon(MemberFilter):
+    tokens = prep_dictionary(NIMI_LINKU_COMMON)
 
 
-class NimiLinku(MemberFilter):
-    tokens = set(NIMI_LINKU)
+class NimiLinkuUncommon(MemberFilter):
+    tokens = prep_dictionary(NIMI_LINKU_UNCOMMON)
 
 
-class NimiLinkuAle(MemberFilter):
-    tokens = set(NIMI_LINKU + NIMI_LINKU_LILI)
+class NimiLinkuObscure(MemberFilter):
+    tokens = prep_dictionary(NIMI_LINKU_OBSCURE)
 
 
 class NimiLinkuSandbox(MemberFilter):
-    tokens = set(NIMI_LINKU + NIMI_LINKU_LILI + NIMI_LINKU_SANDBOX)
+    tokens = prep_dictionary(NIMI_LINKU_SANDBOX)
+
+
+class NimiUCSUR(MemberFilter):
+    tokens = prep_dictionary(NIMI_UCSUR)
 
 
 class Phonotactic(RegexFilter):
@@ -156,6 +187,11 @@ class AlphabeticRe(RegexFilter):
     pattern = re.compile(rf"[{ALPHABET}]+", flags=re.IGNORECASE)
 
 
+class TwoOrMoreAlphabetic(Filter):
+    # TODO: alphabetic implementation that ignores single characters
+    pass
+
+
 class Numeric(Filter):
     """Determine if a given token is entirely numeric.
     Covers all numeric symbols in Unicode.
@@ -175,12 +211,13 @@ class Numeric(Filter):
 class Punctuation(SubsetFilter):
     """Identify whether a token is entirely punctuation. Fastest implementation."""
 
-    tokens = set(POSIX_PUNCT + UNICODE_PUNCT)
+    tokens = set(ALL_PUNCT)
 
 
 class PunctuationRe(RegexFilter):
     """Faster implementation of `PunctuationRe1`.
-    Goes out of date compared to the `regex` library if UNICODE_PUNCT is not updated."""
+    Goes out of date compared to the `regex` library if UNICODE_PUNCT_RANGES is not updated.
+    """
 
     pattern = re.compile(rf"[{ALL_PUNCT_RANGES}]+")
 
@@ -188,17 +225,111 @@ class PunctuationRe(RegexFilter):
 class PunctuationRe1(Regex1Filter):
     """Reference implementation for identifying tokens made entirely of punctuation."""
 
-    pattern = regex.compile(r"[\p{Punctuation}\p{posix_punct}]+")
+    pattern = regex.compile(
+        rf"[\p{{Punctuation}}\p{{posix_punct}}{UCSUR_PUNCT_RANGES}]+"
+    )
+
+
+class OrFilter:
+    """Instantiate with more than one filter to compose them into one filter,
+    returning True when any individual filter matches or False otherwise.
+    Requires at least two filters.
+
+    OrFilter exists as a compromise between the need to score some filters equally,
+    while not adding custom behavior to scorers.
+    I could have allowed a position to have a list of filters instead of one filter,
+    but this would require cleaning the user's input, and nested handling of lists.
+    It also would not have been as powerful- I would need another param for the and/or switch,
+    or to not give users the choice.
+
+    Instead, the user is responsible for building an OrFilter out of their desired filters.
+    """
+
+    @staticmethod
+    def __generic_filter(*filters_: Type[Filter]) -> Type[Filter]:
+
+        class CombinedFilter(Filter):
+            filters: List[Type[Filter]] = list(filters_)  # TODO: tuple better?
+
+            @classmethod
+            @override
+            @cache(maxsize=None)
+            def filter(cls, token: str) -> bool:
+                for f in cls.filters:
+                    if f.filter(token):
+                        return True
+                return False
+
+        return CombinedFilter
+
+    def __new__(cls, *filters: Type[Filter]) -> Type[Filter]:
+        if not len(filters) >= 2:
+            raise ValueError("Provide at least two Filters to OrFilter.")
+
+        subset_filters = [f for f in filters if issubclass(f, MemberFilter)]
+        if len(subset_filters) >= 2:
+            raise Warning(
+                "Prefer OrMemberFilter for combining two or more MemberFilters."
+            )
+
+        filter = cls.__generic_filter(*filters)
+
+        return filter
+
+
+class OrMemberFilter:
+    @staticmethod
+    def __subset_filter(*filters: Type[MemberFilter]) -> Type[MemberFilter]:
+        all_token_sets: List[Set[str]] = [f.tokens for f in filters]
+        all_tokens: Set[str] = set().union(*all_token_sets)
+
+        class CombinedFilter(MemberFilter):
+            tokens = all_tokens
+
+        return CombinedFilter
+
+    def __new__(cls, *filters_: Type[MemberFilter]) -> Type[MemberFilter]:
+        if not len(filters_) >= 2:
+            raise ValueError("Provide two or more MemberFilters to OrMemberFilter.")
+        filter = cls.__subset_filter(*filters_)
+        return filter
+
+
+class AndFilter(Filter):
+    """Instantiate with more than one filter to compose them into one filter,
+    returning False when any individual filter fails to match or True otherwise.
+    Requires at least two filters."""
+
+    def __new__(cls, *filters_: Type[Filter]) -> Type[Filter]:
+        if not len(filters_) >= 2:
+            raise ValueError("Must provide at least two Filters to AndFilter.")
+
+        class AnonymousAndFilter(Filter):
+            filters: List[Type[Filter]] = list(filters_)  # TODO: tuple better?
+
+            @classmethod
+            @override
+            @cache(maxsize=None)
+            def filter(cls, token: str) -> bool:
+                for f in cls.filters:
+                    if not f.filter(token):
+                        return False
+                return True
+
+        return AnonymousAndFilter
 
 
 __all__ = [
     "Alphabetic",
-    "NimiLinku",
-    "NimiLinkuAle",
+    "AndFilter",
+    "EnglishIgnorables",
+    "NimiLinkuCore",
     "NimiLinkuSandbox",
     "NimiPu",
-    "NimiPuAle",
+    "NimiPuSynonyms",
+    "NimiUCSUR",
     "Numeric",
+    "OrFilter",
     "Phonotactic",
     "ProperName",
     "Punctuation",

{sonatoki-0.2.2 → sonatoki-0.3.1}/src/sonatoki/Tokenizers.py

@@ -5,16 +5,12 @@ from typing import Set, List
 
 # PDM
 import regex
-from typing_extensions import override
+from typing_extensions import override, deprecated
 
 # LOCAL
 from sonatoki.utils import regex_escape
-from sonatoki.constants import (
-    POSIX_PUNCT,
-    UNICODE_PUNCT,
-    SENTENCE_PUNCT,
-    ALL_PUNCT_RANGES,
-)
+from sonatoki.Filters import NimiUCSUR  # seriously this sucks
+from sonatoki.constants import ALL_PUNCT, SENTENCE_PUNCT, ALL_PUNCT_RANGES
 
 regex.DEFAULT_VERSION = regex.VERSION1
 
@@ -50,7 +46,12 @@ class Regex1Tokenizer(Tokenizer):
 
 
 class WordTokenizer(SetTokenizer):
-    delimiters = set(POSIX_PUNCT + UNICODE_PUNCT)
+    delimiters = set(ALL_PUNCT)
+
+    @classmethod
+    def __helper(cls, s: str, tokens: List[str], last_match: int, i: int):
+        match = s[last_match:i].split()
+        [tokens.append(t) for t in match if t]
 
     @classmethod
     @override
@@ -60,32 +61,47 @@ class WordTokenizer(SetTokenizer):
 
         tokens: List[str] = []
 
+        i = 0  # ensure i is bound
         last_match = 0
         last_membership = s[0] in cls.delimiters
         for i, char in enumerate(s):
             mem = char in cls.delimiters
-            if mem == last_membership:
+            ucsur = NimiUCSUR.filter(char)  # always "changed" means
+            changed = (mem != last_membership) or ucsur
+            # this keeps contiguous words together, but splits UCSUR
+            if not changed:
+                continue
+
+            if ucsur:
+                if i > last_match:
+                    # Add the token before UCSUR character
+                    cls.__helper(s, tokens, last_match, i)
+                # Add UCSUR character itself as a token
+                tokens.append(char)
+                last_match = i + 1
+                last_membership = mem
                 continue
 
-            match = s[last_match:i].split()
-            # TODO: kinda sucks? what about unicode whitespace?
+            cls.__helper(s, tokens, last_match, i)
             last_match = i
             last_membership = mem
-            [tokens.append(t) for t in match if t]
-
-        match = s[last_match:].strip().split()
-        if match:
-            tokens.extend(match)
 
+        cls.__helper(s, tokens, last_match, i + 1)
         return tokens
 
 
+@deprecated(
+    "WordTokenizerRe is a previous reference implementation. Its behavior has diverged from WordTokenizer and it may not be restored."
+)
 class WordTokenizerRe(RegexTokenizer):
     pattern = re.compile(rf"""([{ALL_PUNCT_RANGES}]+|\s+)""")
 
 
+@deprecated(
+    "WordTokenizerRe1 is a previous reference implementation. Its behavior has diverged from WordTokenizer and it may not be restored."
+)
 class WordTokenizerRe1(Regex1Tokenizer):
-    """Reference implementation for WorkTokenizer."""
+    """Reference implementation for WordTokenizer."""
 
     pattern = regex.compile(r"""([\p{posix_punct}\p{Punctuation}]+|\s+)""")