sonatoki 0.1.0__py3-none-any.whl

sonatoki-0.1.0.dist-info/METADATA

@@ -0,0 +1,84 @@
+Metadata-Version: 2.1
+Name: sonatoki
+Version: 0.1.0
+Summary: ilo li moku e toki li pana e sona ni: ni li toki ala toki pona?
+Author-Email: "jan Kekan San (@gregdan3)" <gregory.danielson3@gmail.com>
+License: AGPL-3.0-or-later
+Requires-Python: >=3.8
+Requires-Dist: unidecode>=1.3.6
+Requires-Dist: regex>=2023.12.25
+Requires-Dist: typing-extensions>=4.11.0
+Requires-Dist: nltk>=3.8.1; extra == "nltk"
+Provides-Extra: nltk
+Description-Content-Type: text/markdown
+
+# sona toki
+
+## What is **sona toki**?
+
+This library, "Language Knowledge," helps you identify whether a message is in Toki Pona. No grammar checking, yet, which means this more checks whether a given message has enough Toki Pona words.
+
+I wrote it with a variety of scraps and lessons learned from a prior project, [ilo pi toki pona taso, "toki-pona-only tool"](https://github.com/gregdan3/ilo-pi-toki-pona-taso). That tool will be rewritten to use this library shortly.
+
+If you've ever worked on a similar project, you know the question "is this message in [language]" is not a consistent one- the environment, time, preferences of the speaker, and much more, can all alter whether a given message is "in toki pona," and this applies to essentially any language.
+
+This project "solves" that complex problem by offering a highly configurable and incredibly lazy parser
+
+## Quick Start
+
+Install with your preferred Python package manager. Example:
+
+```sh
+pdm init  # if your pyproject.toml doesn't exist yet
+pdm add sonatoki
+```
+
+Then get started with a script along these lines:
+
+```py
+from sonatoki.Filters import (
+    Numerics,
+    Syllabic,
+    NimiLinku,
+    Alphabetic,
+    ProperName,
+    Punctuations,
+)
+from sonatoki.Scorers import Scaling
+from sonatoki.Cleaners import ConsecutiveDuplicates
+from sonatoki.Tokenizers import word_tokenize_tok
+from sonatoki.Preprocessors import URLs, DiscordEmotes
+
+def main():
+    ilo = Ilo(
+        preprocessors=[URLs, DiscordEmotes],
+        ignoring_filters=[Numerics, Punctuations],
+        scoring_filters=[NimiLinku, Syllabic, ProperName, Alphabetic],
+        cleaners=[ConsecutiveDuplicates],
+        scorer=Scaling,
+        tokenizer=word_tokenize_tok,
+    )
+    ilo.is_toki_pona("imagine how is touch the sky")  # False
+    ilo.is_toki_pona("o pilin insa e ni: sina pilin e sewi")  # True
+
+if __name__ == "__main__":
+    main()
+```
+
+`Ilo` is highly configurable by design, so I recommend exploring the `Preprocessors`, `Filters`, and `Scorers` modules. The `Cleaners` module only contains one cleaner, which I highly recommend. The `Tokenizers` module contains several other word tokenizers, but their performance will be worse than the
+
+## Development
+
+1. Install [pdm](https://github.com/pdm-project/pdm)
+1. `pdm sync --dev`
+1. Open any file you like!
+
+## FAQ
+
+### Why isn't this README/library written in Toki Pona?
+
+The intent is to show our methodology to the Unicode Consortium, particularly to the Script Encoding Working Group (previously the Script Ad Hoc Group). As far as we're aware, zero members of the committee know Toki Pona, which unfortunately means we fall back on English.
+
+After our proposal has been examined and a result given by the committee, I will translate this file and library into Toki Pona, with a note left behind for those who do not understand it.
+
+### Why aren't any of the specific

sonatoki-0.1.0.dist-info/RECORD

@@ -0,0 +1,14 @@
+sonatoki-0.1.0.dist-info/METADATA,sha256=EQaB5tsicEQ4wYn5curehbhkzGF0qHqC1bnUbOVDCu0,3332
+sonatoki-0.1.0.dist-info/WHEEL,sha256=vnE8JVcI2Wz7GRKorsPArnBdnW2SWKWGow5gu5tHlRU,90
+sonatoki-0.1.0.dist-info/licenses/LICENSE,sha256=DZak_2itbUtvHzD3E7GNUYSRK6jdOJ-GqncQ2weavLA,34523
+sonatoki/Cleaners.py,sha256=gTZ9dSsnvKVUtxM_ECSZ-_2heh--nD5A9dCQR1ATb1c,1160
+sonatoki/Filters.py,sha256=yzhYF79GX03cOwlR_-B8SPMQPZv4UpAPytH0fQwBE70,4093
+sonatoki/Preprocessors.py,sha256=G2up2jKKSrHQtTQWYNWH_fkjgroL45ZeajVn1KUECt8,3431
+sonatoki/Scorers.py,sha256=X1vo-eIPbtl0IC5suIX6hu-4VG7NSzR90rkrLpep8WY,3690
+sonatoki/Tokenizers.py,sha256=epOG3jZHI3MSO_L_6Z3zsSkexDEMLVzA2ARg6EnPMO0,1628
+sonatoki/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+sonatoki/__main__.py,sha256=6xc-wIrrFo9wTyn4zRQNAmqwmJBtVvCMwV-CrM-hueA,82
+sonatoki/constants.py,sha256=h5rbCfu9YF76BsjQYud5d2wq1HODY05zOaw0Ir1cwjo,1320
+sonatoki/ilo.py,sha256=Uu0zipAF-L-5Wxw_EBB7-EMc40PM4WBa59Atq0zmYYE,3482
+sonatoki/linku.json,sha256=MdFuFRIHniPDUVxKEKuUg1KyzPVgcCj4ZeyvburCwD0,270928
+sonatoki-0.1.0.dist-info/RECORD,,

sonatoki-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: pdm-backend (2.3.0)
+Root-Is-Purelib: true
+Tag: py3-none-any