rustling 0.6.0__tar.gz → 0.8.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (186) hide show
  1. rustling-0.8.0/.github/workflows/build-wheel.yml +42 -0
  2. {rustling-0.6.0 → rustling-0.8.0}/.github/workflows/python.yml +1 -1
  3. {rustling-0.6.0 → rustling-0.8.0}/.github/workflows/release.yml +2 -1
  4. {rustling-0.6.0 → rustling-0.8.0}/.gitignore +3 -0
  5. {rustling-0.6.0 → rustling-0.8.0}/.pre-commit-config.yaml +11 -1
  6. rustling-0.8.0/CHANGELOG.md +83 -0
  7. {rustling-0.6.0 → rustling-0.8.0}/Cargo.lock +26 -29
  8. {rustling-0.6.0 → rustling-0.8.0}/Cargo.toml +5 -15
  9. {rustling-0.6.0 → rustling-0.8.0}/PKG-INFO +43 -33
  10. rustling-0.8.0/README.md +70 -0
  11. rustling-0.8.0/benchmarks/README.md +132 -0
  12. rustling-0.8.0/benchmarks/run_conllu.py +300 -0
  13. rustling-0.8.0/benchmarks/run_elan.py +299 -0
  14. rustling-0.8.0/benchmarks/run_textgrid.py +320 -0
  15. rustling-0.6.0/benchmarks/update_docs.py → rustling-0.8.0/benchmarks/update_readme.py +86 -66
  16. {rustling-0.6.0 → rustling-0.8.0}/pyproject.toml +3 -4
  17. rustling-0.8.0/python/docs/_static/brown_eve_mlum.png +0 -0
  18. {rustling-0.6.0 → rustling-0.8.0}/python/docs/api.rst +4 -0
  19. rustling-0.8.0/python/docs/chat/formats.rst +219 -0
  20. rustling-0.8.0/python/docs/chat/frequencies.rst +99 -0
  21. rustling-0.8.0/python/docs/chat/headers.rst +171 -0
  22. rustling-0.8.0/python/docs/chat/measures.rst +62 -0
  23. rustling-0.8.0/python/docs/chat/quickstart.rst +349 -0
  24. rustling-0.8.0/python/docs/chat/read.rst +479 -0
  25. rustling-0.8.0/python/docs/chat/transcriptions.rst +626 -0
  26. rustling-0.8.0/python/docs/chat/write.rst +48 -0
  27. rustling-0.8.0/python/docs/chat.rst +19 -0
  28. rustling-0.8.0/python/docs/conllu.rst +221 -0
  29. rustling-0.8.0/python/docs/elan.rst +275 -0
  30. rustling-0.8.0/python/docs/index.rst +104 -0
  31. rustling-0.8.0/python/docs/srt.rst +240 -0
  32. rustling-0.8.0/python/docs/textgrid.rst +288 -0
  33. {rustling-0.6.0 → rustling-0.8.0}/python/docs/wordseg.rst +8 -8
  34. rustling-0.8.0/python/rustling/__init__.py +36 -0
  35. rustling-0.8.0/python/rustling/_lib_name.pyi +3 -0
  36. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/chat/__init__.py +2 -0
  37. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/chat/__init__.pyi +365 -15
  38. rustling-0.8.0/python/rustling/chat/_read_chat.py +78 -0
  39. rustling-0.8.0/python/rustling/conllu/__init__.py +13 -0
  40. rustling-0.8.0/python/rustling/conllu/__init__.pyi +326 -0
  41. rustling-0.8.0/python/rustling/conllu/_read_conllu.py +53 -0
  42. rustling-0.8.0/python/rustling/elan/__init__.py +19 -0
  43. rustling-0.8.0/python/rustling/elan/__init__.pyi +442 -0
  44. rustling-0.8.0/python/rustling/elan/_read_elan.py +53 -0
  45. rustling-0.8.0/python/rustling/srt/__init__.py +11 -0
  46. rustling-0.8.0/python/rustling/srt/__init__.pyi +345 -0
  47. rustling-0.8.0/python/rustling/srt/_read_srt.py +53 -0
  48. rustling-0.8.0/python/rustling/textgrid/__init__.py +23 -0
  49. rustling-0.8.0/python/rustling/textgrid/__init__.pyi +348 -0
  50. rustling-0.8.0/python/rustling/textgrid/_read_textgrid.py +54 -0
  51. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/wordseg/__init__.pyi +113 -9
  52. {rustling-0.6.0 → rustling-0.8.0}/python/stubtest_allowlist.txt +35 -0
  53. {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_chat/test_chat.py +299 -40
  54. rustling-0.8.0/python/tests/test_conllu/conftest.py +45 -0
  55. rustling-0.8.0/python/tests/test_conllu/test_conllu.py +276 -0
  56. rustling-0.8.0/python/tests/test_elan/conftest.py +33 -0
  57. rustling-0.8.0/python/tests/test_elan/test_elan.py +546 -0
  58. rustling-0.8.0/python/tests/test_textgrid/conftest.py +0 -0
  59. rustling-0.8.0/python/tests/test_textgrid/test_textgrid.py +398 -0
  60. rustling-0.8.0/python/tests/test_wordseg/test_dag_hmm_segmenter.py +69 -0
  61. {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_wordseg/test_hmm_segmenter.py +42 -0
  62. {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_wordseg/test_longest_string_matching.py +36 -0
  63. {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_wordseg/test_random_segmenter.py +31 -0
  64. {rustling-0.6.0 → rustling-0.8.0}/src/chat/clean_utterance.rs +481 -11
  65. rustling-0.8.0/src/chat/conllu_writer.rs +147 -0
  66. rustling-0.8.0/src/chat/elan_writer.rs +486 -0
  67. {rustling-0.6.0 → rustling-0.8.0}/src/chat/header.rs +85 -140
  68. rustling-0.8.0/src/chat/header_py.rs +312 -0
  69. {rustling-0.6.0 → rustling-0.8.0}/src/chat/ipsyn.rs +4 -0
  70. {rustling-0.6.0 → rustling-0.8.0}/src/chat/mod.rs +20 -5
  71. {rustling-0.6.0 → rustling-0.8.0}/src/chat/reader.rs +711 -1191
  72. rustling-0.8.0/src/chat/reader_py.rs +1358 -0
  73. rustling-0.8.0/src/chat/srt_writer.rs +324 -0
  74. rustling-0.8.0/src/chat/textgrid_writer.rs +201 -0
  75. {rustling-0.6.0 → rustling-0.8.0}/src/chat/utterance.rs +108 -309
  76. rustling-0.8.0/src/chat/utterance_py.rs +330 -0
  77. rustling-0.8.0/src/chat/validation.rs +1121 -0
  78. rustling-0.8.0/src/conllu/chat_writer.rs +222 -0
  79. rustling-0.8.0/src/conllu/mod.rs +885 -0
  80. rustling-0.8.0/src/conllu/reader_py.rs +547 -0
  81. rustling-0.8.0/src/elan/chat_writer.rs +414 -0
  82. rustling-0.8.0/src/elan/mod.rs +32 -0
  83. rustling-0.8.0/src/elan/reader.rs +1182 -0
  84. rustling-0.8.0/src/elan/reader_py.rs +629 -0
  85. rustling-0.8.0/src/elan/srt_writer.rs +242 -0
  86. rustling-0.8.0/src/elan/textgrid_writer.rs +155 -0
  87. {rustling-0.6.0 → rustling-0.8.0}/src/hmm/mod.rs +52 -276
  88. rustling-0.8.0/src/hmm/py.rs +254 -0
  89. {rustling-0.6.0 → rustling-0.8.0}/src/lib.rs +14 -1
  90. {rustling-0.6.0 → rustling-0.8.0}/src/lm/mod.rs +40 -382
  91. rustling-0.8.0/src/lm/py.rs +370 -0
  92. rustling-0.6.0/src/ngram.rs → rustling-0.8.0/src/ngram/mod.rs +30 -222
  93. rustling-0.8.0/src/ngram/py.rs +204 -0
  94. {rustling-0.6.0 → rustling-0.8.0}/src/perceptron_pos_tagger/mod.rs +8 -141
  95. rustling-0.8.0/src/perceptron_pos_tagger/py.rs +144 -0
  96. {rustling-0.6.0 → rustling-0.8.0}/src/persistence.rs +28 -2
  97. rustling-0.8.0/src/prelude.rs +15 -0
  98. rustling-0.6.0/src/seq_feature.rs → rustling-0.8.0/src/seq_feature/mod.rs +23 -80
  99. rustling-0.8.0/src/seq_feature/py.rs +74 -0
  100. rustling-0.8.0/src/sources/cache.rs +76 -0
  101. rustling-0.8.0/src/sources/git.rs +123 -0
  102. rustling-0.8.0/src/sources/mod.rs +57 -0
  103. rustling-0.8.0/src/sources/url.rs +68 -0
  104. rustling-0.8.0/src/srt/chat_writer.rs +140 -0
  105. rustling-0.8.0/src/srt/elan_writer.rs +211 -0
  106. rustling-0.8.0/src/srt/mod.rs +986 -0
  107. rustling-0.8.0/src/srt/reader_py.rs +523 -0
  108. rustling-0.8.0/src/srt/textgrid_writer.rs +122 -0
  109. rustling-0.8.0/src/textgrid/chat_writer.rs +219 -0
  110. rustling-0.8.0/src/textgrid/elan_writer.rs +245 -0
  111. rustling-0.8.0/src/textgrid/mod.rs +34 -0
  112. rustling-0.8.0/src/textgrid/reader.rs +1691 -0
  113. rustling-0.8.0/src/textgrid/reader_py.rs +750 -0
  114. rustling-0.8.0/src/textgrid/srt_writer.rs +208 -0
  115. {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/dag_hmm.rs +45 -60
  116. rustling-0.8.0/src/wordseg/dag_hmm_py.rs +76 -0
  117. {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/hmm.rs +86 -83
  118. rustling-0.8.0/src/wordseg/hmm_py.rs +99 -0
  119. {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/longest_string_matching/mod.rs +15 -100
  120. rustling-0.8.0/src/wordseg/longest_string_matching/py.rs +109 -0
  121. rustling-0.8.0/src/wordseg/mod.rs +141 -0
  122. {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/random_segmenter.rs +12 -59
  123. rustling-0.8.0/src/wordseg/random_segmenter_py.rs +68 -0
  124. rustling-0.6.0/AGENTS.md +0 -335
  125. rustling-0.6.0/CHANGELOG.md +0 -42
  126. rustling-0.6.0/CLAUDE.md +0 -335
  127. rustling-0.6.0/README.md +0 -57
  128. rustling-0.6.0/benchmarks/README.md +0 -89
  129. rustling-0.6.0/python/docs/chat.rst +0 -7
  130. rustling-0.6.0/python/docs/index.rst +0 -110
  131. rustling-0.6.0/python/rustling/__init__.py +0 -11
  132. rustling-0.6.0/src/wordseg/mod.rs +0 -55
  133. {rustling-0.6.0 → rustling-0.8.0}/.flake8 +0 -0
  134. {rustling-0.6.0 → rustling-0.8.0}/.gitattributes +0 -0
  135. {rustling-0.6.0 → rustling-0.8.0}/.github/dependabot.yml +0 -0
  136. {rustling-0.6.0 → rustling-0.8.0}/.github/workflows/rust.yml +0 -0
  137. {rustling-0.6.0 → rustling-0.8.0}/.readthedocs.yaml +0 -0
  138. {rustling-0.6.0 → rustling-0.8.0}/LICENSE.md +0 -0
  139. {rustling-0.6.0 → rustling-0.8.0}/benchmarks/common/__init__.py +0 -0
  140. {rustling-0.6.0 → rustling-0.8.0}/benchmarks/common/data.py +0 -0
  141. {rustling-0.6.0 → rustling-0.8.0}/benchmarks/run_chat.py +0 -0
  142. {rustling-0.6.0 → rustling-0.8.0}/benchmarks/run_hmm.py +0 -0
  143. {rustling-0.6.0 → rustling-0.8.0}/benchmarks/run_lm.py +0 -0
  144. {rustling-0.6.0 → rustling-0.8.0}/benchmarks/run_perceptron_pos_tagger.py +0 -0
  145. {rustling-0.6.0 → rustling-0.8.0}/benchmarks/run_wordseg.py +0 -0
  146. {rustling-0.6.0 → rustling-0.8.0}/build.rs +0 -0
  147. {rustling-0.6.0 → rustling-0.8.0}/python/docs/Makefile +0 -0
  148. {rustling-0.6.0 → rustling-0.8.0}/python/docs/README.md +0 -0
  149. {rustling-0.6.0 → rustling-0.8.0}/python/docs/_static/favicon.ico +0 -0
  150. {rustling-0.6.0 → rustling-0.8.0}/python/docs/_static/logo-with-text.svg +0 -0
  151. {rustling-0.6.0 → rustling-0.8.0}/python/docs/_templates/copyright.html +0 -0
  152. {rustling-0.6.0 → rustling-0.8.0}/python/docs/conf.py +0 -0
  153. {rustling-0.6.0 → rustling-0.8.0}/python/docs/hmm.rst +0 -0
  154. {rustling-0.6.0 → rustling-0.8.0}/python/docs/lm.rst +0 -0
  155. {rustling-0.6.0 → rustling-0.8.0}/python/docs/make.bat +0 -0
  156. {rustling-0.6.0 → rustling-0.8.0}/python/docs/ngram.rst +0 -0
  157. {rustling-0.6.0 → rustling-0.8.0}/python/docs/perceptron_pos_tagger.rst +0 -0
  158. {rustling-0.6.0 → rustling-0.8.0}/python/docs/requirements.txt +0 -0
  159. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/hmm/__init__.py +0 -0
  160. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/hmm/__init__.pyi +0 -0
  161. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/lm/__init__.py +0 -0
  162. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/lm/__init__.pyi +0 -0
  163. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/ngram/__init__.py +0 -0
  164. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/ngram/__init__.pyi +0 -0
  165. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/perceptron_pos_tagger/__init__.py +0 -0
  166. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/perceptron_pos_tagger/__init__.pyi +0 -0
  167. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/py.typed +0 -0
  168. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/seq_feature/__init__.py +0 -0
  169. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/seq_feature/__init__.pyi +0 -0
  170. {rustling-0.6.0 → rustling-0.8.0}/python/rustling/wordseg/__init__.py +0 -0
  171. {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_benchmarks.py +0 -0
  172. {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_chat/conftest.py +0 -0
  173. {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_hmm.py +0 -0
  174. {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_lm.py +0 -0
  175. {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_ngram.py +0 -0
  176. {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_perceptron_pos_tagger.py +0 -0
  177. {rustling-0.6.0 → rustling-0.8.0}/src/hmm/model.fbs +0 -0
  178. {rustling-0.6.0 → rustling-0.8.0}/src/hmm/model_generated.rs +0 -0
  179. {rustling-0.6.0 → rustling-0.8.0}/src/lm/model.fbs +0 -0
  180. {rustling-0.6.0 → rustling-0.8.0}/src/lm/model_generated.rs +0 -0
  181. {rustling-0.6.0 → rustling-0.8.0}/src/perceptron_pos_tagger/model.fbs +0 -0
  182. {rustling-0.6.0 → rustling-0.8.0}/src/perceptron_pos_tagger/model_generated.rs +0 -0
  183. {rustling-0.6.0 → rustling-0.8.0}/src/trie.rs +0 -0
  184. {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/bmes.rs +0 -0
  185. {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/longest_string_matching/model.fbs +0 -0
  186. {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/longest_string_matching/model_generated.rs +0 -0
@@ -0,0 +1,42 @@
1
+ name: Build Linux x86_64 wheel
2
+
3
+ on:
4
+ workflow_dispatch:
5
+ inputs:
6
+ git_ref:
7
+ description: "Git ref to build from (tag, branch, or commit)"
8
+ required: true
9
+ default: "v0.6.0"
10
+
11
+ env:
12
+ FLATC_VERSION: "25.12.19"
13
+
14
+ jobs:
15
+ build:
16
+ name: Build manylinux x86_64 wheel
17
+ runs-on: ubuntu-latest
18
+ steps:
19
+ - uses: actions/checkout@v6
20
+ with:
21
+ ref: ${{ github.event.inputs.git_ref }}
22
+
23
+ - name: Build wheels
24
+ uses: PyO3/maturin-action@v1
25
+ with:
26
+ target: x86_64
27
+ args: --release --out dist -i python3.10
28
+ sccache: "true"
29
+ manylinux: auto
30
+ before-script-linux: |
31
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
32
+ command -v unzip || yum install -y unzip 2>/dev/null || (apt-get update -qq && apt-get install -y -qq unzip)
33
+ unzip -q flatc.zip
34
+ rm flatc.zip
35
+ chmod +x flatc
36
+ mv flatc /usr/local/bin/
37
+
38
+ - name: Upload wheel
39
+ uses: actions/upload-artifact@v7
40
+ with:
41
+ name: wheel-linux-x86_64
42
+ path: dist
@@ -146,7 +146,7 @@ jobs:
146
146
  uses: PyO3/maturin-action@v1
147
147
  with:
148
148
  target: ${{ matrix.target }}
149
- args: --release --out dist
149
+ args: --release --out dist -i python3.10
150
150
  sccache: "true"
151
151
  manylinux: auto
152
152
  before-script-linux: |
@@ -71,7 +71,7 @@ jobs:
71
71
  uses: PyO3/maturin-action@v1
72
72
  with:
73
73
  target: ${{ matrix.target }}
74
- args: --release --out dist
74
+ args: --release --out dist -i python3.10
75
75
  sccache: "true"
76
76
  manylinux: auto
77
77
  before-script-linux: |
@@ -201,6 +201,7 @@ jobs:
201
201
  run: |
202
202
  curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
203
203
  unzip -q flatc.zip
204
+ rm flatc.zip
204
205
  chmod +x flatc
205
206
  sudo mv flatc /usr/local/bin/
206
207
 
@@ -77,3 +77,6 @@ benchmarks/.results/
77
77
  # This is a library, not an application.
78
78
  Cargo.lock
79
79
  uv.lock
80
+
81
+ # From maturin develop
82
+ python/rustling/librustling.dylib.dSYM/
@@ -14,7 +14,7 @@ repos:
14
14
  types: [rust]
15
15
  pass_filenames: false
16
16
  - repo: https://github.com/psf/black-pre-commit-mirror
17
- rev: 26.1.0
17
+ rev: 26.3.0
18
18
  hooks:
19
19
  - id: black
20
20
  files: ^(python|benchmarks)/
@@ -23,3 +23,13 @@ repos:
23
23
  hooks:
24
24
  - id: flake8
25
25
  files: ^(python|benchmarks)/
26
+ - repo: local
27
+ hooks:
28
+ - id: mypy
29
+ name: mypy
30
+ entry: uvx mypy
31
+ language: system
32
+ types: [python]
33
+ files: ^python/rustling/
34
+ pass_filenames: false
35
+ args: [python/rustling/]
@@ -0,0 +1,83 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [Unreleased]
9
+
10
+ ## [0.8.0] - 2026-03-20
11
+
12
+ ### Added
13
+
14
+ - Support for data formats:
15
+ * CoNLL-U for Universal Dependencies
16
+ * ELAN for annotated multimedia data
17
+ * TextGrid for Praat annotations
18
+ * SRT for subtitles
19
+ - CHAT data handling:
20
+ * Added a convenience function `read_chat.
21
+ * Added `from_git` and `from_url` methods for remote data sources.
22
+
23
+ ## [0.7.0] - 2026-03-14
24
+
25
+ ### Added
26
+
27
+ - Word segmentation:
28
+ * Added `score` method for the HMM and DAG-HMM segmenters.
29
+ * `predict` method can optionally output offsets for the (start, end) indices
30
+ of segmented words compared to the original string.
31
+ - CHAT parsing: Support custom tier names other than the standard %mor and %gra.
32
+ - Python model classes are now subclassable.
33
+
34
+ ### Changed
35
+
36
+ - Ngram counters: `Ngrams.most_common` now sorts tuples lexicographically
37
+ when counts are tied.
38
+ - CHAT parsing:
39
+ * If a date is available at `Headers`'s `date`,
40
+ it's now a Python `datetime.date` object instead of a string.
41
+ * In handling the main tier transcription for creating `Token` objects:
42
+ - Special form markers suffixed with "@" are now stripped.
43
+ - Words that have partiallly parenthetical material have the parentheses
44
+ removed, e.g., (un)til -> until, sit(ting) -> sitting.
45
+ * Renamed the `CHAT.raw` attribute to `CHAT.audible` for a best-effort,
46
+ audibly faithful transcription string, to facilitate automatic speech recognition,
47
+ forced alignment, etc.
48
+ * A subset of the testchat/bad dataset is now used to validate CHAT data format.
49
+ - Refactored core Rust code so that Rust-only consumers no longer need PyO3/Python.
50
+
51
+ ## [0.6.0] - 2026-03-05
52
+
53
+ ### Added
54
+
55
+ - Hidden Markov Model (HMM)
56
+ - Word segmentation: Added DAG-HMM word segmenter
57
+ - CHAT parsing: Added `from_utterances` method
58
+
59
+ ### Changed
60
+
61
+ - Models are now persisted as a zstd-compressed FlatBuffers binary.
62
+
63
+ ## [0.5.0] - 2026-02-18
64
+
65
+ ### Added
66
+
67
+ - CHAT parsing for TalkBank and CHILDES data
68
+
69
+ ## [0.4.0] - 2026-02-08
70
+
71
+ ### Added
72
+
73
+ - N-grams and language models
74
+
75
+ ## [0.3.0] - 2026-02-06
76
+
77
+ ### Added
78
+
79
+ - Averaged perceptron tagger
80
+
81
+ ## [0.2.0] - 2026-02-04
82
+
83
+ - Initial release, with longest string matching and random segmenter for word segmentation
@@ -81,9 +81,9 @@ dependencies = [
81
81
 
82
82
  [[package]]
83
83
  name = "cc"
84
- version = "1.2.56"
84
+ version = "1.2.57"
85
85
  source = "registry+https://github.com/rust-lang/crates.io-index"
86
- checksum = "aebf35691d1bfb0ac386a69bac2fde4dd276fb618cf8bf4f5318fe285e821bb2"
86
+ checksum = "7a0dd1ca384932ff3641c8718a02769f1698e7563dc6974ffd03346116310423"
87
87
  dependencies = [
88
88
  "find-msvc-tools",
89
89
  "jobserver",
@@ -380,9 +380,9 @@ dependencies = [
380
380
 
381
381
  [[package]]
382
382
  name = "itoa"
383
- version = "1.0.17"
383
+ version = "1.0.18"
384
384
  source = "registry+https://github.com/rust-lang/crates.io-index"
385
- checksum = "92ecc6618181def0457392ccd0ee51198e065e016d1d527a7ac1b6dc7c1f09d2"
385
+ checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
386
386
 
387
387
  [[package]]
388
388
  name = "jobserver"
@@ -418,9 +418,9 @@ checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7"
418
418
 
419
419
  [[package]]
420
420
  name = "libc"
421
- version = "0.2.182"
421
+ version = "0.2.183"
422
422
  source = "registry+https://github.com/rust-lang/crates.io-index"
423
- checksum = "6800badb6cb2082ffd7b6a67e6125bb39f18782f793520caee8cb8846be06112"
423
+ checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d"
424
424
 
425
425
  [[package]]
426
426
  name = "linux-raw-sys"
@@ -467,9 +467,9 @@ checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050"
467
467
 
468
468
  [[package]]
469
469
  name = "once_cell"
470
- version = "1.21.3"
470
+ version = "1.21.4"
471
471
  source = "registry+https://github.com/rust-lang/crates.io-index"
472
- checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d"
472
+ checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
473
473
 
474
474
  [[package]]
475
475
  name = "pbkdf2"
@@ -582,6 +582,16 @@ dependencies = [
582
582
  "syn",
583
583
  ]
584
584
 
585
+ [[package]]
586
+ name = "quick-xml"
587
+ version = "0.39.2"
588
+ source = "registry+https://github.com/rust-lang/crates.io-index"
589
+ checksum = "958f21e8e7ceb5a1aa7fa87fab28e7c75976e0bfe7e23ff069e0a260f894067d"
590
+ dependencies = [
591
+ "memchr",
592
+ "serde",
593
+ ]
594
+
585
595
  [[package]]
586
596
  name = "quote"
587
597
  version = "1.0.45"
@@ -699,11 +709,12 @@ dependencies = [
699
709
 
700
710
  [[package]]
701
711
  name = "rustling"
702
- version = "0.6.0"
712
+ version = "0.8.0"
703
713
  dependencies = [
704
714
  "fancy-regex",
705
715
  "flatbuffers",
706
716
  "pyo3",
717
+ "quick-xml",
707
718
  "rand",
708
719
  "rayon",
709
720
  "regex",
@@ -840,9 +851,9 @@ checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
840
851
 
841
852
  [[package]]
842
853
  name = "tempfile"
843
- version = "3.26.0"
854
+ version = "3.27.0"
844
855
  source = "registry+https://github.com/rust-lang/crates.io-index"
845
- checksum = "82a72c767771b47409d2345987fda8628641887d5466101319899796367354a0"
856
+ checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
846
857
  dependencies = [
847
858
  "fastrand",
848
859
  "getrandom 0.4.2",
@@ -897,9 +908,9 @@ checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853"
897
908
 
898
909
  [[package]]
899
910
  name = "uuid"
900
- version = "1.21.0"
911
+ version = "1.22.0"
901
912
  source = "registry+https://github.com/rust-lang/crates.io-index"
902
- checksum = "b672338555252d43fd2240c714dc444b8c6fb0a5c5335e65a07bba7742735ddb"
913
+ checksum = "a68d3c8f01c0cfa54a75291d83601161799e4a89a39e0929f4b0354d88757a37"
903
914
  dependencies = [
904
915
  "getrandom 0.4.2",
905
916
  "js-sys",
@@ -1136,26 +1147,12 @@ name = "zeroize"
1136
1147
  version = "1.8.2"
1137
1148
  source = "registry+https://github.com/rust-lang/crates.io-index"
1138
1149
  checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0"
1139
- dependencies = [
1140
- "zeroize_derive",
1141
- ]
1142
-
1143
- [[package]]
1144
- name = "zeroize_derive"
1145
- version = "1.4.3"
1146
- source = "registry+https://github.com/rust-lang/crates.io-index"
1147
- checksum = "85a5b4158499876c763cb03bc4e49185d3cccbabb15b33c627f7884f43db852e"
1148
- dependencies = [
1149
- "proc-macro2",
1150
- "quote",
1151
- "syn",
1152
- ]
1153
1150
 
1154
1151
  [[package]]
1155
1152
  name = "zip"
1156
- version = "8.2.0"
1153
+ version = "8.3.0"
1157
1154
  source = "registry+https://github.com/rust-lang/crates.io-index"
1158
- checksum = "b680f2a0cd479b4cff6e1233c483fdead418106eae419dc60200ae9850f6d004"
1155
+ checksum = "4a243cfad17427fc077f529da5a95abe4e94fd2bfdb601611870a6557cc67657"
1159
1156
  dependencies = [
1160
1157
  "aes",
1161
1158
  "bzip2",
@@ -1,6 +1,6 @@
1
1
  [package]
2
2
  name = "rustling"
3
- version = "0.6.0"
3
+ version = "0.8.0"
4
4
  edition = "2024"
5
5
  description = "A blazingly fast library for computational linguistics"
6
6
  license = "MIT"
@@ -8,21 +8,10 @@ repository = "https://github.com/jacksonllee/rustling"
8
8
  documentation = "https://docs.rs/rustling"
9
9
  readme = "README.md"
10
10
  keywords = [
11
- "computational-linguistics",
11
+ # At most 5 keywords per crate. Each keyword must have fewer than 20 characters.
12
12
  "linguistics",
13
- "natural-language-processing",
14
13
  "nlp",
15
14
  "text-processing",
16
- "word-segmentation",
17
- "part-of-speech-tagging",
18
- "language-models",
19
- "ngrams",
20
- "childes",
21
- "talkbank",
22
- "chat",
23
- "averaged-perceptron",
24
- "hidden-markov-model",
25
- "longest-string-matching",
26
15
  ]
27
16
  categories = ["science", "text-processing"]
28
17
 
@@ -32,7 +21,7 @@ name = "rustling"
32
21
  crate-type = ["rlib"]
33
22
 
34
23
  [dependencies]
35
- pyo3 = { version = "0.28.2", features = ["abi3-py310"] }
24
+ pyo3 = { version = "0.28.2", features = ["abi3-py310"], optional = true }
36
25
  rand = "0.10"
37
26
  rayon = { version = "1.10", optional = true }
38
27
  fancy-regex = "0.17"
@@ -44,6 +33,7 @@ serde_json = "1.0"
44
33
  uuid = { version = "1", features = ["v4"] }
45
34
  walkdir = "2"
46
35
  zip = "8"
36
+ quick-xml = { version = "0.39", features = ["serialize"] }
47
37
  zstd = "0.13"
48
38
 
49
39
  [dev-dependencies]
@@ -51,5 +41,5 @@ tempfile = "3"
51
41
 
52
42
  [features]
53
43
  default = ["parallel"]
54
- extension-module = ["pyo3/extension-module"]
44
+ extension-module = ["pyo3", "pyo3/extension-module"]
55
45
  parallel = ["rayon"]
@@ -1,7 +1,7 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: rustling
3
- Version: 0.6.0
4
- Classifier: Development Status :: 2 - Pre-Alpha
3
+ Version: 0.8.0
4
+ Classifier: Development Status :: 3 - Alpha
5
5
  Classifier: Intended Audience :: Developers
6
6
  Classifier: Intended Audience :: Education
7
7
  Classifier: Intended Audience :: Information Technology
@@ -27,64 +27,74 @@ Classifier: Topic :: Text Processing :: Indexing
27
27
  Classifier: Topic :: Text Processing :: Linguistic
28
28
  License-File: LICENSE.md
29
29
  Summary: A blazingly fast library for computational linguistics
30
- Keywords: computational-linguistics,linguistics,natural-language-processing,nlp,text-processing,word-segmentation,part-of-speech-tagging,language-models,ngrams,childes,talkbank,chat,averaged-perceptron,hidden-markov-model,longest-string-matching
30
+ Keywords: linguistics,nlp,text-processing
31
31
  Author-email: "Jackson L. Lee" <jacksonlunlee@gmail.com>
32
32
  License-Expression: MIT
33
33
  Requires-Python: >=3.10
34
34
  Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
35
- Project-URL: Changelog, https://github.com/jacksonllee/rustling/blob/main/CHANGELOG.md
36
- Project-URL: Python Documentation, https://rustling.readthedocs.io
37
- Project-URL: Rust Documentation, https://docs.rs/rustling
38
35
  Project-URL: Source, https://github.com/jacksonllee/rustling
39
36
 
40
- <img src="https://raw.githubusercontent.com/jacksonllee/rustling/main/python/docs/_static/logo-with-text.svg" alt="Rustling" height="120">
37
+ <div align="center">
38
+ <a href="https://github.com/jacksonllee/rustling">
39
+ <img src="https://raw.githubusercontent.com/jacksonllee/rustling/main/python/docs/_static/logo-with-text.svg" alt="Rustling" height="120">
40
+ </a>
41
+ </div>
42
+ <br>
41
43
 
42
44
  [![PyPI](https://img.shields.io/pypi/v/rustling.svg)](https://pypi.org/project/rustling/)
45
+ [![Conda Version](https://img.shields.io/conda/vn/conda-forge/rustling.svg)](https://anaconda.org/conda-forge/rustling)
43
46
  [![crates.io](https://img.shields.io/crates/v/rustling.svg)](https://crates.io/crates/rustling)
44
47
 
45
48
  Rustling is a blazingly fast library for computational linguistics.
46
- It is written in Rust, with Python bindings.
49
+ It aims to provide flexible and efficient tools to facilitate further research.
47
50
 
48
- Documentation: [Python](https://rustling.readthedocs.io/) | [Rust](https://docs.rs/rustling)
51
+ Documentation: [Python](https://docs.rustling.io/) | [Rust](https://docs.rs/rustling)
49
52
 
50
- ## Features
53
+ Currently implemented features:
51
54
 
52
- - N-grams
53
- - Language models
54
- - Hidden Markov model
55
- - Word segmentation
56
- - Part-of-speech tagging
57
- - CHAT parsing for TalkBank and CHILDES data
55
+ * Sequence modeling:
56
+
57
+ - N-grams and related language models
58
+ - Hidden Markov model
59
+ - Word segmentation
60
+ - Averaged perceptron part-of-speech tagging
61
+
62
+ * Handling richly formatted data,
63
+ supporting cross-format conversion as well as both local and remote sources for data ingestion:
64
+
65
+ - CHAT for TalkBank and CHILDES
66
+ - ELAN for annotated multimedia data
67
+ - TextGrid for Praat annotations
68
+ - CoNLL-U for University Dependencies
69
+ - SRT for SubRip subtitles
58
70
 
59
71
  ## Performance
60
72
 
61
- | Component | Task | Speedup | vs. |
62
- |---|---|---|---|
63
- | **Language Models** | Fit | **10x** | NLTK |
64
- | | Score | **1.9x** | NLTK |
65
- | | Generate | **106--114x** | NLTK |
66
- | **Word Segmentation** | LongestStringMatching | **9x** | wordseg |
67
- | **POS Tagging** | Training | **5x** | NLTK |
68
- | | Tagging | **18x** | NLTK |
69
- | **HMM** | Fit | **13x** | hmmlearn |
70
- | | Predict | **0.9x** | hmmlearn |
71
- | | Score | **5x** | hmmlearn |
72
- | **CHAT Parsing** | Reading from a ZIP archive | **43x** | pylangacq |
73
- | | Reading from strings | **70x** | pylangacq |
74
- | | Parsing utterances | **15x** | pylangacq |
75
- | | Parsing tokens | **9x** | pylangacq |
76
-
77
- See [`benchmarks/`](https://github.com/jacksonllee/rustling/tree/main/benchmarks) for reproduction scripts.
73
+ Rustling is highly performant because it is implemented in Rust under the hood.
74
+ For benchmarks comparing Rustling against other Python packages with similar functionalities,
75
+ please see [`benchmarks`](https://github.com/jacksonllee/rustling/tree/main/benchmarks).
78
76
 
79
77
 
80
78
  ## Installation
81
79
 
82
80
  ### Python
83
81
 
82
+ Using pip:
83
+
84
84
  ```bash
85
85
  pip install rustling
86
86
  ```
87
87
 
88
+ Using conda:
89
+
90
+ ```bash
91
+ conda install -c conda-forge rustling
92
+ ```
93
+
94
+ For Pyodide, pre-built WASM wheels (with multithreading disabled, as Pyodide does not support it)
95
+ are available from each [GitHub release](https://github.com/jacksonllee/rustling/releases)
96
+ — look for the ``.whl`` file with ``emscripten`` in the filename.
97
+
88
98
  ### Rust
89
99
 
90
100
  ```bash
@@ -0,0 +1,70 @@
1
+ <div align="center">
2
+ <a href="https://github.com/jacksonllee/rustling">
3
+ <img src="https://raw.githubusercontent.com/jacksonllee/rustling/main/python/docs/_static/logo-with-text.svg" alt="Rustling" height="120">
4
+ </a>
5
+ </div>
6
+ <br>
7
+
8
+ [![PyPI](https://img.shields.io/pypi/v/rustling.svg)](https://pypi.org/project/rustling/)
9
+ [![Conda Version](https://img.shields.io/conda/vn/conda-forge/rustling.svg)](https://anaconda.org/conda-forge/rustling)
10
+ [![crates.io](https://img.shields.io/crates/v/rustling.svg)](https://crates.io/crates/rustling)
11
+
12
+ Rustling is a blazingly fast library for computational linguistics.
13
+ It aims to provide flexible and efficient tools to facilitate further research.
14
+
15
+ Documentation: [Python](https://docs.rustling.io/) | [Rust](https://docs.rs/rustling)
16
+
17
+ Currently implemented features:
18
+
19
+ * Sequence modeling:
20
+
21
+ - N-grams and related language models
22
+ - Hidden Markov model
23
+ - Word segmentation
24
+ - Averaged perceptron part-of-speech tagging
25
+
26
+ * Handling richly formatted data,
27
+ supporting cross-format conversion as well as both local and remote sources for data ingestion:
28
+
29
+ - CHAT for TalkBank and CHILDES
30
+ - ELAN for annotated multimedia data
31
+ - TextGrid for Praat annotations
32
+ - CoNLL-U for University Dependencies
33
+ - SRT for SubRip subtitles
34
+
35
+ ## Performance
36
+
37
+ Rustling is highly performant because it is implemented in Rust under the hood.
38
+ For benchmarks comparing Rustling against other Python packages with similar functionalities,
39
+ please see [`benchmarks`](https://github.com/jacksonllee/rustling/tree/main/benchmarks).
40
+
41
+
42
+ ## Installation
43
+
44
+ ### Python
45
+
46
+ Using pip:
47
+
48
+ ```bash
49
+ pip install rustling
50
+ ```
51
+
52
+ Using conda:
53
+
54
+ ```bash
55
+ conda install -c conda-forge rustling
56
+ ```
57
+
58
+ For Pyodide, pre-built WASM wheels (with multithreading disabled, as Pyodide does not support it)
59
+ are available from each [GitHub release](https://github.com/jacksonllee/rustling/releases)
60
+ — look for the ``.whl`` file with ``emscripten`` in the filename.
61
+
62
+ ### Rust
63
+
64
+ ```bash
65
+ cargo add rustling
66
+ ```
67
+
68
+ ## License
69
+
70
+ MIT License