rustling 0.5.0__tar.gz → 0.7.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (133) hide show
  1. rustling-0.7.0/.github/dependabot.yml +16 -0
  2. rustling-0.7.0/.github/workflows/build-wheel.yml +42 -0
  3. {rustling-0.5.0 → rustling-0.7.0}/.github/workflows/python.yml +67 -10
  4. {rustling-0.5.0 → rustling-0.7.0}/.github/workflows/release.yml +49 -12
  5. rustling-0.7.0/.github/workflows/rust.yml +73 -0
  6. {rustling-0.5.0 → rustling-0.7.0}/.gitignore +3 -0
  7. rustling-0.7.0/.pre-commit-config.yaml +35 -0
  8. rustling-0.7.0/CHANGELOG.md +70 -0
  9. {rustling-0.5.0 → rustling-0.7.0}/Cargo.lock +194 -242
  10. {rustling-0.5.0 → rustling-0.7.0}/Cargo.toml +18 -8
  11. {rustling-0.5.0 → rustling-0.7.0}/PKG-INFO +45 -37
  12. rustling-0.7.0/README.md +74 -0
  13. rustling-0.7.0/benchmarks/README.md +89 -0
  14. {rustling-0.5.0 → rustling-0.7.0}/benchmarks/common/__init__.py +2 -1
  15. {rustling-0.5.0 → rustling-0.7.0}/benchmarks/common/data.py +26 -1
  16. {rustling-0.5.0 → rustling-0.7.0}/benchmarks/run_chat.py +46 -146
  17. rustling-0.7.0/benchmarks/run_hmm.py +519 -0
  18. {rustling-0.5.0 → rustling-0.7.0}/benchmarks/run_lm.py +14 -0
  19. rustling-0.5.0/benchmarks/run_tagger.py → rustling-0.7.0/benchmarks/run_perceptron_pos_tagger.py +28 -15
  20. {rustling-0.5.0 → rustling-0.7.0}/benchmarks/run_wordseg.py +9 -64
  21. rustling-0.7.0/benchmarks/update_docs.py +327 -0
  22. rustling-0.7.0/build.rs +62 -0
  23. {rustling-0.5.0 → rustling-0.7.0}/pyproject.toml +14 -6
  24. rustling-0.7.0/python/docs/_static/favicon.ico +0 -0
  25. rustling-0.7.0/python/docs/_static/logo-with-text.svg +15 -0
  26. rustling-0.7.0/python/docs/_templates/copyright.html +1 -0
  27. {rustling-0.5.0 → rustling-0.7.0}/python/docs/api.rst +3 -2
  28. {rustling-0.5.0 → rustling-0.7.0}/python/docs/conf.py +10 -0
  29. rustling-0.7.0/python/docs/hmm.rst +31 -0
  30. rustling-0.7.0/python/docs/index.rst +123 -0
  31. rustling-0.7.0/python/docs/perceptron_pos_tagger.rst +28 -0
  32. {rustling-0.5.0 → rustling-0.7.0}/python/docs/requirements.txt +5 -2
  33. rustling-0.7.0/python/docs/wordseg.rst +74 -0
  34. {rustling-0.5.0 → rustling-0.7.0}/python/rustling/__init__.py +2 -2
  35. {rustling-0.5.0 → rustling-0.7.0}/python/rustling/chat/__init__.pyi +84 -16
  36. rustling-0.7.0/python/rustling/hmm/__init__.py +7 -0
  37. rustling-0.7.0/python/rustling/hmm/__init__.pyi +158 -0
  38. rustling-0.7.0/python/rustling/lm/__init__.py +13 -0
  39. rustling-0.7.0/python/rustling/lm/__init__.pyi +441 -0
  40. {rustling-0.5.0 → rustling-0.7.0}/python/rustling/ngram/__init__.pyi +1 -1
  41. rustling-0.7.0/python/rustling/perceptron_pos_tagger/__init__.py +11 -0
  42. {rustling-0.5.0/python/rustling/tagging → rustling-0.7.0/python/rustling/perceptron_pos_tagger}/__init__.pyi +34 -17
  43. rustling-0.7.0/python/rustling/seq_feature/__init__.py +9 -0
  44. rustling-0.7.0/python/rustling/seq_feature/__init__.pyi +54 -0
  45. {rustling-0.5.0 → rustling-0.7.0}/python/rustling/wordseg/__init__.py +8 -1
  46. rustling-0.7.0/python/rustling/wordseg/__init__.pyi +398 -0
  47. rustling-0.7.0/python/stubtest_allowlist.txt +110 -0
  48. rustling-0.7.0/python/tests/test_benchmarks.py +190 -0
  49. {rustling-0.5.0 → rustling-0.7.0}/python/tests/test_chat/test_chat.py +250 -14
  50. rustling-0.7.0/python/tests/test_hmm.py +229 -0
  51. {rustling-0.5.0/python/tests/test_lm → rustling-0.7.0/python/tests}/test_lm.py +108 -55
  52. rustling-0.5.0/python/tests/test_tagging/test_averaged_perceptron_tagger.py → rustling-0.7.0/python/tests/test_perceptron_pos_tagger.py +54 -35
  53. rustling-0.7.0/python/tests/test_wordseg/test_dag_hmm_segmenter.py +69 -0
  54. rustling-0.7.0/python/tests/test_wordseg/test_hmm_segmenter.py +157 -0
  55. {rustling-0.5.0 → rustling-0.7.0}/python/tests/test_wordseg/test_longest_string_matching.py +36 -0
  56. {rustling-0.5.0 → rustling-0.7.0}/python/tests/test_wordseg/test_random_segmenter.py +31 -7
  57. {rustling-0.5.0 → rustling-0.7.0}/src/chat/clean_utterance.rs +481 -11
  58. {rustling-0.5.0 → rustling-0.7.0}/src/chat/header.rs +85 -140
  59. rustling-0.7.0/src/chat/header_py.rs +312 -0
  60. {rustling-0.5.0 → rustling-0.7.0}/src/chat/ipsyn.rs +4 -0
  61. {rustling-0.5.0 → rustling-0.7.0}/src/chat/mod.rs +22 -6
  62. {rustling-0.5.0 → rustling-0.7.0}/src/chat/reader.rs +1018 -1007
  63. rustling-0.7.0/src/chat/reader_py.rs +1128 -0
  64. {rustling-0.5.0 → rustling-0.7.0}/src/chat/utterance.rs +280 -226
  65. rustling-0.7.0/src/chat/utterance_py.rs +330 -0
  66. rustling-0.7.0/src/chat/validation.rs +1091 -0
  67. rustling-0.7.0/src/hmm/mod.rs +1911 -0
  68. rustling-0.7.0/src/hmm/model.fbs +34 -0
  69. rustling-0.7.0/src/hmm/model_generated.rs +630 -0
  70. rustling-0.7.0/src/hmm/py.rs +254 -0
  71. rustling-0.7.0/src/lib.rs +39 -0
  72. rustling-0.7.0/src/lm/mod.rs +1011 -0
  73. rustling-0.7.0/src/lm/model.fbs +18 -0
  74. rustling-0.7.0/src/lm/model_generated.rs +367 -0
  75. rustling-0.7.0/src/lm/py.rs +370 -0
  76. rustling-0.5.0/src/ngram/counter.rs → rustling-0.7.0/src/ngram/mod.rs +217 -177
  77. rustling-0.7.0/src/ngram/py.rs +204 -0
  78. rustling-0.7.0/src/perceptron_pos_tagger/mod.rs +624 -0
  79. rustling-0.7.0/src/perceptron_pos_tagger/model.fbs +25 -0
  80. rustling-0.7.0/src/perceptron_pos_tagger/model_generated.rs +483 -0
  81. rustling-0.7.0/src/perceptron_pos_tagger/py.rs +144 -0
  82. rustling-0.7.0/src/persistence.rs +115 -0
  83. rustling-0.7.0/src/prelude.rs +15 -0
  84. rustling-0.7.0/src/seq_feature/mod.rs +627 -0
  85. rustling-0.7.0/src/seq_feature/py.rs +74 -0
  86. {rustling-0.5.0 → rustling-0.7.0}/src/trie.rs +47 -0
  87. rustling-0.7.0/src/wordseg/bmes.rs +236 -0
  88. rustling-0.7.0/src/wordseg/dag_hmm.rs +515 -0
  89. rustling-0.7.0/src/wordseg/dag_hmm_py.rs +76 -0
  90. rustling-0.7.0/src/wordseg/hmm.rs +471 -0
  91. rustling-0.7.0/src/wordseg/hmm_py.rs +99 -0
  92. rustling-0.7.0/src/wordseg/longest_string_matching/mod.rs +358 -0
  93. rustling-0.7.0/src/wordseg/longest_string_matching/model.fbs +8 -0
  94. rustling-0.7.0/src/wordseg/longest_string_matching/model_generated.rs +200 -0
  95. rustling-0.7.0/src/wordseg/longest_string_matching/py.rs +109 -0
  96. rustling-0.7.0/src/wordseg/mod.rs +141 -0
  97. {rustling-0.5.0 → rustling-0.7.0}/src/wordseg/random_segmenter.rs +61 -95
  98. rustling-0.7.0/src/wordseg/random_segmenter_py.rs +68 -0
  99. rustling-0.5.0/.github/workflows/rust.yml +0 -43
  100. rustling-0.5.0/AGENTS.md +0 -82
  101. rustling-0.5.0/CLAUDE.md +0 -82
  102. rustling-0.5.0/README.md +0 -67
  103. rustling-0.5.0/benchmarks/README.md +0 -252
  104. rustling-0.5.0/python/docs/index.rst +0 -98
  105. rustling-0.5.0/python/docs/tagging.rst +0 -26
  106. rustling-0.5.0/python/docs/wordseg.rst +0 -36
  107. rustling-0.5.0/python/rustling/lm/__init__.py +0 -45
  108. rustling-0.5.0/python/rustling/lm/__init__.pyi +0 -182
  109. rustling-0.5.0/python/rustling/tagging/__init__.py +0 -11
  110. rustling-0.5.0/python/rustling/wordseg/__init__.pyi +0 -92
  111. rustling-0.5.0/src/lib.rs +0 -24
  112. rustling-0.5.0/src/lm/mod.rs +0 -33
  113. rustling-0.5.0/src/lm/model.rs +0 -666
  114. rustling-0.5.0/src/ngram/mod.rs +0 -18
  115. rustling-0.5.0/src/tagging/averaged_perceptron_tagger.rs +0 -674
  116. rustling-0.5.0/src/tagging/mod.rs +0 -47
  117. rustling-0.5.0/src/wordseg/longest_string_matching.rs +0 -224
  118. rustling-0.5.0/src/wordseg/mod.rs +0 -43
  119. {rustling-0.5.0 → rustling-0.7.0}/.flake8 +0 -0
  120. {rustling-0.5.0 → rustling-0.7.0}/.gitattributes +0 -0
  121. {rustling-0.5.0 → rustling-0.7.0}/.readthedocs.yaml +0 -0
  122. {rustling-0.5.0 → rustling-0.7.0}/LICENSE.md +0 -0
  123. {rustling-0.5.0 → rustling-0.7.0}/python/docs/Makefile +0 -0
  124. {rustling-0.5.0 → rustling-0.7.0}/python/docs/README.md +0 -0
  125. {rustling-0.5.0 → rustling-0.7.0}/python/docs/chat.rst +0 -0
  126. {rustling-0.5.0 → rustling-0.7.0}/python/docs/lm.rst +0 -0
  127. {rustling-0.5.0 → rustling-0.7.0}/python/docs/make.bat +0 -0
  128. {rustling-0.5.0 → rustling-0.7.0}/python/docs/ngram.rst +0 -0
  129. {rustling-0.5.0 → rustling-0.7.0}/python/rustling/chat/__init__.py +0 -0
  130. {rustling-0.5.0 → rustling-0.7.0}/python/rustling/ngram/__init__.py +0 -0
  131. {rustling-0.5.0 → rustling-0.7.0}/python/rustling/py.typed +0 -0
  132. {rustling-0.5.0 → rustling-0.7.0}/python/tests/test_chat/conftest.py +0 -0
  133. {rustling-0.5.0/python/tests/test_ngram → rustling-0.7.0/python/tests}/test_ngram.py +0 -0
@@ -0,0 +1,16 @@
1
+ version: 2
2
+ updates:
3
+ - package-ecosystem: "cargo"
4
+ directory: "/"
5
+ schedule:
6
+ interval: "weekly"
7
+
8
+ - package-ecosystem: "pip"
9
+ directory: "/"
10
+ schedule:
11
+ interval: "weekly"
12
+
13
+ - package-ecosystem: "github-actions"
14
+ directory: "/"
15
+ schedule:
16
+ interval: "weekly"
@@ -0,0 +1,42 @@
1
+ name: Build Linux x86_64 wheel
2
+
3
+ on:
4
+ workflow_dispatch:
5
+ inputs:
6
+ git_ref:
7
+ description: "Git ref to build from (tag, branch, or commit)"
8
+ required: true
9
+ default: "v0.6.0"
10
+
11
+ env:
12
+ FLATC_VERSION: "25.12.19"
13
+
14
+ jobs:
15
+ build:
16
+ name: Build manylinux x86_64 wheel
17
+ runs-on: ubuntu-latest
18
+ steps:
19
+ - uses: actions/checkout@v6
20
+ with:
21
+ ref: ${{ github.event.inputs.git_ref }}
22
+
23
+ - name: Build wheels
24
+ uses: PyO3/maturin-action@v1
25
+ with:
26
+ target: x86_64
27
+ args: --release --out dist -i python3.10
28
+ sccache: "true"
29
+ manylinux: auto
30
+ before-script-linux: |
31
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
32
+ command -v unzip || yum install -y unzip 2>/dev/null || (apt-get update -qq && apt-get install -y -qq unzip)
33
+ unzip -q flatc.zip
34
+ rm flatc.zip
35
+ chmod +x flatc
36
+ mv flatc /usr/local/bin/
37
+
38
+ - name: Upload wheel
39
+ uses: actions/upload-artifact@v7
40
+ with:
41
+ name: wheel-linux-x86_64
42
+ path: dist
@@ -6,16 +6,24 @@ on:
6
6
 
7
7
  env:
8
8
  PYTHON_VERSION: "3.14"
9
+ FLATC_VERSION: "25.12.19"
9
10
 
10
11
  jobs:
11
12
  lint:
12
13
  name: Lint
13
14
  runs-on: ubuntu-latest
14
15
  steps:
15
- - uses: actions/checkout@v4
16
+ - uses: actions/checkout@v6
17
+
18
+ - name: Install FlatBuffers compiler
19
+ run: |
20
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
21
+ unzip -q flatc.zip
22
+ chmod +x flatc
23
+ sudo mv flatc /usr/local/bin/
16
24
 
17
25
  - name: Install uv
18
- uses: astral-sh/setup-uv@v5
26
+ uses: astral-sh/setup-uv@v7
19
27
 
20
28
  - name: Set up Python
21
29
  run: uv python install ${{ env.PYTHON_VERSION }}
@@ -36,10 +44,17 @@ jobs:
36
44
  matrix:
37
45
  python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
38
46
  steps:
39
- - uses: actions/checkout@v4
47
+ - uses: actions/checkout@v6
48
+
49
+ - name: Install FlatBuffers compiler
50
+ run: |
51
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
52
+ unzip -q flatc.zip
53
+ chmod +x flatc
54
+ sudo mv flatc /usr/local/bin/
40
55
 
41
56
  - name: Install uv
42
- uses: astral-sh/setup-uv@v5
57
+ uses: astral-sh/setup-uv@v7
43
58
 
44
59
  - name: Set up Python ${{ matrix.python-version }}
45
60
  run: uv python install ${{ matrix.python-version }}
@@ -59,12 +74,28 @@ jobs:
59
74
  PRIVATE_TEST_REPO: ${{ secrets.PRIVATE_TEST_REPO }}
60
75
  run: uv run pytest python/tests/ -v
61
76
 
77
+ - name: Verify stub files
78
+ if: matrix.python-version == '3.14'
79
+ run: uv run python -m mypy.stubtest rustling --concise --ignore-missing-stub --allowlist python/stubtest_allowlist.txt
80
+
81
+ audit:
82
+ name: Audit
83
+ runs-on: ubuntu-latest
84
+ steps:
85
+ - uses: actions/checkout@v6
86
+
87
+ - name: Install uv
88
+ uses: astral-sh/setup-uv@v7
89
+
90
+ - name: Audit dev dependencies
91
+ run: uv export --no-hashes --group dev | uvx pip-audit -r /dev/stdin --desc
92
+
62
93
  # This job should be the same as the "python-sdist" job in release.yml, but without the artifact upload.
63
94
  sdist:
64
95
  name: Build source distribution (release readiness check)
65
96
  runs-on: ubuntu-latest
66
97
  steps:
67
- - uses: actions/checkout@v4
98
+ - uses: actions/checkout@v6
68
99
 
69
100
  - name: Build sdist
70
101
  uses: PyO3/maturin-action@v1
@@ -96,15 +127,34 @@ jobs:
96
127
  - os: windows-latest
97
128
  target: x86_64
98
129
  steps:
99
- - uses: actions/checkout@v4
130
+ - uses: actions/checkout@v6
131
+
132
+ - name: Install FlatBuffers compiler (macOS)
133
+ if: runner.os == 'macOS'
134
+ run: brew install flatbuffers
135
+
136
+ - name: Install FlatBuffers compiler (Windows)
137
+ if: runner.os == 'Windows'
138
+ shell: pwsh
139
+ run: |
140
+ $url = "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Windows.flatc.binary.zip"
141
+ Invoke-WebRequest -Uri $url -OutFile flatc.zip
142
+ Expand-Archive -Path flatc.zip -DestinationPath flatc_bin
143
+ echo "$PWD\flatc_bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
100
144
 
101
145
  - name: Build wheels
102
146
  uses: PyO3/maturin-action@v1
103
147
  with:
104
148
  target: ${{ matrix.target }}
105
- args: --release --out dist
149
+ args: --release --out dist -i python3.10
106
150
  sccache: "true"
107
151
  manylinux: auto
152
+ before-script-linux: |
153
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
154
+ command -v unzip || yum install -y unzip 2>/dev/null || (apt-get update -qq && apt-get install -y -qq unzip)
155
+ unzip -q flatc.zip
156
+ chmod +x flatc
157
+ mv flatc /usr/local/bin/
108
158
 
109
159
  # This job should be similar to the "python-wasm-wheel" job in release.yml, but without the artifact upload.
110
160
  # Not using maturin-action for WASM because it doesn't set up Rust nightly, Emscripten SDK, or wasm-eh sysroot.
@@ -112,7 +162,14 @@ jobs:
112
162
  name: Build wasm32-unknown-emscripten wheel (release readiness check)
113
163
  runs-on: ubuntu-latest
114
164
  steps:
115
- - uses: actions/checkout@v4
165
+ - uses: actions/checkout@v6
166
+
167
+ - name: Install FlatBuffers compiler
168
+ run: |
169
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
170
+ unzip -q flatc.zip
171
+ chmod +x flatc
172
+ sudo mv flatc /usr/local/bin/
116
173
 
117
174
  - name: Install Rust nightly
118
175
  uses: dtolnay/rust-toolchain@master
@@ -124,7 +181,7 @@ jobs:
124
181
  run: |
125
182
  TOOLCHAIN_ROOT=$(rustup run nightly-2025-06-27 rustc --print sysroot)
126
183
  RUSTLIB=$TOOLCHAIN_ROOT/lib/rustlib
127
- wget -q https://github.com/pyodide/rust-emscripten-wasm-eh-sysroot/releases/download/emcc-4.0.9_nightly-2025-06-27/emcc-4.0.9_nightly-2025-06-27.tar.bz2
184
+ curl -sL https://github.com/pyodide/rust-emscripten-wasm-eh-sysroot/releases/download/emcc-4.0.9_nightly-2025-06-27/emcc-4.0.9_nightly-2025-06-27.tar.bz2 -o emcc-4.0.9_nightly-2025-06-27.tar.bz2
128
185
  tar -xf emcc-4.0.9_nightly-2025-06-27.tar.bz2 --directory=$RUSTLIB
129
186
 
130
187
  - name: Cache Rust build artifacts
@@ -146,7 +203,7 @@ jobs:
146
203
  curl -sL https://raw.githubusercontent.com/pyodide/pyodide/refs/heads/main/emsdk/patches/0002-Don-t-check-exports-for-being-valid-C-C-identifiers-.patch | git -C "$EMSCRIPTEN_DIR" apply
147
204
 
148
205
  - name: Install uv
149
- uses: astral-sh/setup-uv@v5
206
+ uses: astral-sh/setup-uv@v7
150
207
 
151
208
  - name: Build wasm wheel
152
209
  env:
@@ -4,6 +4,9 @@ on:
4
4
  release:
5
5
  types: [published]
6
6
 
7
+ env:
8
+ FLATC_VERSION: "25.12.19"
9
+
7
10
  permissions:
8
11
  contents: read
9
12
 
@@ -12,7 +15,7 @@ jobs:
12
15
  name: Python - Build source distribution
13
16
  runs-on: ubuntu-latest
14
17
  steps:
15
- - uses: actions/checkout@v4
18
+ - uses: actions/checkout@v6
16
19
 
17
20
  - name: Build sdist
18
21
  uses: PyO3/maturin-action@v1
@@ -21,7 +24,7 @@ jobs:
21
24
  args: --out dist
22
25
 
23
26
  - name: Upload sdist
24
- uses: actions/upload-artifact@v4
27
+ uses: actions/upload-artifact@v7
25
28
  with:
26
29
  name: wheels-sdist
27
30
  path: dist
@@ -49,18 +52,37 @@ jobs:
49
52
  - os: windows-latest
50
53
  target: x86_64
51
54
  steps:
52
- - uses: actions/checkout@v4
55
+ - uses: actions/checkout@v6
56
+
57
+ - name: Install FlatBuffers compiler (macOS)
58
+ if: runner.os == 'macOS'
59
+ run: brew install flatbuffers
60
+
61
+ - name: Install FlatBuffers compiler (Windows)
62
+ if: runner.os == 'Windows'
63
+ shell: pwsh
64
+ run: |
65
+ $url = "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Windows.flatc.binary.zip"
66
+ Invoke-WebRequest -Uri $url -OutFile flatc.zip
67
+ Expand-Archive -Path flatc.zip -DestinationPath flatc_bin
68
+ echo "$PWD\flatc_bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
53
69
 
54
70
  - name: Build wheels
55
71
  uses: PyO3/maturin-action@v1
56
72
  with:
57
73
  target: ${{ matrix.target }}
58
- args: --release --out dist
74
+ args: --release --out dist -i python3.10
59
75
  sccache: "true"
60
76
  manylinux: auto
77
+ before-script-linux: |
78
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
79
+ command -v unzip || yum install -y unzip 2>/dev/null || (apt-get update -qq && apt-get install -y -qq unzip)
80
+ unzip -q flatc.zip
81
+ chmod +x flatc
82
+ mv flatc /usr/local/bin/
61
83
 
62
84
  - name: Upload wheels
63
- uses: actions/upload-artifact@v4
85
+ uses: actions/upload-artifact@v7
64
86
  with:
65
87
  name: wheels-${{ matrix.os }}-${{ matrix.target }}
66
88
  path: dist
@@ -70,7 +92,14 @@ jobs:
70
92
  name: Python - Build wasm32-unknown-emscripten wheel
71
93
  runs-on: ubuntu-latest
72
94
  steps:
73
- - uses: actions/checkout@v4
95
+ - uses: actions/checkout@v6
96
+
97
+ - name: Install FlatBuffers compiler
98
+ run: |
99
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
100
+ unzip -q flatc.zip
101
+ chmod +x flatc
102
+ sudo mv flatc /usr/local/bin/
74
103
 
75
104
  - name: Install Rust nightly
76
105
  uses: dtolnay/rust-toolchain@master
@@ -82,7 +111,7 @@ jobs:
82
111
  run: |
83
112
  TOOLCHAIN_ROOT=$(rustup run nightly-2025-06-27 rustc --print sysroot)
84
113
  RUSTLIB=$TOOLCHAIN_ROOT/lib/rustlib
85
- wget -q https://github.com/pyodide/rust-emscripten-wasm-eh-sysroot/releases/download/emcc-4.0.9_nightly-2025-06-27/emcc-4.0.9_nightly-2025-06-27.tar.bz2
114
+ curl -sL https://github.com/pyodide/rust-emscripten-wasm-eh-sysroot/releases/download/emcc-4.0.9_nightly-2025-06-27/emcc-4.0.9_nightly-2025-06-27.tar.bz2 -o emcc-4.0.9_nightly-2025-06-27.tar.bz2
86
115
  tar -xf emcc-4.0.9_nightly-2025-06-27.tar.bz2 --directory=$RUSTLIB
87
116
 
88
117
  - name: Cache Rust build artifacts
@@ -104,7 +133,7 @@ jobs:
104
133
  curl -sL https://raw.githubusercontent.com/pyodide/pyodide/refs/heads/main/emsdk/patches/0002-Don-t-check-exports-for-being-valid-C-C-identifiers-.patch | git -C "$EMSCRIPTEN_DIR" apply
105
134
 
106
135
  - name: Install uv
107
- uses: astral-sh/setup-uv@v5
136
+ uses: astral-sh/setup-uv@v7
108
137
 
109
138
  - name: Build wasm wheel
110
139
  env:
@@ -113,7 +142,7 @@ jobs:
113
142
  run: uvx maturin build --release --target wasm32-unknown-emscripten --out dist -i 3.13 --no-default-features --features extension-module
114
143
 
115
144
  - name: Upload wasm wheel
116
- uses: actions/upload-artifact@v4
145
+ uses: actions/upload-artifact@v7
117
146
  with:
118
147
  # Not "wheels-*" to avoid being picked up by the PyPI publish step,
119
148
  # which rejects the emscripten platform tag.
@@ -131,7 +160,7 @@ jobs:
131
160
  id-token: write
132
161
  steps:
133
162
  - name: Download all artifacts
134
- uses: actions/download-artifact@v4
163
+ uses: actions/download-artifact@v8
135
164
  with:
136
165
  path: dist
137
166
  pattern: wheels-*
@@ -148,7 +177,7 @@ jobs:
148
177
  contents: write
149
178
  steps:
150
179
  - name: Download all artifacts
151
- uses: actions/download-artifact@v4
180
+ uses: actions/download-artifact@v8
152
181
  with:
153
182
  path: dist
154
183
  merge-multiple: true
@@ -166,7 +195,15 @@ jobs:
166
195
  permissions:
167
196
  id-token: write
168
197
  steps:
169
- - uses: actions/checkout@v4
198
+ - uses: actions/checkout@v6
199
+
200
+ - name: Install FlatBuffers compiler
201
+ run: |
202
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
203
+ unzip -q flatc.zip
204
+ rm flatc.zip
205
+ chmod +x flatc
206
+ sudo mv flatc /usr/local/bin/
170
207
 
171
208
  - name: Install Rust toolchain
172
209
  uses: dtolnay/rust-toolchain@stable
@@ -0,0 +1,73 @@
1
+ name: Rust
2
+
3
+ on:
4
+ push:
5
+ pull_request:
6
+
7
+ env:
8
+ CARGO_TERM_COLOR: always
9
+ FLATC_VERSION: "25.12.19"
10
+
11
+ jobs:
12
+ lint:
13
+ name: Lint
14
+ runs-on: ubuntu-latest
15
+ steps:
16
+ - uses: actions/checkout@v6
17
+
18
+ - name: Install FlatBuffers compiler
19
+ run: |
20
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
21
+ unzip -q flatc.zip
22
+ chmod +x flatc
23
+ sudo mv flatc /usr/local/bin/
24
+
25
+ - name: Install Rust toolchain
26
+ uses: dtolnay/rust-toolchain@stable
27
+ with:
28
+ components: rustfmt, clippy
29
+
30
+ - name: Check formatting
31
+ run: cargo fmt --all -- --check
32
+
33
+ - name: Run clippy
34
+ run: cargo clippy --all-targets --all-features -- -D warnings
35
+
36
+ test:
37
+ name: Test
38
+ runs-on: ubuntu-latest
39
+ steps:
40
+ - uses: actions/checkout@v6
41
+
42
+ - name: Install FlatBuffers compiler
43
+ run: |
44
+ curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
45
+ unzip -q flatc.zip
46
+ chmod +x flatc
47
+ sudo mv flatc /usr/local/bin/
48
+
49
+ - name: Set up Python
50
+ uses: actions/setup-python@v6
51
+ with:
52
+ python-version: "3.14"
53
+
54
+ - name: Install Rust toolchain
55
+ uses: dtolnay/rust-toolchain@stable
56
+
57
+ - name: Run tests
58
+ run: cargo test
59
+
60
+ audit:
61
+ name: Audit
62
+ runs-on: ubuntu-latest
63
+ steps:
64
+ - uses: actions/checkout@v6
65
+
66
+ - name: Install Rust toolchain
67
+ uses: dtolnay/rust-toolchain@stable
68
+
69
+ - name: Install cargo-audit
70
+ run: cargo install cargo-audit
71
+
72
+ - name: Run cargo audit
73
+ run: cargo audit
@@ -71,6 +71,9 @@ python/docs/_build/
71
71
  # Pyenv
72
72
  .python-version
73
73
 
74
+ # Benchmark results
75
+ benchmarks/.results/
76
+
74
77
  # This is a library, not an application.
75
78
  Cargo.lock
76
79
  uv.lock
@@ -0,0 +1,35 @@
1
+ repos:
2
+ - repo: local
3
+ hooks:
4
+ - id: cargo-fmt
5
+ name: cargo fmt
6
+ entry: cargo fmt --all -- --check
7
+ language: system
8
+ types: [rust]
9
+ pass_filenames: false
10
+ - id: cargo-clippy
11
+ name: cargo clippy
12
+ entry: cargo clippy --all-targets --all-features -- -D warnings
13
+ language: system
14
+ types: [rust]
15
+ pass_filenames: false
16
+ - repo: https://github.com/psf/black-pre-commit-mirror
17
+ rev: 26.3.0
18
+ hooks:
19
+ - id: black
20
+ files: ^(python|benchmarks)/
21
+ - repo: https://github.com/pycqa/flake8
22
+ rev: 7.3.0
23
+ hooks:
24
+ - id: flake8
25
+ files: ^(python|benchmarks)/
26
+ - repo: local
27
+ hooks:
28
+ - id: mypy
29
+ name: mypy
30
+ entry: uvx mypy
31
+ language: system
32
+ types: [python]
33
+ files: ^python/rustling/
34
+ pass_filenames: false
35
+ args: [python/rustling/]
@@ -0,0 +1,70 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [Unreleased]
9
+
10
+ ## [0.7.0] - 2026-03-14
11
+
12
+ ### Added
13
+
14
+ - Word segmentation:
15
+ * Added `score` method for the HMM and DAG-HMM segmenters.
16
+ * `predict` method can optionally output offsets for the (start, end) indices
17
+ of segmented words compared to the original string.
18
+ - CHAT parsing: Support custom tier names other than the standard %mor and %gra.
19
+ - Python model classes are now subclassable.
20
+
21
+ ### Changed
22
+
23
+ - Ngram counters: `Ngrams.most_common` now sorts tuples lexicographically
24
+ when counts are tied.
25
+ - CHAT parsing:
26
+ * If a date is available at `Headers`'s `date`,
27
+ it's now a Python `datetime.date` object instead of a string.
28
+ * In handling the main tier transcription for creating `Token` objects:
29
+ - Special form markers suffixed with "@" are now stripped.
30
+ - Words that have partiallly parenthetical material have the parentheses
31
+ removed, e.g., (un)til -> until, sit(ting) -> sitting.
32
+ * Renamed the `CHAT.raw` attribute to `CHAT.audible` for a best-effort,
33
+ audibly faithful transcription string, to facilitate automatic speech recognition,
34
+ forced alignment, etc.
35
+ * A subset of the testchat/bad dataset is now used to validate CHAT data format.
36
+ - Refactored core Rust code so that Rust-only consumers no longer need PyO3/Python.
37
+
38
+ ## [0.6.0] - 2026-03-05
39
+
40
+ ### Added
41
+
42
+ - Hidden Markov Model (HMM)
43
+ - Word segmentation: Added DAG-HMM word segmenter
44
+ - CHAT parsing: Added `from_utterances` method
45
+
46
+ ### Changed
47
+
48
+ - Models are now persisted as a zstd-compressed FlatBuffers binary.
49
+
50
+ ## [0.5.0] - 2026-02-18
51
+
52
+ ### Added
53
+
54
+ - CHAT parsing for TalkBank and CHILDES data
55
+
56
+ ## [0.4.0] - 2026-02-08
57
+
58
+ ### Added
59
+
60
+ - N-grams and language models
61
+
62
+ ## [0.3.0] - 2026-02-06
63
+
64
+ ### Added
65
+
66
+ - Averaged perceptron tagger
67
+
68
+ ## [0.2.0] - 2026-02-04
69
+
70
+ - Initial release, with longest string matching and random segmenter for word segmentation