rustling 0.5.0__tar.gz → 0.7.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- rustling-0.7.0/.github/dependabot.yml +16 -0
- rustling-0.7.0/.github/workflows/build-wheel.yml +42 -0
- {rustling-0.5.0 → rustling-0.7.0}/.github/workflows/python.yml +67 -10
- {rustling-0.5.0 → rustling-0.7.0}/.github/workflows/release.yml +49 -12
- rustling-0.7.0/.github/workflows/rust.yml +73 -0
- {rustling-0.5.0 → rustling-0.7.0}/.gitignore +3 -0
- rustling-0.7.0/.pre-commit-config.yaml +35 -0
- rustling-0.7.0/CHANGELOG.md +70 -0
- {rustling-0.5.0 → rustling-0.7.0}/Cargo.lock +194 -242
- {rustling-0.5.0 → rustling-0.7.0}/Cargo.toml +18 -8
- {rustling-0.5.0 → rustling-0.7.0}/PKG-INFO +45 -37
- rustling-0.7.0/README.md +74 -0
- rustling-0.7.0/benchmarks/README.md +89 -0
- {rustling-0.5.0 → rustling-0.7.0}/benchmarks/common/__init__.py +2 -1
- {rustling-0.5.0 → rustling-0.7.0}/benchmarks/common/data.py +26 -1
- {rustling-0.5.0 → rustling-0.7.0}/benchmarks/run_chat.py +46 -146
- rustling-0.7.0/benchmarks/run_hmm.py +519 -0
- {rustling-0.5.0 → rustling-0.7.0}/benchmarks/run_lm.py +14 -0
- rustling-0.5.0/benchmarks/run_tagger.py → rustling-0.7.0/benchmarks/run_perceptron_pos_tagger.py +28 -15
- {rustling-0.5.0 → rustling-0.7.0}/benchmarks/run_wordseg.py +9 -64
- rustling-0.7.0/benchmarks/update_docs.py +327 -0
- rustling-0.7.0/build.rs +62 -0
- {rustling-0.5.0 → rustling-0.7.0}/pyproject.toml +14 -6
- rustling-0.7.0/python/docs/_static/favicon.ico +0 -0
- rustling-0.7.0/python/docs/_static/logo-with-text.svg +15 -0
- rustling-0.7.0/python/docs/_templates/copyright.html +1 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/docs/api.rst +3 -2
- {rustling-0.5.0 → rustling-0.7.0}/python/docs/conf.py +10 -0
- rustling-0.7.0/python/docs/hmm.rst +31 -0
- rustling-0.7.0/python/docs/index.rst +123 -0
- rustling-0.7.0/python/docs/perceptron_pos_tagger.rst +28 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/docs/requirements.txt +5 -2
- rustling-0.7.0/python/docs/wordseg.rst +74 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/rustling/__init__.py +2 -2
- {rustling-0.5.0 → rustling-0.7.0}/python/rustling/chat/__init__.pyi +84 -16
- rustling-0.7.0/python/rustling/hmm/__init__.py +7 -0
- rustling-0.7.0/python/rustling/hmm/__init__.pyi +158 -0
- rustling-0.7.0/python/rustling/lm/__init__.py +13 -0
- rustling-0.7.0/python/rustling/lm/__init__.pyi +441 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/rustling/ngram/__init__.pyi +1 -1
- rustling-0.7.0/python/rustling/perceptron_pos_tagger/__init__.py +11 -0
- {rustling-0.5.0/python/rustling/tagging → rustling-0.7.0/python/rustling/perceptron_pos_tagger}/__init__.pyi +34 -17
- rustling-0.7.0/python/rustling/seq_feature/__init__.py +9 -0
- rustling-0.7.0/python/rustling/seq_feature/__init__.pyi +54 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/rustling/wordseg/__init__.py +8 -1
- rustling-0.7.0/python/rustling/wordseg/__init__.pyi +398 -0
- rustling-0.7.0/python/stubtest_allowlist.txt +110 -0
- rustling-0.7.0/python/tests/test_benchmarks.py +190 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/tests/test_chat/test_chat.py +250 -14
- rustling-0.7.0/python/tests/test_hmm.py +229 -0
- {rustling-0.5.0/python/tests/test_lm → rustling-0.7.0/python/tests}/test_lm.py +108 -55
- rustling-0.5.0/python/tests/test_tagging/test_averaged_perceptron_tagger.py → rustling-0.7.0/python/tests/test_perceptron_pos_tagger.py +54 -35
- rustling-0.7.0/python/tests/test_wordseg/test_dag_hmm_segmenter.py +69 -0
- rustling-0.7.0/python/tests/test_wordseg/test_hmm_segmenter.py +157 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/tests/test_wordseg/test_longest_string_matching.py +36 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/tests/test_wordseg/test_random_segmenter.py +31 -7
- {rustling-0.5.0 → rustling-0.7.0}/src/chat/clean_utterance.rs +481 -11
- {rustling-0.5.0 → rustling-0.7.0}/src/chat/header.rs +85 -140
- rustling-0.7.0/src/chat/header_py.rs +312 -0
- {rustling-0.5.0 → rustling-0.7.0}/src/chat/ipsyn.rs +4 -0
- {rustling-0.5.0 → rustling-0.7.0}/src/chat/mod.rs +22 -6
- {rustling-0.5.0 → rustling-0.7.0}/src/chat/reader.rs +1018 -1007
- rustling-0.7.0/src/chat/reader_py.rs +1128 -0
- {rustling-0.5.0 → rustling-0.7.0}/src/chat/utterance.rs +280 -226
- rustling-0.7.0/src/chat/utterance_py.rs +330 -0
- rustling-0.7.0/src/chat/validation.rs +1091 -0
- rustling-0.7.0/src/hmm/mod.rs +1911 -0
- rustling-0.7.0/src/hmm/model.fbs +34 -0
- rustling-0.7.0/src/hmm/model_generated.rs +630 -0
- rustling-0.7.0/src/hmm/py.rs +254 -0
- rustling-0.7.0/src/lib.rs +39 -0
- rustling-0.7.0/src/lm/mod.rs +1011 -0
- rustling-0.7.0/src/lm/model.fbs +18 -0
- rustling-0.7.0/src/lm/model_generated.rs +367 -0
- rustling-0.7.0/src/lm/py.rs +370 -0
- rustling-0.5.0/src/ngram/counter.rs → rustling-0.7.0/src/ngram/mod.rs +217 -177
- rustling-0.7.0/src/ngram/py.rs +204 -0
- rustling-0.7.0/src/perceptron_pos_tagger/mod.rs +624 -0
- rustling-0.7.0/src/perceptron_pos_tagger/model.fbs +25 -0
- rustling-0.7.0/src/perceptron_pos_tagger/model_generated.rs +483 -0
- rustling-0.7.0/src/perceptron_pos_tagger/py.rs +144 -0
- rustling-0.7.0/src/persistence.rs +115 -0
- rustling-0.7.0/src/prelude.rs +15 -0
- rustling-0.7.0/src/seq_feature/mod.rs +627 -0
- rustling-0.7.0/src/seq_feature/py.rs +74 -0
- {rustling-0.5.0 → rustling-0.7.0}/src/trie.rs +47 -0
- rustling-0.7.0/src/wordseg/bmes.rs +236 -0
- rustling-0.7.0/src/wordseg/dag_hmm.rs +515 -0
- rustling-0.7.0/src/wordseg/dag_hmm_py.rs +76 -0
- rustling-0.7.0/src/wordseg/hmm.rs +471 -0
- rustling-0.7.0/src/wordseg/hmm_py.rs +99 -0
- rustling-0.7.0/src/wordseg/longest_string_matching/mod.rs +358 -0
- rustling-0.7.0/src/wordseg/longest_string_matching/model.fbs +8 -0
- rustling-0.7.0/src/wordseg/longest_string_matching/model_generated.rs +200 -0
- rustling-0.7.0/src/wordseg/longest_string_matching/py.rs +109 -0
- rustling-0.7.0/src/wordseg/mod.rs +141 -0
- {rustling-0.5.0 → rustling-0.7.0}/src/wordseg/random_segmenter.rs +61 -95
- rustling-0.7.0/src/wordseg/random_segmenter_py.rs +68 -0
- rustling-0.5.0/.github/workflows/rust.yml +0 -43
- rustling-0.5.0/AGENTS.md +0 -82
- rustling-0.5.0/CLAUDE.md +0 -82
- rustling-0.5.0/README.md +0 -67
- rustling-0.5.0/benchmarks/README.md +0 -252
- rustling-0.5.0/python/docs/index.rst +0 -98
- rustling-0.5.0/python/docs/tagging.rst +0 -26
- rustling-0.5.0/python/docs/wordseg.rst +0 -36
- rustling-0.5.0/python/rustling/lm/__init__.py +0 -45
- rustling-0.5.0/python/rustling/lm/__init__.pyi +0 -182
- rustling-0.5.0/python/rustling/tagging/__init__.py +0 -11
- rustling-0.5.0/python/rustling/wordseg/__init__.pyi +0 -92
- rustling-0.5.0/src/lib.rs +0 -24
- rustling-0.5.0/src/lm/mod.rs +0 -33
- rustling-0.5.0/src/lm/model.rs +0 -666
- rustling-0.5.0/src/ngram/mod.rs +0 -18
- rustling-0.5.0/src/tagging/averaged_perceptron_tagger.rs +0 -674
- rustling-0.5.0/src/tagging/mod.rs +0 -47
- rustling-0.5.0/src/wordseg/longest_string_matching.rs +0 -224
- rustling-0.5.0/src/wordseg/mod.rs +0 -43
- {rustling-0.5.0 → rustling-0.7.0}/.flake8 +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/.gitattributes +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/.readthedocs.yaml +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/LICENSE.md +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/docs/Makefile +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/docs/README.md +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/docs/chat.rst +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/docs/lm.rst +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/docs/make.bat +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/docs/ngram.rst +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/rustling/chat/__init__.py +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/rustling/ngram/__init__.py +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/rustling/py.typed +0 -0
- {rustling-0.5.0 → rustling-0.7.0}/python/tests/test_chat/conftest.py +0 -0
- {rustling-0.5.0/python/tests/test_ngram → rustling-0.7.0/python/tests}/test_ngram.py +0 -0
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
version: 2
|
|
2
|
+
updates:
|
|
3
|
+
- package-ecosystem: "cargo"
|
|
4
|
+
directory: "/"
|
|
5
|
+
schedule:
|
|
6
|
+
interval: "weekly"
|
|
7
|
+
|
|
8
|
+
- package-ecosystem: "pip"
|
|
9
|
+
directory: "/"
|
|
10
|
+
schedule:
|
|
11
|
+
interval: "weekly"
|
|
12
|
+
|
|
13
|
+
- package-ecosystem: "github-actions"
|
|
14
|
+
directory: "/"
|
|
15
|
+
schedule:
|
|
16
|
+
interval: "weekly"
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
name: Build Linux x86_64 wheel
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
workflow_dispatch:
|
|
5
|
+
inputs:
|
|
6
|
+
git_ref:
|
|
7
|
+
description: "Git ref to build from (tag, branch, or commit)"
|
|
8
|
+
required: true
|
|
9
|
+
default: "v0.6.0"
|
|
10
|
+
|
|
11
|
+
env:
|
|
12
|
+
FLATC_VERSION: "25.12.19"
|
|
13
|
+
|
|
14
|
+
jobs:
|
|
15
|
+
build:
|
|
16
|
+
name: Build manylinux x86_64 wheel
|
|
17
|
+
runs-on: ubuntu-latest
|
|
18
|
+
steps:
|
|
19
|
+
- uses: actions/checkout@v6
|
|
20
|
+
with:
|
|
21
|
+
ref: ${{ github.event.inputs.git_ref }}
|
|
22
|
+
|
|
23
|
+
- name: Build wheels
|
|
24
|
+
uses: PyO3/maturin-action@v1
|
|
25
|
+
with:
|
|
26
|
+
target: x86_64
|
|
27
|
+
args: --release --out dist -i python3.10
|
|
28
|
+
sccache: "true"
|
|
29
|
+
manylinux: auto
|
|
30
|
+
before-script-linux: |
|
|
31
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
32
|
+
command -v unzip || yum install -y unzip 2>/dev/null || (apt-get update -qq && apt-get install -y -qq unzip)
|
|
33
|
+
unzip -q flatc.zip
|
|
34
|
+
rm flatc.zip
|
|
35
|
+
chmod +x flatc
|
|
36
|
+
mv flatc /usr/local/bin/
|
|
37
|
+
|
|
38
|
+
- name: Upload wheel
|
|
39
|
+
uses: actions/upload-artifact@v7
|
|
40
|
+
with:
|
|
41
|
+
name: wheel-linux-x86_64
|
|
42
|
+
path: dist
|
|
@@ -6,16 +6,24 @@ on:
|
|
|
6
6
|
|
|
7
7
|
env:
|
|
8
8
|
PYTHON_VERSION: "3.14"
|
|
9
|
+
FLATC_VERSION: "25.12.19"
|
|
9
10
|
|
|
10
11
|
jobs:
|
|
11
12
|
lint:
|
|
12
13
|
name: Lint
|
|
13
14
|
runs-on: ubuntu-latest
|
|
14
15
|
steps:
|
|
15
|
-
- uses: actions/checkout@
|
|
16
|
+
- uses: actions/checkout@v6
|
|
17
|
+
|
|
18
|
+
- name: Install FlatBuffers compiler
|
|
19
|
+
run: |
|
|
20
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
21
|
+
unzip -q flatc.zip
|
|
22
|
+
chmod +x flatc
|
|
23
|
+
sudo mv flatc /usr/local/bin/
|
|
16
24
|
|
|
17
25
|
- name: Install uv
|
|
18
|
-
uses: astral-sh/setup-uv@
|
|
26
|
+
uses: astral-sh/setup-uv@v7
|
|
19
27
|
|
|
20
28
|
- name: Set up Python
|
|
21
29
|
run: uv python install ${{ env.PYTHON_VERSION }}
|
|
@@ -36,10 +44,17 @@ jobs:
|
|
|
36
44
|
matrix:
|
|
37
45
|
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
|
|
38
46
|
steps:
|
|
39
|
-
- uses: actions/checkout@
|
|
47
|
+
- uses: actions/checkout@v6
|
|
48
|
+
|
|
49
|
+
- name: Install FlatBuffers compiler
|
|
50
|
+
run: |
|
|
51
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
52
|
+
unzip -q flatc.zip
|
|
53
|
+
chmod +x flatc
|
|
54
|
+
sudo mv flatc /usr/local/bin/
|
|
40
55
|
|
|
41
56
|
- name: Install uv
|
|
42
|
-
uses: astral-sh/setup-uv@
|
|
57
|
+
uses: astral-sh/setup-uv@v7
|
|
43
58
|
|
|
44
59
|
- name: Set up Python ${{ matrix.python-version }}
|
|
45
60
|
run: uv python install ${{ matrix.python-version }}
|
|
@@ -59,12 +74,28 @@ jobs:
|
|
|
59
74
|
PRIVATE_TEST_REPO: ${{ secrets.PRIVATE_TEST_REPO }}
|
|
60
75
|
run: uv run pytest python/tests/ -v
|
|
61
76
|
|
|
77
|
+
- name: Verify stub files
|
|
78
|
+
if: matrix.python-version == '3.14'
|
|
79
|
+
run: uv run python -m mypy.stubtest rustling --concise --ignore-missing-stub --allowlist python/stubtest_allowlist.txt
|
|
80
|
+
|
|
81
|
+
audit:
|
|
82
|
+
name: Audit
|
|
83
|
+
runs-on: ubuntu-latest
|
|
84
|
+
steps:
|
|
85
|
+
- uses: actions/checkout@v6
|
|
86
|
+
|
|
87
|
+
- name: Install uv
|
|
88
|
+
uses: astral-sh/setup-uv@v7
|
|
89
|
+
|
|
90
|
+
- name: Audit dev dependencies
|
|
91
|
+
run: uv export --no-hashes --group dev | uvx pip-audit -r /dev/stdin --desc
|
|
92
|
+
|
|
62
93
|
# This job should be the same as the "python-sdist" job in release.yml, but without the artifact upload.
|
|
63
94
|
sdist:
|
|
64
95
|
name: Build source distribution (release readiness check)
|
|
65
96
|
runs-on: ubuntu-latest
|
|
66
97
|
steps:
|
|
67
|
-
- uses: actions/checkout@
|
|
98
|
+
- uses: actions/checkout@v6
|
|
68
99
|
|
|
69
100
|
- name: Build sdist
|
|
70
101
|
uses: PyO3/maturin-action@v1
|
|
@@ -96,15 +127,34 @@ jobs:
|
|
|
96
127
|
- os: windows-latest
|
|
97
128
|
target: x86_64
|
|
98
129
|
steps:
|
|
99
|
-
- uses: actions/checkout@
|
|
130
|
+
- uses: actions/checkout@v6
|
|
131
|
+
|
|
132
|
+
- name: Install FlatBuffers compiler (macOS)
|
|
133
|
+
if: runner.os == 'macOS'
|
|
134
|
+
run: brew install flatbuffers
|
|
135
|
+
|
|
136
|
+
- name: Install FlatBuffers compiler (Windows)
|
|
137
|
+
if: runner.os == 'Windows'
|
|
138
|
+
shell: pwsh
|
|
139
|
+
run: |
|
|
140
|
+
$url = "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Windows.flatc.binary.zip"
|
|
141
|
+
Invoke-WebRequest -Uri $url -OutFile flatc.zip
|
|
142
|
+
Expand-Archive -Path flatc.zip -DestinationPath flatc_bin
|
|
143
|
+
echo "$PWD\flatc_bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
|
|
100
144
|
|
|
101
145
|
- name: Build wheels
|
|
102
146
|
uses: PyO3/maturin-action@v1
|
|
103
147
|
with:
|
|
104
148
|
target: ${{ matrix.target }}
|
|
105
|
-
args: --release --out dist
|
|
149
|
+
args: --release --out dist -i python3.10
|
|
106
150
|
sccache: "true"
|
|
107
151
|
manylinux: auto
|
|
152
|
+
before-script-linux: |
|
|
153
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
154
|
+
command -v unzip || yum install -y unzip 2>/dev/null || (apt-get update -qq && apt-get install -y -qq unzip)
|
|
155
|
+
unzip -q flatc.zip
|
|
156
|
+
chmod +x flatc
|
|
157
|
+
mv flatc /usr/local/bin/
|
|
108
158
|
|
|
109
159
|
# This job should be similar to the "python-wasm-wheel" job in release.yml, but without the artifact upload.
|
|
110
160
|
# Not using maturin-action for WASM because it doesn't set up Rust nightly, Emscripten SDK, or wasm-eh sysroot.
|
|
@@ -112,7 +162,14 @@ jobs:
|
|
|
112
162
|
name: Build wasm32-unknown-emscripten wheel (release readiness check)
|
|
113
163
|
runs-on: ubuntu-latest
|
|
114
164
|
steps:
|
|
115
|
-
- uses: actions/checkout@
|
|
165
|
+
- uses: actions/checkout@v6
|
|
166
|
+
|
|
167
|
+
- name: Install FlatBuffers compiler
|
|
168
|
+
run: |
|
|
169
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
170
|
+
unzip -q flatc.zip
|
|
171
|
+
chmod +x flatc
|
|
172
|
+
sudo mv flatc /usr/local/bin/
|
|
116
173
|
|
|
117
174
|
- name: Install Rust nightly
|
|
118
175
|
uses: dtolnay/rust-toolchain@master
|
|
@@ -124,7 +181,7 @@ jobs:
|
|
|
124
181
|
run: |
|
|
125
182
|
TOOLCHAIN_ROOT=$(rustup run nightly-2025-06-27 rustc --print sysroot)
|
|
126
183
|
RUSTLIB=$TOOLCHAIN_ROOT/lib/rustlib
|
|
127
|
-
|
|
184
|
+
curl -sL https://github.com/pyodide/rust-emscripten-wasm-eh-sysroot/releases/download/emcc-4.0.9_nightly-2025-06-27/emcc-4.0.9_nightly-2025-06-27.tar.bz2 -o emcc-4.0.9_nightly-2025-06-27.tar.bz2
|
|
128
185
|
tar -xf emcc-4.0.9_nightly-2025-06-27.tar.bz2 --directory=$RUSTLIB
|
|
129
186
|
|
|
130
187
|
- name: Cache Rust build artifacts
|
|
@@ -146,7 +203,7 @@ jobs:
|
|
|
146
203
|
curl -sL https://raw.githubusercontent.com/pyodide/pyodide/refs/heads/main/emsdk/patches/0002-Don-t-check-exports-for-being-valid-C-C-identifiers-.patch | git -C "$EMSCRIPTEN_DIR" apply
|
|
147
204
|
|
|
148
205
|
- name: Install uv
|
|
149
|
-
uses: astral-sh/setup-uv@
|
|
206
|
+
uses: astral-sh/setup-uv@v7
|
|
150
207
|
|
|
151
208
|
- name: Build wasm wheel
|
|
152
209
|
env:
|
|
@@ -4,6 +4,9 @@ on:
|
|
|
4
4
|
release:
|
|
5
5
|
types: [published]
|
|
6
6
|
|
|
7
|
+
env:
|
|
8
|
+
FLATC_VERSION: "25.12.19"
|
|
9
|
+
|
|
7
10
|
permissions:
|
|
8
11
|
contents: read
|
|
9
12
|
|
|
@@ -12,7 +15,7 @@ jobs:
|
|
|
12
15
|
name: Python - Build source distribution
|
|
13
16
|
runs-on: ubuntu-latest
|
|
14
17
|
steps:
|
|
15
|
-
- uses: actions/checkout@
|
|
18
|
+
- uses: actions/checkout@v6
|
|
16
19
|
|
|
17
20
|
- name: Build sdist
|
|
18
21
|
uses: PyO3/maturin-action@v1
|
|
@@ -21,7 +24,7 @@ jobs:
|
|
|
21
24
|
args: --out dist
|
|
22
25
|
|
|
23
26
|
- name: Upload sdist
|
|
24
|
-
uses: actions/upload-artifact@
|
|
27
|
+
uses: actions/upload-artifact@v7
|
|
25
28
|
with:
|
|
26
29
|
name: wheels-sdist
|
|
27
30
|
path: dist
|
|
@@ -49,18 +52,37 @@ jobs:
|
|
|
49
52
|
- os: windows-latest
|
|
50
53
|
target: x86_64
|
|
51
54
|
steps:
|
|
52
|
-
- uses: actions/checkout@
|
|
55
|
+
- uses: actions/checkout@v6
|
|
56
|
+
|
|
57
|
+
- name: Install FlatBuffers compiler (macOS)
|
|
58
|
+
if: runner.os == 'macOS'
|
|
59
|
+
run: brew install flatbuffers
|
|
60
|
+
|
|
61
|
+
- name: Install FlatBuffers compiler (Windows)
|
|
62
|
+
if: runner.os == 'Windows'
|
|
63
|
+
shell: pwsh
|
|
64
|
+
run: |
|
|
65
|
+
$url = "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Windows.flatc.binary.zip"
|
|
66
|
+
Invoke-WebRequest -Uri $url -OutFile flatc.zip
|
|
67
|
+
Expand-Archive -Path flatc.zip -DestinationPath flatc_bin
|
|
68
|
+
echo "$PWD\flatc_bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
|
|
53
69
|
|
|
54
70
|
- name: Build wheels
|
|
55
71
|
uses: PyO3/maturin-action@v1
|
|
56
72
|
with:
|
|
57
73
|
target: ${{ matrix.target }}
|
|
58
|
-
args: --release --out dist
|
|
74
|
+
args: --release --out dist -i python3.10
|
|
59
75
|
sccache: "true"
|
|
60
76
|
manylinux: auto
|
|
77
|
+
before-script-linux: |
|
|
78
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
79
|
+
command -v unzip || yum install -y unzip 2>/dev/null || (apt-get update -qq && apt-get install -y -qq unzip)
|
|
80
|
+
unzip -q flatc.zip
|
|
81
|
+
chmod +x flatc
|
|
82
|
+
mv flatc /usr/local/bin/
|
|
61
83
|
|
|
62
84
|
- name: Upload wheels
|
|
63
|
-
uses: actions/upload-artifact@
|
|
85
|
+
uses: actions/upload-artifact@v7
|
|
64
86
|
with:
|
|
65
87
|
name: wheels-${{ matrix.os }}-${{ matrix.target }}
|
|
66
88
|
path: dist
|
|
@@ -70,7 +92,14 @@ jobs:
|
|
|
70
92
|
name: Python - Build wasm32-unknown-emscripten wheel
|
|
71
93
|
runs-on: ubuntu-latest
|
|
72
94
|
steps:
|
|
73
|
-
- uses: actions/checkout@
|
|
95
|
+
- uses: actions/checkout@v6
|
|
96
|
+
|
|
97
|
+
- name: Install FlatBuffers compiler
|
|
98
|
+
run: |
|
|
99
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
100
|
+
unzip -q flatc.zip
|
|
101
|
+
chmod +x flatc
|
|
102
|
+
sudo mv flatc /usr/local/bin/
|
|
74
103
|
|
|
75
104
|
- name: Install Rust nightly
|
|
76
105
|
uses: dtolnay/rust-toolchain@master
|
|
@@ -82,7 +111,7 @@ jobs:
|
|
|
82
111
|
run: |
|
|
83
112
|
TOOLCHAIN_ROOT=$(rustup run nightly-2025-06-27 rustc --print sysroot)
|
|
84
113
|
RUSTLIB=$TOOLCHAIN_ROOT/lib/rustlib
|
|
85
|
-
|
|
114
|
+
curl -sL https://github.com/pyodide/rust-emscripten-wasm-eh-sysroot/releases/download/emcc-4.0.9_nightly-2025-06-27/emcc-4.0.9_nightly-2025-06-27.tar.bz2 -o emcc-4.0.9_nightly-2025-06-27.tar.bz2
|
|
86
115
|
tar -xf emcc-4.0.9_nightly-2025-06-27.tar.bz2 --directory=$RUSTLIB
|
|
87
116
|
|
|
88
117
|
- name: Cache Rust build artifacts
|
|
@@ -104,7 +133,7 @@ jobs:
|
|
|
104
133
|
curl -sL https://raw.githubusercontent.com/pyodide/pyodide/refs/heads/main/emsdk/patches/0002-Don-t-check-exports-for-being-valid-C-C-identifiers-.patch | git -C "$EMSCRIPTEN_DIR" apply
|
|
105
134
|
|
|
106
135
|
- name: Install uv
|
|
107
|
-
uses: astral-sh/setup-uv@
|
|
136
|
+
uses: astral-sh/setup-uv@v7
|
|
108
137
|
|
|
109
138
|
- name: Build wasm wheel
|
|
110
139
|
env:
|
|
@@ -113,7 +142,7 @@ jobs:
|
|
|
113
142
|
run: uvx maturin build --release --target wasm32-unknown-emscripten --out dist -i 3.13 --no-default-features --features extension-module
|
|
114
143
|
|
|
115
144
|
- name: Upload wasm wheel
|
|
116
|
-
uses: actions/upload-artifact@
|
|
145
|
+
uses: actions/upload-artifact@v7
|
|
117
146
|
with:
|
|
118
147
|
# Not "wheels-*" to avoid being picked up by the PyPI publish step,
|
|
119
148
|
# which rejects the emscripten platform tag.
|
|
@@ -131,7 +160,7 @@ jobs:
|
|
|
131
160
|
id-token: write
|
|
132
161
|
steps:
|
|
133
162
|
- name: Download all artifacts
|
|
134
|
-
uses: actions/download-artifact@
|
|
163
|
+
uses: actions/download-artifact@v8
|
|
135
164
|
with:
|
|
136
165
|
path: dist
|
|
137
166
|
pattern: wheels-*
|
|
@@ -148,7 +177,7 @@ jobs:
|
|
|
148
177
|
contents: write
|
|
149
178
|
steps:
|
|
150
179
|
- name: Download all artifacts
|
|
151
|
-
uses: actions/download-artifact@
|
|
180
|
+
uses: actions/download-artifact@v8
|
|
152
181
|
with:
|
|
153
182
|
path: dist
|
|
154
183
|
merge-multiple: true
|
|
@@ -166,7 +195,15 @@ jobs:
|
|
|
166
195
|
permissions:
|
|
167
196
|
id-token: write
|
|
168
197
|
steps:
|
|
169
|
-
- uses: actions/checkout@
|
|
198
|
+
- uses: actions/checkout@v6
|
|
199
|
+
|
|
200
|
+
- name: Install FlatBuffers compiler
|
|
201
|
+
run: |
|
|
202
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
203
|
+
unzip -q flatc.zip
|
|
204
|
+
rm flatc.zip
|
|
205
|
+
chmod +x flatc
|
|
206
|
+
sudo mv flatc /usr/local/bin/
|
|
170
207
|
|
|
171
208
|
- name: Install Rust toolchain
|
|
172
209
|
uses: dtolnay/rust-toolchain@stable
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
name: Rust
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
pull_request:
|
|
6
|
+
|
|
7
|
+
env:
|
|
8
|
+
CARGO_TERM_COLOR: always
|
|
9
|
+
FLATC_VERSION: "25.12.19"
|
|
10
|
+
|
|
11
|
+
jobs:
|
|
12
|
+
lint:
|
|
13
|
+
name: Lint
|
|
14
|
+
runs-on: ubuntu-latest
|
|
15
|
+
steps:
|
|
16
|
+
- uses: actions/checkout@v6
|
|
17
|
+
|
|
18
|
+
- name: Install FlatBuffers compiler
|
|
19
|
+
run: |
|
|
20
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
21
|
+
unzip -q flatc.zip
|
|
22
|
+
chmod +x flatc
|
|
23
|
+
sudo mv flatc /usr/local/bin/
|
|
24
|
+
|
|
25
|
+
- name: Install Rust toolchain
|
|
26
|
+
uses: dtolnay/rust-toolchain@stable
|
|
27
|
+
with:
|
|
28
|
+
components: rustfmt, clippy
|
|
29
|
+
|
|
30
|
+
- name: Check formatting
|
|
31
|
+
run: cargo fmt --all -- --check
|
|
32
|
+
|
|
33
|
+
- name: Run clippy
|
|
34
|
+
run: cargo clippy --all-targets --all-features -- -D warnings
|
|
35
|
+
|
|
36
|
+
test:
|
|
37
|
+
name: Test
|
|
38
|
+
runs-on: ubuntu-latest
|
|
39
|
+
steps:
|
|
40
|
+
- uses: actions/checkout@v6
|
|
41
|
+
|
|
42
|
+
- name: Install FlatBuffers compiler
|
|
43
|
+
run: |
|
|
44
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
45
|
+
unzip -q flatc.zip
|
|
46
|
+
chmod +x flatc
|
|
47
|
+
sudo mv flatc /usr/local/bin/
|
|
48
|
+
|
|
49
|
+
- name: Set up Python
|
|
50
|
+
uses: actions/setup-python@v6
|
|
51
|
+
with:
|
|
52
|
+
python-version: "3.14"
|
|
53
|
+
|
|
54
|
+
- name: Install Rust toolchain
|
|
55
|
+
uses: dtolnay/rust-toolchain@stable
|
|
56
|
+
|
|
57
|
+
- name: Run tests
|
|
58
|
+
run: cargo test
|
|
59
|
+
|
|
60
|
+
audit:
|
|
61
|
+
name: Audit
|
|
62
|
+
runs-on: ubuntu-latest
|
|
63
|
+
steps:
|
|
64
|
+
- uses: actions/checkout@v6
|
|
65
|
+
|
|
66
|
+
- name: Install Rust toolchain
|
|
67
|
+
uses: dtolnay/rust-toolchain@stable
|
|
68
|
+
|
|
69
|
+
- name: Install cargo-audit
|
|
70
|
+
run: cargo install cargo-audit
|
|
71
|
+
|
|
72
|
+
- name: Run cargo audit
|
|
73
|
+
run: cargo audit
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
repos:
|
|
2
|
+
- repo: local
|
|
3
|
+
hooks:
|
|
4
|
+
- id: cargo-fmt
|
|
5
|
+
name: cargo fmt
|
|
6
|
+
entry: cargo fmt --all -- --check
|
|
7
|
+
language: system
|
|
8
|
+
types: [rust]
|
|
9
|
+
pass_filenames: false
|
|
10
|
+
- id: cargo-clippy
|
|
11
|
+
name: cargo clippy
|
|
12
|
+
entry: cargo clippy --all-targets --all-features -- -D warnings
|
|
13
|
+
language: system
|
|
14
|
+
types: [rust]
|
|
15
|
+
pass_filenames: false
|
|
16
|
+
- repo: https://github.com/psf/black-pre-commit-mirror
|
|
17
|
+
rev: 26.3.0
|
|
18
|
+
hooks:
|
|
19
|
+
- id: black
|
|
20
|
+
files: ^(python|benchmarks)/
|
|
21
|
+
- repo: https://github.com/pycqa/flake8
|
|
22
|
+
rev: 7.3.0
|
|
23
|
+
hooks:
|
|
24
|
+
- id: flake8
|
|
25
|
+
files: ^(python|benchmarks)/
|
|
26
|
+
- repo: local
|
|
27
|
+
hooks:
|
|
28
|
+
- id: mypy
|
|
29
|
+
name: mypy
|
|
30
|
+
entry: uvx mypy
|
|
31
|
+
language: system
|
|
32
|
+
types: [python]
|
|
33
|
+
files: ^python/rustling/
|
|
34
|
+
pass_filenames: false
|
|
35
|
+
args: [python/rustling/]
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.7.0] - 2026-03-14
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- Word segmentation:
|
|
15
|
+
* Added `score` method for the HMM and DAG-HMM segmenters.
|
|
16
|
+
* `predict` method can optionally output offsets for the (start, end) indices
|
|
17
|
+
of segmented words compared to the original string.
|
|
18
|
+
- CHAT parsing: Support custom tier names other than the standard %mor and %gra.
|
|
19
|
+
- Python model classes are now subclassable.
|
|
20
|
+
|
|
21
|
+
### Changed
|
|
22
|
+
|
|
23
|
+
- Ngram counters: `Ngrams.most_common` now sorts tuples lexicographically
|
|
24
|
+
when counts are tied.
|
|
25
|
+
- CHAT parsing:
|
|
26
|
+
* If a date is available at `Headers`'s `date`,
|
|
27
|
+
it's now a Python `datetime.date` object instead of a string.
|
|
28
|
+
* In handling the main tier transcription for creating `Token` objects:
|
|
29
|
+
- Special form markers suffixed with "@" are now stripped.
|
|
30
|
+
- Words that have partiallly parenthetical material have the parentheses
|
|
31
|
+
removed, e.g., (un)til -> until, sit(ting) -> sitting.
|
|
32
|
+
* Renamed the `CHAT.raw` attribute to `CHAT.audible` for a best-effort,
|
|
33
|
+
audibly faithful transcription string, to facilitate automatic speech recognition,
|
|
34
|
+
forced alignment, etc.
|
|
35
|
+
* A subset of the testchat/bad dataset is now used to validate CHAT data format.
|
|
36
|
+
- Refactored core Rust code so that Rust-only consumers no longer need PyO3/Python.
|
|
37
|
+
|
|
38
|
+
## [0.6.0] - 2026-03-05
|
|
39
|
+
|
|
40
|
+
### Added
|
|
41
|
+
|
|
42
|
+
- Hidden Markov Model (HMM)
|
|
43
|
+
- Word segmentation: Added DAG-HMM word segmenter
|
|
44
|
+
- CHAT parsing: Added `from_utterances` method
|
|
45
|
+
|
|
46
|
+
### Changed
|
|
47
|
+
|
|
48
|
+
- Models are now persisted as a zstd-compressed FlatBuffers binary.
|
|
49
|
+
|
|
50
|
+
## [0.5.0] - 2026-02-18
|
|
51
|
+
|
|
52
|
+
### Added
|
|
53
|
+
|
|
54
|
+
- CHAT parsing for TalkBank and CHILDES data
|
|
55
|
+
|
|
56
|
+
## [0.4.0] - 2026-02-08
|
|
57
|
+
|
|
58
|
+
### Added
|
|
59
|
+
|
|
60
|
+
- N-grams and language models
|
|
61
|
+
|
|
62
|
+
## [0.3.0] - 2026-02-06
|
|
63
|
+
|
|
64
|
+
### Added
|
|
65
|
+
|
|
66
|
+
- Averaged perceptron tagger
|
|
67
|
+
|
|
68
|
+
## [0.2.0] - 2026-02-04
|
|
69
|
+
|
|
70
|
+
- Initial release, with longest string matching and random segmenter for word segmentation
|