rustling 0.6.0__tar.gz → 0.8.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- rustling-0.8.0/.github/workflows/build-wheel.yml +42 -0
- {rustling-0.6.0 → rustling-0.8.0}/.github/workflows/python.yml +1 -1
- {rustling-0.6.0 → rustling-0.8.0}/.github/workflows/release.yml +2 -1
- {rustling-0.6.0 → rustling-0.8.0}/.gitignore +3 -0
- {rustling-0.6.0 → rustling-0.8.0}/.pre-commit-config.yaml +11 -1
- rustling-0.8.0/CHANGELOG.md +83 -0
- {rustling-0.6.0 → rustling-0.8.0}/Cargo.lock +26 -29
- {rustling-0.6.0 → rustling-0.8.0}/Cargo.toml +5 -15
- {rustling-0.6.0 → rustling-0.8.0}/PKG-INFO +43 -33
- rustling-0.8.0/README.md +70 -0
- rustling-0.8.0/benchmarks/README.md +132 -0
- rustling-0.8.0/benchmarks/run_conllu.py +300 -0
- rustling-0.8.0/benchmarks/run_elan.py +299 -0
- rustling-0.8.0/benchmarks/run_textgrid.py +320 -0
- rustling-0.6.0/benchmarks/update_docs.py → rustling-0.8.0/benchmarks/update_readme.py +86 -66
- {rustling-0.6.0 → rustling-0.8.0}/pyproject.toml +3 -4
- rustling-0.8.0/python/docs/_static/brown_eve_mlum.png +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/api.rst +4 -0
- rustling-0.8.0/python/docs/chat/formats.rst +219 -0
- rustling-0.8.0/python/docs/chat/frequencies.rst +99 -0
- rustling-0.8.0/python/docs/chat/headers.rst +171 -0
- rustling-0.8.0/python/docs/chat/measures.rst +62 -0
- rustling-0.8.0/python/docs/chat/quickstart.rst +349 -0
- rustling-0.8.0/python/docs/chat/read.rst +479 -0
- rustling-0.8.0/python/docs/chat/transcriptions.rst +626 -0
- rustling-0.8.0/python/docs/chat/write.rst +48 -0
- rustling-0.8.0/python/docs/chat.rst +19 -0
- rustling-0.8.0/python/docs/conllu.rst +221 -0
- rustling-0.8.0/python/docs/elan.rst +275 -0
- rustling-0.8.0/python/docs/index.rst +104 -0
- rustling-0.8.0/python/docs/srt.rst +240 -0
- rustling-0.8.0/python/docs/textgrid.rst +288 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/wordseg.rst +8 -8
- rustling-0.8.0/python/rustling/__init__.py +36 -0
- rustling-0.8.0/python/rustling/_lib_name.pyi +3 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/chat/__init__.py +2 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/chat/__init__.pyi +365 -15
- rustling-0.8.0/python/rustling/chat/_read_chat.py +78 -0
- rustling-0.8.0/python/rustling/conllu/__init__.py +13 -0
- rustling-0.8.0/python/rustling/conllu/__init__.pyi +326 -0
- rustling-0.8.0/python/rustling/conllu/_read_conllu.py +53 -0
- rustling-0.8.0/python/rustling/elan/__init__.py +19 -0
- rustling-0.8.0/python/rustling/elan/__init__.pyi +442 -0
- rustling-0.8.0/python/rustling/elan/_read_elan.py +53 -0
- rustling-0.8.0/python/rustling/srt/__init__.py +11 -0
- rustling-0.8.0/python/rustling/srt/__init__.pyi +345 -0
- rustling-0.8.0/python/rustling/srt/_read_srt.py +53 -0
- rustling-0.8.0/python/rustling/textgrid/__init__.py +23 -0
- rustling-0.8.0/python/rustling/textgrid/__init__.pyi +348 -0
- rustling-0.8.0/python/rustling/textgrid/_read_textgrid.py +54 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/wordseg/__init__.pyi +113 -9
- {rustling-0.6.0 → rustling-0.8.0}/python/stubtest_allowlist.txt +35 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_chat/test_chat.py +299 -40
- rustling-0.8.0/python/tests/test_conllu/conftest.py +45 -0
- rustling-0.8.0/python/tests/test_conllu/test_conllu.py +276 -0
- rustling-0.8.0/python/tests/test_elan/conftest.py +33 -0
- rustling-0.8.0/python/tests/test_elan/test_elan.py +546 -0
- rustling-0.8.0/python/tests/test_textgrid/conftest.py +0 -0
- rustling-0.8.0/python/tests/test_textgrid/test_textgrid.py +398 -0
- rustling-0.8.0/python/tests/test_wordseg/test_dag_hmm_segmenter.py +69 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_wordseg/test_hmm_segmenter.py +42 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_wordseg/test_longest_string_matching.py +36 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_wordseg/test_random_segmenter.py +31 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/chat/clean_utterance.rs +481 -11
- rustling-0.8.0/src/chat/conllu_writer.rs +147 -0
- rustling-0.8.0/src/chat/elan_writer.rs +486 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/chat/header.rs +85 -140
- rustling-0.8.0/src/chat/header_py.rs +312 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/chat/ipsyn.rs +4 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/chat/mod.rs +20 -5
- {rustling-0.6.0 → rustling-0.8.0}/src/chat/reader.rs +711 -1191
- rustling-0.8.0/src/chat/reader_py.rs +1358 -0
- rustling-0.8.0/src/chat/srt_writer.rs +324 -0
- rustling-0.8.0/src/chat/textgrid_writer.rs +201 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/chat/utterance.rs +108 -309
- rustling-0.8.0/src/chat/utterance_py.rs +330 -0
- rustling-0.8.0/src/chat/validation.rs +1121 -0
- rustling-0.8.0/src/conllu/chat_writer.rs +222 -0
- rustling-0.8.0/src/conllu/mod.rs +885 -0
- rustling-0.8.0/src/conllu/reader_py.rs +547 -0
- rustling-0.8.0/src/elan/chat_writer.rs +414 -0
- rustling-0.8.0/src/elan/mod.rs +32 -0
- rustling-0.8.0/src/elan/reader.rs +1182 -0
- rustling-0.8.0/src/elan/reader_py.rs +629 -0
- rustling-0.8.0/src/elan/srt_writer.rs +242 -0
- rustling-0.8.0/src/elan/textgrid_writer.rs +155 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/hmm/mod.rs +52 -276
- rustling-0.8.0/src/hmm/py.rs +254 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/lib.rs +14 -1
- {rustling-0.6.0 → rustling-0.8.0}/src/lm/mod.rs +40 -382
- rustling-0.8.0/src/lm/py.rs +370 -0
- rustling-0.6.0/src/ngram.rs → rustling-0.8.0/src/ngram/mod.rs +30 -222
- rustling-0.8.0/src/ngram/py.rs +204 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/perceptron_pos_tagger/mod.rs +8 -141
- rustling-0.8.0/src/perceptron_pos_tagger/py.rs +144 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/persistence.rs +28 -2
- rustling-0.8.0/src/prelude.rs +15 -0
- rustling-0.6.0/src/seq_feature.rs → rustling-0.8.0/src/seq_feature/mod.rs +23 -80
- rustling-0.8.0/src/seq_feature/py.rs +74 -0
- rustling-0.8.0/src/sources/cache.rs +76 -0
- rustling-0.8.0/src/sources/git.rs +123 -0
- rustling-0.8.0/src/sources/mod.rs +57 -0
- rustling-0.8.0/src/sources/url.rs +68 -0
- rustling-0.8.0/src/srt/chat_writer.rs +140 -0
- rustling-0.8.0/src/srt/elan_writer.rs +211 -0
- rustling-0.8.0/src/srt/mod.rs +986 -0
- rustling-0.8.0/src/srt/reader_py.rs +523 -0
- rustling-0.8.0/src/srt/textgrid_writer.rs +122 -0
- rustling-0.8.0/src/textgrid/chat_writer.rs +219 -0
- rustling-0.8.0/src/textgrid/elan_writer.rs +245 -0
- rustling-0.8.0/src/textgrid/mod.rs +34 -0
- rustling-0.8.0/src/textgrid/reader.rs +1691 -0
- rustling-0.8.0/src/textgrid/reader_py.rs +750 -0
- rustling-0.8.0/src/textgrid/srt_writer.rs +208 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/dag_hmm.rs +45 -60
- rustling-0.8.0/src/wordseg/dag_hmm_py.rs +76 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/hmm.rs +86 -83
- rustling-0.8.0/src/wordseg/hmm_py.rs +99 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/longest_string_matching/mod.rs +15 -100
- rustling-0.8.0/src/wordseg/longest_string_matching/py.rs +109 -0
- rustling-0.8.0/src/wordseg/mod.rs +141 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/random_segmenter.rs +12 -59
- rustling-0.8.0/src/wordseg/random_segmenter_py.rs +68 -0
- rustling-0.6.0/AGENTS.md +0 -335
- rustling-0.6.0/CHANGELOG.md +0 -42
- rustling-0.6.0/CLAUDE.md +0 -335
- rustling-0.6.0/README.md +0 -57
- rustling-0.6.0/benchmarks/README.md +0 -89
- rustling-0.6.0/python/docs/chat.rst +0 -7
- rustling-0.6.0/python/docs/index.rst +0 -110
- rustling-0.6.0/python/rustling/__init__.py +0 -11
- rustling-0.6.0/src/wordseg/mod.rs +0 -55
- {rustling-0.6.0 → rustling-0.8.0}/.flake8 +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/.gitattributes +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/.github/dependabot.yml +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/.github/workflows/rust.yml +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/.readthedocs.yaml +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/LICENSE.md +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/benchmarks/common/__init__.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/benchmarks/common/data.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/benchmarks/run_chat.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/benchmarks/run_hmm.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/benchmarks/run_lm.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/benchmarks/run_perceptron_pos_tagger.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/benchmarks/run_wordseg.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/build.rs +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/Makefile +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/README.md +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/_static/favicon.ico +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/_static/logo-with-text.svg +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/_templates/copyright.html +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/conf.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/hmm.rst +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/lm.rst +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/make.bat +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/ngram.rst +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/perceptron_pos_tagger.rst +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/docs/requirements.txt +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/hmm/__init__.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/hmm/__init__.pyi +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/lm/__init__.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/lm/__init__.pyi +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/ngram/__init__.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/ngram/__init__.pyi +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/perceptron_pos_tagger/__init__.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/perceptron_pos_tagger/__init__.pyi +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/py.typed +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/seq_feature/__init__.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/seq_feature/__init__.pyi +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/rustling/wordseg/__init__.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_benchmarks.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_chat/conftest.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_hmm.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_lm.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_ngram.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/python/tests/test_perceptron_pos_tagger.py +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/hmm/model.fbs +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/hmm/model_generated.rs +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/lm/model.fbs +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/lm/model_generated.rs +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/perceptron_pos_tagger/model.fbs +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/perceptron_pos_tagger/model_generated.rs +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/trie.rs +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/bmes.rs +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/longest_string_matching/model.fbs +0 -0
- {rustling-0.6.0 → rustling-0.8.0}/src/wordseg/longest_string_matching/model_generated.rs +0 -0
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
name: Build Linux x86_64 wheel
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
workflow_dispatch:
|
|
5
|
+
inputs:
|
|
6
|
+
git_ref:
|
|
7
|
+
description: "Git ref to build from (tag, branch, or commit)"
|
|
8
|
+
required: true
|
|
9
|
+
default: "v0.6.0"
|
|
10
|
+
|
|
11
|
+
env:
|
|
12
|
+
FLATC_VERSION: "25.12.19"
|
|
13
|
+
|
|
14
|
+
jobs:
|
|
15
|
+
build:
|
|
16
|
+
name: Build manylinux x86_64 wheel
|
|
17
|
+
runs-on: ubuntu-latest
|
|
18
|
+
steps:
|
|
19
|
+
- uses: actions/checkout@v6
|
|
20
|
+
with:
|
|
21
|
+
ref: ${{ github.event.inputs.git_ref }}
|
|
22
|
+
|
|
23
|
+
- name: Build wheels
|
|
24
|
+
uses: PyO3/maturin-action@v1
|
|
25
|
+
with:
|
|
26
|
+
target: x86_64
|
|
27
|
+
args: --release --out dist -i python3.10
|
|
28
|
+
sccache: "true"
|
|
29
|
+
manylinux: auto
|
|
30
|
+
before-script-linux: |
|
|
31
|
+
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
32
|
+
command -v unzip || yum install -y unzip 2>/dev/null || (apt-get update -qq && apt-get install -y -qq unzip)
|
|
33
|
+
unzip -q flatc.zip
|
|
34
|
+
rm flatc.zip
|
|
35
|
+
chmod +x flatc
|
|
36
|
+
mv flatc /usr/local/bin/
|
|
37
|
+
|
|
38
|
+
- name: Upload wheel
|
|
39
|
+
uses: actions/upload-artifact@v7
|
|
40
|
+
with:
|
|
41
|
+
name: wheel-linux-x86_64
|
|
42
|
+
path: dist
|
|
@@ -71,7 +71,7 @@ jobs:
|
|
|
71
71
|
uses: PyO3/maturin-action@v1
|
|
72
72
|
with:
|
|
73
73
|
target: ${{ matrix.target }}
|
|
74
|
-
args: --release --out dist
|
|
74
|
+
args: --release --out dist -i python3.10
|
|
75
75
|
sccache: "true"
|
|
76
76
|
manylinux: auto
|
|
77
77
|
before-script-linux: |
|
|
@@ -201,6 +201,7 @@ jobs:
|
|
|
201
201
|
run: |
|
|
202
202
|
curl -sL "https://github.com/google/flatbuffers/releases/download/v${{ env.FLATC_VERSION }}/Linux.flatc.binary.clang++-18.zip" -o flatc.zip
|
|
203
203
|
unzip -q flatc.zip
|
|
204
|
+
rm flatc.zip
|
|
204
205
|
chmod +x flatc
|
|
205
206
|
sudo mv flatc /usr/local/bin/
|
|
206
207
|
|
|
@@ -14,7 +14,7 @@ repos:
|
|
|
14
14
|
types: [rust]
|
|
15
15
|
pass_filenames: false
|
|
16
16
|
- repo: https://github.com/psf/black-pre-commit-mirror
|
|
17
|
-
rev: 26.
|
|
17
|
+
rev: 26.3.0
|
|
18
18
|
hooks:
|
|
19
19
|
- id: black
|
|
20
20
|
files: ^(python|benchmarks)/
|
|
@@ -23,3 +23,13 @@ repos:
|
|
|
23
23
|
hooks:
|
|
24
24
|
- id: flake8
|
|
25
25
|
files: ^(python|benchmarks)/
|
|
26
|
+
- repo: local
|
|
27
|
+
hooks:
|
|
28
|
+
- id: mypy
|
|
29
|
+
name: mypy
|
|
30
|
+
entry: uvx mypy
|
|
31
|
+
language: system
|
|
32
|
+
types: [python]
|
|
33
|
+
files: ^python/rustling/
|
|
34
|
+
pass_filenames: false
|
|
35
|
+
args: [python/rustling/]
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.8.0] - 2026-03-20
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- Support for data formats:
|
|
15
|
+
* CoNLL-U for Universal Dependencies
|
|
16
|
+
* ELAN for annotated multimedia data
|
|
17
|
+
* TextGrid for Praat annotations
|
|
18
|
+
* SRT for subtitles
|
|
19
|
+
- CHAT data handling:
|
|
20
|
+
* Added a convenience function `read_chat.
|
|
21
|
+
* Added `from_git` and `from_url` methods for remote data sources.
|
|
22
|
+
|
|
23
|
+
## [0.7.0] - 2026-03-14
|
|
24
|
+
|
|
25
|
+
### Added
|
|
26
|
+
|
|
27
|
+
- Word segmentation:
|
|
28
|
+
* Added `score` method for the HMM and DAG-HMM segmenters.
|
|
29
|
+
* `predict` method can optionally output offsets for the (start, end) indices
|
|
30
|
+
of segmented words compared to the original string.
|
|
31
|
+
- CHAT parsing: Support custom tier names other than the standard %mor and %gra.
|
|
32
|
+
- Python model classes are now subclassable.
|
|
33
|
+
|
|
34
|
+
### Changed
|
|
35
|
+
|
|
36
|
+
- Ngram counters: `Ngrams.most_common` now sorts tuples lexicographically
|
|
37
|
+
when counts are tied.
|
|
38
|
+
- CHAT parsing:
|
|
39
|
+
* If a date is available at `Headers`'s `date`,
|
|
40
|
+
it's now a Python `datetime.date` object instead of a string.
|
|
41
|
+
* In handling the main tier transcription for creating `Token` objects:
|
|
42
|
+
- Special form markers suffixed with "@" are now stripped.
|
|
43
|
+
- Words that have partiallly parenthetical material have the parentheses
|
|
44
|
+
removed, e.g., (un)til -> until, sit(ting) -> sitting.
|
|
45
|
+
* Renamed the `CHAT.raw` attribute to `CHAT.audible` for a best-effort,
|
|
46
|
+
audibly faithful transcription string, to facilitate automatic speech recognition,
|
|
47
|
+
forced alignment, etc.
|
|
48
|
+
* A subset of the testchat/bad dataset is now used to validate CHAT data format.
|
|
49
|
+
- Refactored core Rust code so that Rust-only consumers no longer need PyO3/Python.
|
|
50
|
+
|
|
51
|
+
## [0.6.0] - 2026-03-05
|
|
52
|
+
|
|
53
|
+
### Added
|
|
54
|
+
|
|
55
|
+
- Hidden Markov Model (HMM)
|
|
56
|
+
- Word segmentation: Added DAG-HMM word segmenter
|
|
57
|
+
- CHAT parsing: Added `from_utterances` method
|
|
58
|
+
|
|
59
|
+
### Changed
|
|
60
|
+
|
|
61
|
+
- Models are now persisted as a zstd-compressed FlatBuffers binary.
|
|
62
|
+
|
|
63
|
+
## [0.5.0] - 2026-02-18
|
|
64
|
+
|
|
65
|
+
### Added
|
|
66
|
+
|
|
67
|
+
- CHAT parsing for TalkBank and CHILDES data
|
|
68
|
+
|
|
69
|
+
## [0.4.0] - 2026-02-08
|
|
70
|
+
|
|
71
|
+
### Added
|
|
72
|
+
|
|
73
|
+
- N-grams and language models
|
|
74
|
+
|
|
75
|
+
## [0.3.0] - 2026-02-06
|
|
76
|
+
|
|
77
|
+
### Added
|
|
78
|
+
|
|
79
|
+
- Averaged perceptron tagger
|
|
80
|
+
|
|
81
|
+
## [0.2.0] - 2026-02-04
|
|
82
|
+
|
|
83
|
+
- Initial release, with longest string matching and random segmenter for word segmentation
|
|
@@ -81,9 +81,9 @@ dependencies = [
|
|
|
81
81
|
|
|
82
82
|
[[package]]
|
|
83
83
|
name = "cc"
|
|
84
|
-
version = "1.2.
|
|
84
|
+
version = "1.2.57"
|
|
85
85
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
86
|
-
checksum = "
|
|
86
|
+
checksum = "7a0dd1ca384932ff3641c8718a02769f1698e7563dc6974ffd03346116310423"
|
|
87
87
|
dependencies = [
|
|
88
88
|
"find-msvc-tools",
|
|
89
89
|
"jobserver",
|
|
@@ -380,9 +380,9 @@ dependencies = [
|
|
|
380
380
|
|
|
381
381
|
[[package]]
|
|
382
382
|
name = "itoa"
|
|
383
|
-
version = "1.0.
|
|
383
|
+
version = "1.0.18"
|
|
384
384
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
385
|
-
checksum = "
|
|
385
|
+
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
|
|
386
386
|
|
|
387
387
|
[[package]]
|
|
388
388
|
name = "jobserver"
|
|
@@ -418,9 +418,9 @@ checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7"
|
|
|
418
418
|
|
|
419
419
|
[[package]]
|
|
420
420
|
name = "libc"
|
|
421
|
-
version = "0.2.
|
|
421
|
+
version = "0.2.183"
|
|
422
422
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
423
|
-
checksum = "
|
|
423
|
+
checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d"
|
|
424
424
|
|
|
425
425
|
[[package]]
|
|
426
426
|
name = "linux-raw-sys"
|
|
@@ -467,9 +467,9 @@ checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050"
|
|
|
467
467
|
|
|
468
468
|
[[package]]
|
|
469
469
|
name = "once_cell"
|
|
470
|
-
version = "1.21.
|
|
470
|
+
version = "1.21.4"
|
|
471
471
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
472
|
-
checksum = "
|
|
472
|
+
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
|
|
473
473
|
|
|
474
474
|
[[package]]
|
|
475
475
|
name = "pbkdf2"
|
|
@@ -582,6 +582,16 @@ dependencies = [
|
|
|
582
582
|
"syn",
|
|
583
583
|
]
|
|
584
584
|
|
|
585
|
+
[[package]]
|
|
586
|
+
name = "quick-xml"
|
|
587
|
+
version = "0.39.2"
|
|
588
|
+
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
589
|
+
checksum = "958f21e8e7ceb5a1aa7fa87fab28e7c75976e0bfe7e23ff069e0a260f894067d"
|
|
590
|
+
dependencies = [
|
|
591
|
+
"memchr",
|
|
592
|
+
"serde",
|
|
593
|
+
]
|
|
594
|
+
|
|
585
595
|
[[package]]
|
|
586
596
|
name = "quote"
|
|
587
597
|
version = "1.0.45"
|
|
@@ -699,11 +709,12 @@ dependencies = [
|
|
|
699
709
|
|
|
700
710
|
[[package]]
|
|
701
711
|
name = "rustling"
|
|
702
|
-
version = "0.
|
|
712
|
+
version = "0.8.0"
|
|
703
713
|
dependencies = [
|
|
704
714
|
"fancy-regex",
|
|
705
715
|
"flatbuffers",
|
|
706
716
|
"pyo3",
|
|
717
|
+
"quick-xml",
|
|
707
718
|
"rand",
|
|
708
719
|
"rayon",
|
|
709
720
|
"regex",
|
|
@@ -840,9 +851,9 @@ checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
|
|
|
840
851
|
|
|
841
852
|
[[package]]
|
|
842
853
|
name = "tempfile"
|
|
843
|
-
version = "3.
|
|
854
|
+
version = "3.27.0"
|
|
844
855
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
845
|
-
checksum = "
|
|
856
|
+
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
|
|
846
857
|
dependencies = [
|
|
847
858
|
"fastrand",
|
|
848
859
|
"getrandom 0.4.2",
|
|
@@ -897,9 +908,9 @@ checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853"
|
|
|
897
908
|
|
|
898
909
|
[[package]]
|
|
899
910
|
name = "uuid"
|
|
900
|
-
version = "1.
|
|
911
|
+
version = "1.22.0"
|
|
901
912
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
902
|
-
checksum = "
|
|
913
|
+
checksum = "a68d3c8f01c0cfa54a75291d83601161799e4a89a39e0929f4b0354d88757a37"
|
|
903
914
|
dependencies = [
|
|
904
915
|
"getrandom 0.4.2",
|
|
905
916
|
"js-sys",
|
|
@@ -1136,26 +1147,12 @@ name = "zeroize"
|
|
|
1136
1147
|
version = "1.8.2"
|
|
1137
1148
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
1138
1149
|
checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0"
|
|
1139
|
-
dependencies = [
|
|
1140
|
-
"zeroize_derive",
|
|
1141
|
-
]
|
|
1142
|
-
|
|
1143
|
-
[[package]]
|
|
1144
|
-
name = "zeroize_derive"
|
|
1145
|
-
version = "1.4.3"
|
|
1146
|
-
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
1147
|
-
checksum = "85a5b4158499876c763cb03bc4e49185d3cccbabb15b33c627f7884f43db852e"
|
|
1148
|
-
dependencies = [
|
|
1149
|
-
"proc-macro2",
|
|
1150
|
-
"quote",
|
|
1151
|
-
"syn",
|
|
1152
|
-
]
|
|
1153
1150
|
|
|
1154
1151
|
[[package]]
|
|
1155
1152
|
name = "zip"
|
|
1156
|
-
version = "8.
|
|
1153
|
+
version = "8.3.0"
|
|
1157
1154
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
1158
|
-
checksum = "
|
|
1155
|
+
checksum = "4a243cfad17427fc077f529da5a95abe4e94fd2bfdb601611870a6557cc67657"
|
|
1159
1156
|
dependencies = [
|
|
1160
1157
|
"aes",
|
|
1161
1158
|
"bzip2",
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
[package]
|
|
2
2
|
name = "rustling"
|
|
3
|
-
version = "0.
|
|
3
|
+
version = "0.8.0"
|
|
4
4
|
edition = "2024"
|
|
5
5
|
description = "A blazingly fast library for computational linguistics"
|
|
6
6
|
license = "MIT"
|
|
@@ -8,21 +8,10 @@ repository = "https://github.com/jacksonllee/rustling"
|
|
|
8
8
|
documentation = "https://docs.rs/rustling"
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
keywords = [
|
|
11
|
-
|
|
11
|
+
# At most 5 keywords per crate. Each keyword must have fewer than 20 characters.
|
|
12
12
|
"linguistics",
|
|
13
|
-
"natural-language-processing",
|
|
14
13
|
"nlp",
|
|
15
14
|
"text-processing",
|
|
16
|
-
"word-segmentation",
|
|
17
|
-
"part-of-speech-tagging",
|
|
18
|
-
"language-models",
|
|
19
|
-
"ngrams",
|
|
20
|
-
"childes",
|
|
21
|
-
"talkbank",
|
|
22
|
-
"chat",
|
|
23
|
-
"averaged-perceptron",
|
|
24
|
-
"hidden-markov-model",
|
|
25
|
-
"longest-string-matching",
|
|
26
15
|
]
|
|
27
16
|
categories = ["science", "text-processing"]
|
|
28
17
|
|
|
@@ -32,7 +21,7 @@ name = "rustling"
|
|
|
32
21
|
crate-type = ["rlib"]
|
|
33
22
|
|
|
34
23
|
[dependencies]
|
|
35
|
-
pyo3 = { version = "0.28.2", features = ["abi3-py310"] }
|
|
24
|
+
pyo3 = { version = "0.28.2", features = ["abi3-py310"], optional = true }
|
|
36
25
|
rand = "0.10"
|
|
37
26
|
rayon = { version = "1.10", optional = true }
|
|
38
27
|
fancy-regex = "0.17"
|
|
@@ -44,6 +33,7 @@ serde_json = "1.0"
|
|
|
44
33
|
uuid = { version = "1", features = ["v4"] }
|
|
45
34
|
walkdir = "2"
|
|
46
35
|
zip = "8"
|
|
36
|
+
quick-xml = { version = "0.39", features = ["serialize"] }
|
|
47
37
|
zstd = "0.13"
|
|
48
38
|
|
|
49
39
|
[dev-dependencies]
|
|
@@ -51,5 +41,5 @@ tempfile = "3"
|
|
|
51
41
|
|
|
52
42
|
[features]
|
|
53
43
|
default = ["parallel"]
|
|
54
|
-
extension-module = ["pyo3/extension-module"]
|
|
44
|
+
extension-module = ["pyo3", "pyo3/extension-module"]
|
|
55
45
|
parallel = ["rayon"]
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: rustling
|
|
3
|
-
Version: 0.
|
|
4
|
-
Classifier: Development Status ::
|
|
3
|
+
Version: 0.8.0
|
|
4
|
+
Classifier: Development Status :: 3 - Alpha
|
|
5
5
|
Classifier: Intended Audience :: Developers
|
|
6
6
|
Classifier: Intended Audience :: Education
|
|
7
7
|
Classifier: Intended Audience :: Information Technology
|
|
@@ -27,64 +27,74 @@ Classifier: Topic :: Text Processing :: Indexing
|
|
|
27
27
|
Classifier: Topic :: Text Processing :: Linguistic
|
|
28
28
|
License-File: LICENSE.md
|
|
29
29
|
Summary: A blazingly fast library for computational linguistics
|
|
30
|
-
Keywords:
|
|
30
|
+
Keywords: linguistics,nlp,text-processing
|
|
31
31
|
Author-email: "Jackson L. Lee" <jacksonlunlee@gmail.com>
|
|
32
32
|
License-Expression: MIT
|
|
33
33
|
Requires-Python: >=3.10
|
|
34
34
|
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
|
|
35
|
-
Project-URL: Changelog, https://github.com/jacksonllee/rustling/blob/main/CHANGELOG.md
|
|
36
|
-
Project-URL: Python Documentation, https://rustling.readthedocs.io
|
|
37
|
-
Project-URL: Rust Documentation, https://docs.rs/rustling
|
|
38
35
|
Project-URL: Source, https://github.com/jacksonllee/rustling
|
|
39
36
|
|
|
40
|
-
<
|
|
37
|
+
<div align="center">
|
|
38
|
+
<a href="https://github.com/jacksonllee/rustling">
|
|
39
|
+
<img src="https://raw.githubusercontent.com/jacksonllee/rustling/main/python/docs/_static/logo-with-text.svg" alt="Rustling" height="120">
|
|
40
|
+
</a>
|
|
41
|
+
</div>
|
|
42
|
+
<br>
|
|
41
43
|
|
|
42
44
|
[](https://pypi.org/project/rustling/)
|
|
45
|
+
[](https://anaconda.org/conda-forge/rustling)
|
|
43
46
|
[](https://crates.io/crates/rustling)
|
|
44
47
|
|
|
45
48
|
Rustling is a blazingly fast library for computational linguistics.
|
|
46
|
-
It
|
|
49
|
+
It aims to provide flexible and efficient tools to facilitate further research.
|
|
47
50
|
|
|
48
|
-
Documentation: [Python](https://rustling.
|
|
51
|
+
Documentation: [Python](https://docs.rustling.io/) | [Rust](https://docs.rs/rustling)
|
|
49
52
|
|
|
50
|
-
|
|
53
|
+
Currently implemented features:
|
|
51
54
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
-
|
|
55
|
-
-
|
|
56
|
-
-
|
|
57
|
-
-
|
|
55
|
+
* Sequence modeling:
|
|
56
|
+
|
|
57
|
+
- N-grams and related language models
|
|
58
|
+
- Hidden Markov model
|
|
59
|
+
- Word segmentation
|
|
60
|
+
- Averaged perceptron part-of-speech tagging
|
|
61
|
+
|
|
62
|
+
* Handling richly formatted data,
|
|
63
|
+
supporting cross-format conversion as well as both local and remote sources for data ingestion:
|
|
64
|
+
|
|
65
|
+
- CHAT for TalkBank and CHILDES
|
|
66
|
+
- ELAN for annotated multimedia data
|
|
67
|
+
- TextGrid for Praat annotations
|
|
68
|
+
- CoNLL-U for University Dependencies
|
|
69
|
+
- SRT for SubRip subtitles
|
|
58
70
|
|
|
59
71
|
## Performance
|
|
60
72
|
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
| | Score | **1.9x** | NLTK |
|
|
65
|
-
| | Generate | **106--114x** | NLTK |
|
|
66
|
-
| **Word Segmentation** | LongestStringMatching | **9x** | wordseg |
|
|
67
|
-
| **POS Tagging** | Training | **5x** | NLTK |
|
|
68
|
-
| | Tagging | **18x** | NLTK |
|
|
69
|
-
| **HMM** | Fit | **13x** | hmmlearn |
|
|
70
|
-
| | Predict | **0.9x** | hmmlearn |
|
|
71
|
-
| | Score | **5x** | hmmlearn |
|
|
72
|
-
| **CHAT Parsing** | Reading from a ZIP archive | **43x** | pylangacq |
|
|
73
|
-
| | Reading from strings | **70x** | pylangacq |
|
|
74
|
-
| | Parsing utterances | **15x** | pylangacq |
|
|
75
|
-
| | Parsing tokens | **9x** | pylangacq |
|
|
76
|
-
|
|
77
|
-
See [`benchmarks/`](https://github.com/jacksonllee/rustling/tree/main/benchmarks) for reproduction scripts.
|
|
73
|
+
Rustling is highly performant because it is implemented in Rust under the hood.
|
|
74
|
+
For benchmarks comparing Rustling against other Python packages with similar functionalities,
|
|
75
|
+
please see [`benchmarks`](https://github.com/jacksonllee/rustling/tree/main/benchmarks).
|
|
78
76
|
|
|
79
77
|
|
|
80
78
|
## Installation
|
|
81
79
|
|
|
82
80
|
### Python
|
|
83
81
|
|
|
82
|
+
Using pip:
|
|
83
|
+
|
|
84
84
|
```bash
|
|
85
85
|
pip install rustling
|
|
86
86
|
```
|
|
87
87
|
|
|
88
|
+
Using conda:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
conda install -c conda-forge rustling
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
For Pyodide, pre-built WASM wheels (with multithreading disabled, as Pyodide does not support it)
|
|
95
|
+
are available from each [GitHub release](https://github.com/jacksonllee/rustling/releases)
|
|
96
|
+
— look for the ``.whl`` file with ``emscripten`` in the filename.
|
|
97
|
+
|
|
88
98
|
### Rust
|
|
89
99
|
|
|
90
100
|
```bash
|
rustling-0.8.0/README.md
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
<div align="center">
|
|
2
|
+
<a href="https://github.com/jacksonllee/rustling">
|
|
3
|
+
<img src="https://raw.githubusercontent.com/jacksonllee/rustling/main/python/docs/_static/logo-with-text.svg" alt="Rustling" height="120">
|
|
4
|
+
</a>
|
|
5
|
+
</div>
|
|
6
|
+
<br>
|
|
7
|
+
|
|
8
|
+
[](https://pypi.org/project/rustling/)
|
|
9
|
+
[](https://anaconda.org/conda-forge/rustling)
|
|
10
|
+
[](https://crates.io/crates/rustling)
|
|
11
|
+
|
|
12
|
+
Rustling is a blazingly fast library for computational linguistics.
|
|
13
|
+
It aims to provide flexible and efficient tools to facilitate further research.
|
|
14
|
+
|
|
15
|
+
Documentation: [Python](https://docs.rustling.io/) | [Rust](https://docs.rs/rustling)
|
|
16
|
+
|
|
17
|
+
Currently implemented features:
|
|
18
|
+
|
|
19
|
+
* Sequence modeling:
|
|
20
|
+
|
|
21
|
+
- N-grams and related language models
|
|
22
|
+
- Hidden Markov model
|
|
23
|
+
- Word segmentation
|
|
24
|
+
- Averaged perceptron part-of-speech tagging
|
|
25
|
+
|
|
26
|
+
* Handling richly formatted data,
|
|
27
|
+
supporting cross-format conversion as well as both local and remote sources for data ingestion:
|
|
28
|
+
|
|
29
|
+
- CHAT for TalkBank and CHILDES
|
|
30
|
+
- ELAN for annotated multimedia data
|
|
31
|
+
- TextGrid for Praat annotations
|
|
32
|
+
- CoNLL-U for University Dependencies
|
|
33
|
+
- SRT for SubRip subtitles
|
|
34
|
+
|
|
35
|
+
## Performance
|
|
36
|
+
|
|
37
|
+
Rustling is highly performant because it is implemented in Rust under the hood.
|
|
38
|
+
For benchmarks comparing Rustling against other Python packages with similar functionalities,
|
|
39
|
+
please see [`benchmarks`](https://github.com/jacksonllee/rustling/tree/main/benchmarks).
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
## Installation
|
|
43
|
+
|
|
44
|
+
### Python
|
|
45
|
+
|
|
46
|
+
Using pip:
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
pip install rustling
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Using conda:
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
conda install -c conda-forge rustling
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
For Pyodide, pre-built WASM wheels (with multithreading disabled, as Pyodide does not support it)
|
|
59
|
+
are available from each [GitHub release](https://github.com/jacksonllee/rustling/releases)
|
|
60
|
+
— look for the ``.whl`` file with ``emscripten`` in the filename.
|
|
61
|
+
|
|
62
|
+
### Rust
|
|
63
|
+
|
|
64
|
+
```bash
|
|
65
|
+
cargo add rustling
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
## License
|
|
69
|
+
|
|
70
|
+
MIT License
|