mittens 0.3.1 → 0.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +4 -0
- data/Gemfile +1 -0
- data/README.md +2 -2
- data/Rakefile +10 -3
- data/lib/mittens/version.rb +1 -1
- data/vendor/snowball/.github/workflows/ci.yml +144 -17
- data/vendor/snowball/.github/workflows/coverage.yml +117 -0
- data/vendor/snowball/.github/workflows/runtime-tests.yml +26 -0
- data/vendor/snowball/.gitignore +33 -7
- data/vendor/snowball/CONTRIBUTING.rst +7 -0
- data/vendor/snowball/COPYING +1 -1
- data/vendor/snowball/GNUmakefile +736 -298
- data/vendor/snowball/NEWS +1394 -7
- data/vendor/snowball/README.rst +8 -8
- data/vendor/snowball/ada/generate/generate.adb +5 -5
- data/vendor/snowball/ada/src/stemmer.adb +177 -188
- data/vendor/snowball/ada/src/stemmer.ads +86 -89
- data/vendor/snowball/ada/src/stemwords.adb +21 -5
- data/vendor/snowball/algorithms/czech.sbl +255 -0
- data/vendor/snowball/algorithms/danish.sbl +22 -10
- data/vendor/snowball/algorithms/english.sbl +7 -4
- data/vendor/snowball/algorithms/esperanto.sbl +6 -7
- data/vendor/snowball/algorithms/estonian.sbl +9 -4
- data/vendor/snowball/algorithms/finnish.sbl +33 -21
- data/vendor/snowball/algorithms/german.sbl +5 -0
- data/vendor/snowball/algorithms/indonesian.sbl +115 -96
- data/vendor/snowball/algorithms/italian.sbl +26 -1
- data/vendor/snowball/algorithms/lithuanian.sbl +13 -13
- data/vendor/snowball/algorithms/norwegian.sbl +17 -6
- data/vendor/snowball/algorithms/persian.sbl +387 -0
- data/vendor/snowball/algorithms/polish.sbl +177 -0
- data/vendor/snowball/algorithms/sesotho.sbl +115 -0
- data/vendor/snowball/algorithms/turkish.sbl +4 -4
- data/vendor/snowball/compiler/analyser.c +2002 -711
- data/vendor/snowball/compiler/driver.c +460 -298
- data/vendor/snowball/compiler/generator.c +439 -1965
- data/vendor/snowball/compiler/generator_ada.c +605 -430
- data/vendor/snowball/compiler/generator_c.c +2227 -0
- data/vendor/snowball/compiler/generator_csharp.c +365 -274
- data/vendor/snowball/compiler/generator_dart.c +1476 -0
- data/vendor/snowball/compiler/generator_go.c +348 -270
- data/vendor/snowball/compiler/generator_java.c +330 -233
- data/vendor/snowball/compiler/generator_js.c +534 -385
- data/vendor/snowball/compiler/generator_pascal.c +361 -320
- data/vendor/snowball/compiler/generator_php.c +1445 -0
- data/vendor/snowball/compiler/generator_python.c +399 -299
- data/vendor/snowball/compiler/generator_rust.c +370 -261
- data/vendor/snowball/compiler/generator_zig.c +1474 -0
- data/vendor/snowball/compiler/header.h +153 -86
- data/vendor/snowball/compiler/space.c +121 -63
- data/vendor/snowball/compiler/tokeniser.c +286 -182
- data/vendor/snowball/compiler/tokens.h +71 -0
- data/vendor/snowball/csharp/Snowball/Among.cs +14 -26
- data/vendor/snowball/csharp/Snowball/AssemblyInfo.cs +3 -3
- data/vendor/snowball/csharp/Snowball/Stemmer.cs +41 -45
- data/vendor/snowball/csharp/Stemwords/Program.cs +20 -14
- data/vendor/snowball/cxx/.gitignore +4 -0
- data/vendor/snowball/cxx/generate_algorithms.pl +87 -0
- data/vendor/snowball/cxx/stemmer.h +12 -0
- data/vendor/snowball/cxx/stemwords.cxx +176 -0
- data/vendor/snowball/cxx/utilities.cxx +2 -0
- data/vendor/snowball/dart/.gitignore +4 -0
- data/vendor/snowball/dart/analysis_options.yaml +1 -0
- data/vendor/snowball/dart/example/test_app.dart +65 -0
- data/vendor/snowball/dart/generate_algorithms.pl +21 -0
- data/vendor/snowball/dart/lib/snowball.dart +18 -0
- data/vendor/snowball/dart/lib/src/snowball.dart +339 -0
- data/vendor/snowball/dart/pubspec.yaml +17 -0
- data/vendor/snowball/doc/libstemmer_dart_README +37 -0
- data/vendor/snowball/doc/libstemmer_js_README +2 -2
- data/vendor/snowball/doc/libstemmer_php_README +38 -0
- data/vendor/snowball/doc/libstemmer_python_README +3 -3
- data/vendor/snowball/examples/stemwords.c +18 -3
- data/vendor/snowball/go/README.md +12 -4
- data/vendor/snowball/go/env.go +12 -14
- data/vendor/snowball/go/stemwords/main.go +2 -1
- data/vendor/snowball/java/org/tartarus/snowball/CharArraySequence.java +36 -0
- data/vendor/snowball/java/org/tartarus/snowball/SnowballProgram.java +17 -34
- data/vendor/snowball/java/org/tartarus/snowball/SnowballStemmer.java +0 -2
- data/vendor/snowball/javascript/base-stemmer.js +219 -252
- data/vendor/snowball/javascript/stemwords.js +86 -58
- data/vendor/snowball/libstemmer/mkmodules.pl +1 -1
- data/vendor/snowball/libstemmer/modules.txt +4 -0
- data/vendor/snowball/pascal/SnowballProgram.pas +18 -31
- data/vendor/snowball/php/base-stemmer.php +453 -0
- data/vendor/snowball/php/stemwords.php +25 -0
- data/vendor/snowball/python/create_init.py +16 -21
- data/vendor/snowball/python/pyproject.toml +3 -0
- data/vendor/snowball/python/setup.py +5 -7
- data/vendor/snowball/python/snowballstemmer/among.py +1 -2
- data/vendor/snowball/python/snowballstemmer/basestemmer.py +24 -33
- data/vendor/snowball/python/stemwords.py +72 -63
- data/vendor/snowball/runtime/api.c +11 -42
- data/vendor/snowball/runtime/api.h +11 -9
- data/vendor/snowball/runtime/snowball_runtime.h +110 -0
- data/vendor/snowball/runtime/utilities.c +282 -106
- data/vendor/snowball/rust/src/main.rs +2 -2
- data/vendor/snowball/rust/src/snowball/snowball_env.rs +7 -9
- data/vendor/snowball/tests/compilertest +62 -0
- data/vendor/snowball/tests/errors/ae-errors.sbl +53 -0
- data/vendor/snowball/tests/errors/ae-errors.stderr +8 -0
- data/vendor/snowball/tests/errors/bad-dollar.sbl +14 -0
- data/vendor/snowball/tests/errors/bad-dollar.stderr +4 -0
- data/vendor/snowball/tests/errors/bad-grouping-definition.sbl +9 -0
- data/vendor/snowball/tests/errors/bad-grouping-definition.stderr +6 -0
- data/vendor/snowball/tests/errors/missing-bra.sbl +6 -0
- data/vendor/snowball/tests/errors/missing-bra.stderr +3 -0
- data/vendor/snowball/tests/errors/missing-command.sbl +9 -0
- data/vendor/snowball/tests/errors/missing-command.stderr +1 -0
- data/vendor/snowball/tests/errors/missing-ket-backwardmode.sbl +3 -0
- data/vendor/snowball/tests/errors/missing-ket-backwardmode.stderr +1 -0
- data/vendor/snowball/tests/errors/missing-ket.sbl +7 -0
- data/vendor/snowball/tests/errors/missing-ket.stderr +6 -0
- data/vendor/snowball/tests/errors/notdefined.sbl +4 -0
- data/vendor/snowball/tests/errors/notdefined.stderr +2 -0
- data/vendor/snowball/tests/errors/string-omitted.sbl +5 -0
- data/vendor/snowball/tests/errors/string-omitted.stderr +1 -0
- data/vendor/snowball/tests/errors/undeclared.sbl +20 -0
- data/vendor/snowball/tests/errors/undeclared.stderr +9 -0
- data/vendor/snowball/tests/errors/unexpected-token.sbl +24 -0
- data/vendor/snowball/tests/errors/unexpected-token.stderr +17 -0
- data/vendor/snowball/tests/errors/wrongdirection.sbl +14 -0
- data/vendor/snowball/tests/errors/wrongdirection.stderr +1 -0
- data/vendor/snowball/tests/runtime/among.sbl +19 -0
- data/vendor/snowball/tests/runtime/arithmeticexpr.sbl +86 -0
- data/vendor/snowball/tests/runtime/attachinsert.sbl +47 -0
- data/vendor/snowball/tests/runtime/booleans.sbl +19 -0
- data/vendor/snowball/tests/runtime/externals.sbl +22 -0
- data/vendor/snowball/tests/runtime/hop.sbl +12 -0
- data/vendor/snowball/tests/runtime/integertests.sbl +60 -0
- data/vendor/snowball/tests/runtime/intlimits.sbl +25 -0
- data/vendor/snowball/tests/runtime/naming.sbl +20 -0
- data/vendor/snowball/tests/runtime/not.sbl +30 -0
- data/vendor/snowball/tests/runtime/or.sbl +17 -0
- data/vendor/snowball/tests/runtime/repeat.sbl +16 -0
- data/vendor/snowball/tests/runtime/setlimit.sbl +24 -0
- data/vendor/snowball/tests/runtime/sizelen.sbl +56 -0
- data/vendor/snowball/tests/runtime/slice.sbl +27 -0
- data/vendor/snowball/tests/runtime/stringdollar.sbl +39 -0
- data/vendor/snowball/tests/runtime/strings.sbl +66 -0
- data/vendor/snowball/tests/runtime/test.sbl +19 -0
- data/vendor/snowball/tests/stemtest.c +41 -1
- data/vendor/snowball/tests/syntax/canon.sbl +32 -0
- data/vendor/snowball/tests/syntax/canon.stderr +0 -0
- data/vendor/snowball/tests/syntax/canon.syntax +57 -0
- data/vendor/snowball/tests/syntax/emptyprogram.sbl +2 -0
- data/vendor/snowball/tests/syntax/emptyprogram.syntax +0 -0
- data/vendor/snowball/tests/syntax/groupings.sbl +11 -0
- data/vendor/snowball/tests/syntax/inline.sbl +144 -0
- data/vendor/snowball/tests/syntax/inline.stderr +0 -0
- data/vendor/snowball/tests/syntax/inline.syntax +121 -0
- data/vendor/snowball/tests/syntax/legacy.sbl +17 -0
- data/vendor/snowball/tests/syntax/legacy.stderr +5 -0
- data/vendor/snowball/tests/syntax/legacy.syntax +19 -0
- data/vendor/snowball/tests/syntax/loops.sbl +14 -0
- data/vendor/snowball/tests/syntax/loops.stderr +9 -0
- data/vendor/snowball/tests/syntax/loops.syntax +29 -0
- data/vendor/snowball/tests/syntax/noops.sbl +76 -0
- data/vendor/snowball/tests/syntax/noops.stderr +27 -0
- data/vendor/snowball/tests/syntax/noops.syntax +135 -0
- data/vendor/snowball/tests/syntax/simplifyae.sbl +21 -0
- data/vendor/snowball/tests/syntax/simplifyae.stderr +0 -0
- data/vendor/snowball/tests/syntax/simplifyae.syntax +39 -0
- data/vendor/snowball/tests/syntax/unused.sbl +18 -0
- data/vendor/snowball/tests/syntax/unused.stderr +7 -0
- data/vendor/snowball/tests/syntax/unused.syntax +3 -0
- data/vendor/snowball/zig/env.zig +599 -0
- data/vendor/snowball/zig/generate_algorithms.pl +19 -0
- data/vendor/snowball/zig/stemwords.zig +123 -0
- metadata +102 -4
- data/vendor/snowball/compiler/syswords.h +0 -86
- data/vendor/snowball/runtime/header.h +0 -62
data/vendor/snowball/NEWS
CHANGED
|
@@ -1,3 +1,1390 @@
|
|
|
1
|
+
Snowball 3.1.1 (2026-06-03)
|
|
2
|
+
===========================
|
|
3
|
+
|
|
4
|
+
Compiler changes
|
|
5
|
+
----------------
|
|
6
|
+
|
|
7
|
+
* Bug fixes:
|
|
8
|
+
|
|
9
|
+
+ Fix a segmentation fault after reporting an error for a string command
|
|
10
|
+
not followed by a string variable name or string literal. Bug introduced
|
|
11
|
+
in 3.1.0. Patch from Jerry James (#287).
|
|
12
|
+
|
|
13
|
+
* Compiler command-line options:
|
|
14
|
+
|
|
15
|
+
+ Emit an error for `-o -`/`-output -`. Output to stdout is not supported
|
|
16
|
+
because we need to generate multiple files for some target languages.
|
|
17
|
+
We were interpreting `-` as a base filename to append extensions to, so
|
|
18
|
+
we'd create `-.c` and `-.h` for C, but creating filenames that start with
|
|
19
|
+
`-` seems unhelpful.
|
|
20
|
+
|
|
21
|
+
Generic code generation changes
|
|
22
|
+
-------------------------------
|
|
23
|
+
|
|
24
|
+
* Bug fixes:
|
|
25
|
+
|
|
26
|
+
+ Variable localisation was failing to check the expression on the RHS of an
|
|
27
|
+
integer test for uses of a variable, so could incorrectly localise an
|
|
28
|
+
integer variable whose value should have persisted between calls to a
|
|
29
|
+
function. This bug won't realistically manifest in real world Snowball
|
|
30
|
+
code.
|
|
31
|
+
|
|
32
|
+
* Optimisations:
|
|
33
|
+
|
|
34
|
+
+ Inline some routines which are only used once. This is done for routines
|
|
35
|
+
consisting of a single non-compound command (or cases such as `not
|
|
36
|
+
<boolean>` and `goto <grouping>` which we internally synthesise a
|
|
37
|
+
non-compound command for). Localisation of variables happens after
|
|
38
|
+
inlining, so variables can now be localised in more cases.
|
|
39
|
+
|
|
40
|
+
+ `test next` and `not next` are both now simplified to a comparison between
|
|
41
|
+
`cursor` and `limit` (like `not atlimit` and `atlimit`). We already
|
|
42
|
+
normalise `hop 1` to `next`, so `test hop 1` and `not hop 1` are also
|
|
43
|
+
simplified in this way.
|
|
44
|
+
|
|
45
|
+
+ Simplify `not` applied to an integer test by removing the `not` and
|
|
46
|
+
flipping the sense of the test (e.g. `not $(x > y)` becomes `$(x <= y)`)
|
|
47
|
+
which results in simpler generated code. More usefully in real world code,
|
|
48
|
+
this also results in simpler generated code for `not atlimit` (since
|
|
49
|
+
`atlimit` is converted `$(cursor >= limit)` or `$(cursor <= limit)`
|
|
50
|
+
(depending on the current direction).
|
|
51
|
+
|
|
52
|
+
+ Canonicalise `delete` to `<-''`. 3.1.0 added a canonicalisation of `<-''`
|
|
53
|
+
to `delete`, but if we instead make `<-''` the canonical representation
|
|
54
|
+
then the generators can easily handle an empty string here specially (and
|
|
55
|
+
if they don't then the results are still correct), we don't have to handle
|
|
56
|
+
the delete token after tokenisation, and the optimisation of among when
|
|
57
|
+
all actions are `<-` plus a literal string can also fire when some actions
|
|
58
|
+
are `delete`. (This case occurs in the stemmers for arabic, czech,
|
|
59
|
+
estonian, hungarian, irish, italian, polish, porter and tamil.)
|
|
60
|
+
|
|
61
|
+
C++
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
* Code quality:
|
|
65
|
+
|
|
66
|
+
+ C++ runtime support is now fully hooked up and can be tested using
|
|
67
|
+
`make check_cxx`.
|
|
68
|
+
|
|
69
|
+
+ Don't use `extern "C"` for functions which can throw because MSVC seems to
|
|
70
|
+
treat such functions as if they were marked `noexcept`.
|
|
71
|
+
|
|
72
|
+
Javascript
|
|
73
|
+
----------
|
|
74
|
+
|
|
75
|
+
* Code quality:
|
|
76
|
+
|
|
77
|
+
+ Fix formatting of code generated by `among` of `<-` optimisation where we
|
|
78
|
+
were emitting a literal `+` rather than increasing the margin size. The
|
|
79
|
+
`+` was actually harmless in practice, but the rest of the file would be
|
|
80
|
+
misindented.
|
|
81
|
+
|
|
82
|
+
Python
|
|
83
|
+
------
|
|
84
|
+
|
|
85
|
+
* Other changes:
|
|
86
|
+
|
|
87
|
+
+ Skip classifier for Sesotho which isn't yet in the official list of
|
|
88
|
+
trove classifiers. Patch from Dmitry Shachnev (#289).
|
|
89
|
+
|
|
90
|
+
+ Add classifier to indicate support for Python 3.14.
|
|
91
|
+
|
|
92
|
+
Build system
|
|
93
|
+
------------
|
|
94
|
+
|
|
95
|
+
* The `generate` target added in 3.1.0 now overrides `FPC` and `MCS` to do
|
|
96
|
+
nothing, to avoid compiling Pascal and C# code.
|
|
97
|
+
|
|
98
|
+
* Provide way to run tests under valgrind:
|
|
99
|
+
|
|
100
|
+
make RUN_STEMWORDS='valgrind ./stemwords' check
|
|
101
|
+
|
|
102
|
+
Testsuite
|
|
103
|
+
---------
|
|
104
|
+
|
|
105
|
+
* compilertest: Require syntax tests have stderr output - we are testing errors
|
|
106
|
+
here, so we should really be checking the errors we get are the ones we
|
|
107
|
+
expect. We expect the compiler to fail, so testing stderr means we will
|
|
108
|
+
catch unrelated failures. This provides a regression test for #287.
|
|
109
|
+
|
|
110
|
+
Documentation
|
|
111
|
+
-------------
|
|
112
|
+
|
|
113
|
+
* NEWS: Fix formatting/wording of a few existing entries.
|
|
114
|
+
|
|
115
|
+
* README: List C++ as target language.
|
|
116
|
+
|
|
117
|
+
Snowball 3.1.0 (2026-05-22)
|
|
118
|
+
===========================
|
|
119
|
+
|
|
120
|
+
Compiler changes
|
|
121
|
+
----------------
|
|
122
|
+
|
|
123
|
+
* Bug fixes:
|
|
124
|
+
|
|
125
|
+
+ Fix segmentation fault if -syntax is used on a program with no code.
|
|
126
|
+
|
|
127
|
+
+ Fix segmentation fault on some assignment syntax errors.
|
|
128
|
+
|
|
129
|
+
+ Fix bug introduced in v3.0.0 with conversion of `among` starter. If there
|
|
130
|
+
were any commands after the among in the same command list then the among
|
|
131
|
+
itself would get lost. Not triggered by any current algorithms.
|
|
132
|
+
|
|
133
|
+
+ Clear name field when removing dead assignments. This is visible in the
|
|
134
|
+
syntax tree shown when command line option -syntax is used, but probably
|
|
135
|
+
doesn't affect anything otherwise.
|
|
136
|
+
|
|
137
|
+
* Compiler command-line options:
|
|
138
|
+
|
|
139
|
+
+ Using `-` for the Snowball source file is now interpreted as stdin.
|
|
140
|
+
|
|
141
|
+
+ Improve comments generated by `-comments` to show more details of the
|
|
142
|
+
corresponding Snowball code (e.g. variable names, arithmetic expressions,
|
|
143
|
+
and literal strings).
|
|
144
|
+
|
|
145
|
+
+ Add `-coverage` option which enables a code coverage feature. So far this
|
|
146
|
+
tracks which among strings and functions are exercised, and which grouping
|
|
147
|
+
characters are exercised. !
|
|
148
|
+
|
|
149
|
+
+ Support `-eprefix` for all target languages. This is easy to do and
|
|
150
|
+
provides a way to deal with externals which collide with keywords in the
|
|
151
|
+
target language. Our build system now uses `-eprefix _` for Python to make
|
|
152
|
+
the `stem` external non-public (it is called by BaseStemmer method
|
|
153
|
+
`stemWord()`) and we no longer hard-code prefixing Python externals with
|
|
154
|
+
`_`.
|
|
155
|
+
|
|
156
|
+
+ Describe more options in `--help` output.
|
|
157
|
+
|
|
158
|
+
+ Sort target language options in `--help` output.
|
|
159
|
+
|
|
160
|
+
+ The `-o` option is now optional. If not specified we now write output(s)
|
|
161
|
+
to the same filename as the first source, but with a different extension
|
|
162
|
+
(e.g. path/to/english.sbl -> path/to/english.c and path/to/english.h).
|
|
163
|
+
|
|
164
|
+
+ The `-o` option can now optionally include an extension so you can now
|
|
165
|
+
write `-c++ -o path/to/foo.cxx` instead of `-c++ -o path/to/foo`, which can
|
|
166
|
+
be more convenient (e.g. in `make` rules) and also provides an easy way to
|
|
167
|
+
specify an alternative extension (for example, `.cxx`, `.cc` and `.cpp` are
|
|
168
|
+
all extensions commonly used for C++ source code).
|
|
169
|
+
|
|
170
|
+
+ Reject `-vprefix` option for target languages which don't support it (it is
|
|
171
|
+
currently only implemented for C/C++).
|
|
172
|
+
|
|
173
|
+
* Diagnostics:
|
|
174
|
+
|
|
175
|
+
+ Clean up and improve error reporting.
|
|
176
|
+
|
|
177
|
+
+ Improve line numbers reported for some errors and warnings by using the
|
|
178
|
+
line number of an appropriate token rather than the current line number
|
|
179
|
+
of the tokeniser (which is often the line after the command being warned
|
|
180
|
+
about).
|
|
181
|
+
|
|
182
|
+
+ Improve recovery after various errors, trying to resynchronise based on
|
|
183
|
+
what's more likely, and eliminating some additional irrelevant errors
|
|
184
|
+
(including reporting the exact same error twice in some situations).
|
|
185
|
+
|
|
186
|
+
+ Emit warnings for uses of legacy Snowball language features.
|
|
187
|
+
|
|
188
|
+
+ The Snowball manual describes `integers (x)` as a declaration of `x` so we
|
|
189
|
+
now warn:
|
|
190
|
+
|
|
191
|
+
integer 'x' declared but not used
|
|
192
|
+
|
|
193
|
+
rather than:
|
|
194
|
+
|
|
195
|
+
integer 'x' defined but not used
|
|
196
|
+
|
|
197
|
+
+ 3.0.0 added a warning if the body of a `repeat` or `atleast` loop always
|
|
198
|
+
signals `t` (meaning it will loop forever which is very undesirable for a
|
|
199
|
+
stemming algorithm) or always signals `f` (meaning it will never loop,
|
|
200
|
+
which seems unlikely to be what was intended). This warning was added to
|
|
201
|
+
the C generator, but has been moved to generic code so it is now issued
|
|
202
|
+
regardless of the current target language.
|
|
203
|
+
|
|
204
|
+
+ Improve the wording of the warning if the body of a `repeat` or `atleast`
|
|
205
|
+
loop always signals 't' to explicitly say this means the loop is infinite.
|
|
206
|
+
|
|
207
|
+
+ Improve warning message for unreachable code after `not`.
|
|
208
|
+
|
|
209
|
+
+ `$x = x + 1` cleared the initialised status of x (rather than just not
|
|
210
|
+
setting it) which could lead to bogus warnings that `x` is never
|
|
211
|
+
initialised.
|
|
212
|
+
|
|
213
|
+
+ The compiler no longer exits immediately after reporting a division by zero
|
|
214
|
+
error in the Snowball code.
|
|
215
|
+
|
|
216
|
+
+ We now report a division by zero error for `$x /= 0` (this was meant to be
|
|
217
|
+
already implemented but wasn't working due to a code typo).
|
|
218
|
+
|
|
219
|
+
+ More consistent wording of "is a no-op" warnings.
|
|
220
|
+
|
|
221
|
+
+ Warn that `insert ''` and `attach ''` are no-ops (and don't generate code
|
|
222
|
+
for them).
|
|
223
|
+
|
|
224
|
+
+ Warn if a string used to define a grouping repeats characters. There's no
|
|
225
|
+
reason to do this, so it seems likely to be a typo.
|
|
226
|
+
|
|
227
|
+
+ Avoid sometimes reporting "-1 blocks unfreed".
|
|
228
|
+
|
|
229
|
+
* Optimisations:
|
|
230
|
+
|
|
231
|
+
+ Speed up processing larger Snowball programs by growing large string
|
|
232
|
+
buffers exponentially to avoid a huge number of reallocations. For
|
|
233
|
+
example, this reduced the time to compile serbian.sbl to C by about 80%!
|
|
234
|
+
|
|
235
|
+
+ Optimise reading of input file when it is seekable (which it is in typical
|
|
236
|
+
usage). Non-seekable input files are still supported.
|
|
237
|
+
|
|
238
|
+
+ Optimise writing integers when generating code. 72% of integers we write
|
|
239
|
+
are 0 to 9 and these are now written as a character. Other values are now
|
|
240
|
+
handled without a temporary buffer, avoiding a copy. This reduced the time
|
|
241
|
+
to compile serbian.sbl to C by about 8%, for example.
|
|
242
|
+
|
|
243
|
+
+ Optimise comparing among actions to find and merge equivalent actions.
|
|
244
|
+
The comparison function used for this was carefully returning a full order,
|
|
245
|
+
but actually we only need to know if the actions are equivalent or not
|
|
246
|
+
which can be tested more efficiently. For example, this reduced the time
|
|
247
|
+
to compile serbian.sbl to C by about 2%.
|
|
248
|
+
|
|
249
|
+
+ We now precompute the possible signals from each command which means this
|
|
250
|
+
is now done exactly once per command, whereas previously we could end up
|
|
251
|
+
doing it many times for some commands in some cases. The only functional
|
|
252
|
+
change should we no longer make a pessimistic assumption if the function
|
|
253
|
+
call depth reaches 100. This is cleaner but is unlikely to make a
|
|
254
|
+
difference for any real-world Snowball programs.
|
|
255
|
+
|
|
256
|
+
+ Handle possible_signals for string-$ which just passes on signals from its
|
|
257
|
+
subcommand. This doesn't affect code generation for any algorithms we
|
|
258
|
+
currently ship.
|
|
259
|
+
|
|
260
|
+
+ We now only generate function bodies to a temporary buffer for target
|
|
261
|
+
languages where we need to. This makes the code a bit clearer and reduces
|
|
262
|
+
the amount of copying of data so will make the Snowball compiler a little
|
|
263
|
+
faster. This change produces identical output for all current algorithms.
|
|
264
|
+
|
|
265
|
+
+ Tokenisation now decodes symbol tokens using switch statements. We don't
|
|
266
|
+
know the length of these tokens in advance, so the old approach of binary
|
|
267
|
+
chop on a sorted list required searching the list multiple times with
|
|
268
|
+
different possible lengths. Alphabetical tokens are still decoded by
|
|
269
|
+
binary chop.
|
|
270
|
+
|
|
271
|
+
* Code quality:
|
|
272
|
+
|
|
273
|
+
+ Remove unused routines and groupings from the program during the analysis
|
|
274
|
+
phase, which avoids each generator having to have duplicate code to skip
|
|
275
|
+
them.
|
|
276
|
+
|
|
277
|
+
+ Fix small memory leak if all uses of a name are eliminated.
|
|
278
|
+
|
|
279
|
+
+ Always use `snprintf()` instead of `sprintf()`. If the buffer passed was
|
|
280
|
+
too small we now emit an error rather than quietly using truncated output.
|
|
281
|
+
|
|
282
|
+
+ Fix GCC -Wcast-qual warnings in compiler and enable this warning by
|
|
283
|
+
default.
|
|
284
|
+
|
|
285
|
+
+ Switch to using the standard C `bool` type in the code of the compiler.
|
|
286
|
+
(The generated code still aims to require only C90.)
|
|
287
|
+
|
|
288
|
+
* Other changes:
|
|
289
|
+
|
|
290
|
+
+ Provide a simpler way to build a cut-down Snowball compiler. The
|
|
291
|
+
motivation here was to have a way to more quickly build a smaller Snowball
|
|
292
|
+
compiler which only targets C. Rather than have a DISABLE_xxx macro for
|
|
293
|
+
each language, just check if TARGET_C_ONLY is defined, and only turn off
|
|
294
|
+
the code to actually call the other generators which greatly reduces the
|
|
295
|
+
amount of conditionalisation required.
|
|
296
|
+
|
|
297
|
+
Generic code generation changes
|
|
298
|
+
-------------------------------
|
|
299
|
+
|
|
300
|
+
* Bug fixes:
|
|
301
|
+
|
|
302
|
+
+ Fix code generated for `setlimit tomark` for all target languages to
|
|
303
|
+
restore the limit correctly afterwards. The bug was not triggered by any
|
|
304
|
+
of the existing stemmers.
|
|
305
|
+
|
|
306
|
+
+ We no longer optimise repeat/atleast applied to goto/gopast on a (non)
|
|
307
|
+
grouping. This optimisation was flawed - it requires that the code in the
|
|
308
|
+
loop preserves the cursor's value on failure, but the target language
|
|
309
|
+
helper functions used here don't currently do that (they probably easily
|
|
310
|
+
could so there's scope to reinstate this optimisation).
|
|
311
|
+
|
|
312
|
+
Looking at the stemmers we ship, this affects the code generated for one
|
|
313
|
+
loop in indonesian.sbl, but it happens the cursor value is overwritten
|
|
314
|
+
immediately after this loop anyway. The bug could affect non-shipped
|
|
315
|
+
Snowball code though so isn't purely latent. This bug was introduced in
|
|
316
|
+
Snowball 3.0.0.
|
|
317
|
+
|
|
318
|
+
+ When generating target language literal strings we now always escape
|
|
319
|
+
characters which can be problematic when viewing the generated source code.
|
|
320
|
+
We always escape control characters U+007F to U+009F, non-breaking space
|
|
321
|
+
U+00A0 (visually identical to a space), and U+0590 and above (as a crude
|
|
322
|
+
way to avoid literal LTR characters in sources which can result in
|
|
323
|
+
confusing rendering).
|
|
324
|
+
|
|
325
|
+
+ Fix line numbers given to various tokens (the line numbers previously given
|
|
326
|
+
were at least of lines in the same command or the line after it). These
|
|
327
|
+
lone numbers can be seem in the target language comments generated when
|
|
328
|
+
`-comments` is used.
|
|
329
|
+
|
|
330
|
+
+ Fix warning and simplification of code when `not` is applied to a command
|
|
331
|
+
which always signals `t`. Bug introduced in v3.0.0. Fixes #271.
|
|
332
|
+
|
|
333
|
+
+ Warn and simplify `not` applied to a command which always signals `f`.
|
|
334
|
+
|
|
335
|
+
* Optimisations:
|
|
336
|
+
|
|
337
|
+
+ Add machinery to generate a Snowball variable as a local variable in the
|
|
338
|
+
target language instead of it being "global" (typically a private class
|
|
339
|
+
member in the target language). This reduces the amount of state in
|
|
340
|
+
stemmer objects, and typically reduces the overhead of accessing these
|
|
341
|
+
variables a little.
|
|
342
|
+
|
|
343
|
+
We now do this for integers and booleans in all target languages, and
|
|
344
|
+
for strings in target languages where benchmarking seems to show it
|
|
345
|
+
is faster (Dart, Go, JS, Pascal, PHP, Python).
|
|
346
|
+
|
|
347
|
+
It's done for a Snowball variable which is only used in one routine, that
|
|
348
|
+
routine doesn't (directly or indirectly) call itself, and the variable is
|
|
349
|
+
set by any code path which leads to a use of the variable. The mechanism
|
|
350
|
+
which traces the code paths errs on being too conservative in some cases,
|
|
351
|
+
but it's good enough for all instances in the code we ship, and is likely
|
|
352
|
+
to handle the vast majority of real-world cases. We issue an "info"
|
|
353
|
+
diagnostic to report when a variable which is only used in one routine
|
|
354
|
+
can't be localised - please report if you see this in real world code and
|
|
355
|
+
we can try to enhance the code path tracing.
|
|
356
|
+
|
|
357
|
+
+ Tail-calling and similar optimisations can now work for non-trivial
|
|
358
|
+
routines (previously they only worked for routines consisting of a
|
|
359
|
+
single command and not enclosed in parentheses).
|
|
360
|
+
|
|
361
|
+
+ A grouping test at the end of a routine now generates simpler code.
|
|
362
|
+
|
|
363
|
+
+ A string test at the end of a routine now generates simpler code.
|
|
364
|
+
|
|
365
|
+
+ Optimise testing a boolean (optionally preceded by `not`) when used at the
|
|
366
|
+
end of a routine.
|
|
367
|
+
|
|
368
|
+
+ Optimise an `among` with no commands at the end of a routine.
|
|
369
|
+
|
|
370
|
+
+ Generate simpler code for `not` applied to testing a boolean variable.
|
|
371
|
+
|
|
372
|
+
+ A `not` only needs to restore the cursor when its subcommand fails, so we
|
|
373
|
+
now consider whether its subcommand can modify the cursor on failure
|
|
374
|
+
(rather than whether it can modify the cursor at all). Related to #226.
|
|
375
|
+
|
|
376
|
+
+ An `or` only needs to restore the cursor when a subcommand fails, so we
|
|
377
|
+
now consider whether its subcommand can modify the cursor on failure
|
|
378
|
+
(rather than whether it can modify the cursor at all). We also now
|
|
379
|
+
consider each subcommand individually, and only emit the cursor restore for
|
|
380
|
+
those subcommands which need it. (#226)
|
|
381
|
+
|
|
382
|
+
+ Both `and` and `or` only need to restore the cursor between sub-commands
|
|
383
|
+
so no longer consider if the final subcommand might change the cursor.
|
|
384
|
+
This makes some small improvements to the generated code for a few
|
|
385
|
+
of the currently shipped algorithms. (#226)
|
|
386
|
+
|
|
387
|
+
+ Handle more commands when checking if the cursor needs restoring - this
|
|
388
|
+
improves the generated code for tamil.sbl a bit.
|
|
389
|
+
|
|
390
|
+
+ Single case amongs are now refactored to eliminate the `among` and so no
|
|
391
|
+
longer call the among machinery. Sometimes a single case among is the
|
|
392
|
+
natural way to express a single rule in Snowball code as it can show
|
|
393
|
+
commonality with rulesets with multiple rules, but it's inefficient to
|
|
394
|
+
actually generate as an among. Of the stemmers we currently ship, this
|
|
395
|
+
improves code generation for arabic, estonian, greek and lithuanian.
|
|
396
|
+
|
|
397
|
+
+ Avoid unnecessary cursor update in among helpers. We only need to update
|
|
398
|
+
the cursor on success, but were unconditionally doing so after calling an
|
|
399
|
+
among function.
|
|
400
|
+
|
|
401
|
+
+ Handle more commands in repeat_score(). None of these help code generation
|
|
402
|
+
for any currently shipped algorithms, but they are valid to optimise here.
|
|
403
|
+
|
|
404
|
+
+ Canonicalise `<-''` to `delete`.
|
|
405
|
+
|
|
406
|
+
+ Simplify some cases of compound assignment operators when the argument is
|
|
407
|
+
(or can be simplified to) a constant integer, like we already do for
|
|
408
|
+
arithmetic expressions. For example, `$x += len '{a"}' - len 'a'` is a
|
|
409
|
+
no-op when using a fixed-width encoding.
|
|
410
|
+
|
|
411
|
+
+ Canonicalise `fail C` to `false` in some cases where `C` has no
|
|
412
|
+
side-effects.
|
|
413
|
+
|
|
414
|
+
+ Removing unreachable code could leave single-entry `and`/`or` nodes
|
|
415
|
+
which could result in generating target language code with unused
|
|
416
|
+
variables. These nodes are now replaced with their subnode.
|
|
417
|
+
|
|
418
|
+
+ Eliminate `true` below `and`/`false` below `or`. These are unlikely to
|
|
419
|
+
appear verbatim in real programs, but can be created by optimisations, and
|
|
420
|
+
also can appear in runtime tests, leading to the generated target language
|
|
421
|
+
code having unused variables and/or unreachable code.
|
|
422
|
+
|
|
423
|
+
+ Canonicalise setmark, atmark and atlimit by converting `setmark x` to
|
|
424
|
+
`$x = cursor`, `atmark x` to `$(cursor == x)` and `atlimit` to either
|
|
425
|
+
`$(cursor >= limit)` or `$(cursor <= limit)` (depending on whether we're
|
|
426
|
+
in backwardmode or not). This means the target language generators have
|
|
427
|
+
three fewer commands to handle, and also gives us tail-calling of `atlimit`
|
|
428
|
+
and `atmark x` (there's a tail-callable use of `atlimit` in the Turkish
|
|
429
|
+
stemmer).
|
|
430
|
+
|
|
431
|
+
+ Remerge among actions after optimisation. It seems hard to fully move the
|
|
432
|
+
code to merge them later, but we can check for actions which have become
|
|
433
|
+
equivalent to `true` or to other actions after optimisation but before we
|
|
434
|
+
generate code.
|
|
435
|
+
|
|
436
|
+
* Code quality:
|
|
437
|
+
|
|
438
|
+
+ Find Snowball routines which are not reachable by calling an external. We
|
|
439
|
+
no longer generate code for such routines, nor for variables and groupings
|
|
440
|
+
which are only used in them, which helps to avoid "unused" warnings in the
|
|
441
|
+
generated target language code.
|
|
442
|
+
|
|
443
|
+
+ If the sub-command of `repeat`/`atleast` always signals `t` or always
|
|
444
|
+
signals `f` we now prune the rest of the current command list, and simplify
|
|
445
|
+
the command for always `f` (c_repeat -> c_do; c_atleast -> c_bra). These
|
|
446
|
+
changes help to avoid generating redundant target language code which can
|
|
447
|
+
trigger errors or warnings.
|
|
448
|
+
|
|
449
|
+
* Other changes:
|
|
450
|
+
|
|
451
|
+
+ `delete` and `<-` now update the slice end (see the "Snowball Language
|
|
452
|
+
Changes" section).
|
|
453
|
+
|
|
454
|
+
Ada
|
|
455
|
+
---
|
|
456
|
+
|
|
457
|
+
* Bug fixes:
|
|
458
|
+
|
|
459
|
+
+ Ada variable names are case-insensitive, so if two Snowball names of the
|
|
460
|
+
same type differed only by case we would generate Ada code with a name
|
|
461
|
+
collision. We now avoid such collisions by adding a counter after the type
|
|
462
|
+
code for the second and subsequent names that differ only by case.
|
|
463
|
+
|
|
464
|
+
+ Ada stemmer names are now prefixed with `S_` so `or.sbl` now generates
|
|
465
|
+
stemmer `S_or`, avoiding a name clash with an Ada keyword.
|
|
466
|
+
|
|
467
|
+
+ Fix Ada code generated `for setlimit tomark p`. This affected the
|
|
468
|
+
generated code for the Lithuanian stemmer, but it appears by luck in this
|
|
469
|
+
case the bug didn't actually affect the stemmer's output for any input.
|
|
470
|
+
|
|
471
|
+
+ Fix `setlimit` ... `repeat` bug. The generated Ada code was running the
|
|
472
|
+
code to recover from a failure inside a `repeat` loop twice due to a
|
|
473
|
+
missing line of code compared to other generators. In `backwardmode`, the
|
|
474
|
+
failure code happens to be idempotent so running it twice doesn't cause a
|
|
475
|
+
problem, but in forwards mode this results in the cursor getting double
|
|
476
|
+
adjusted if the length of the stem has changed due to insertions, deletions
|
|
477
|
+
or substitutions. None of the existing algorithms use `setlimit` in
|
|
478
|
+
forwards mode, so they're unaffected by this bug. Fixes #275.
|
|
479
|
+
|
|
480
|
+
+ Fix overcopying in string replacement code. The code to move the tail
|
|
481
|
+
up/down was copying one byte too many. We're working in a 1024 byte
|
|
482
|
+
fixed-length string buffer, and the maximum allowed input word is one byte
|
|
483
|
+
shorter, so it seems this was harmless in practice.
|
|
484
|
+
|
|
485
|
+
+ Allow characters <32 and 127 in string literals.
|
|
486
|
+
|
|
487
|
+
+ `=S` can no longer result in the slice ends becoming negative and
|
|
488
|
+
triggering a CONSTRAINT_ERROR (the slice is now specified to be unset
|
|
489
|
+
after `=S` - see the "Snowball Language Changes" section).
|
|
490
|
+
|
|
491
|
+
+ Fix Ada code generated for string-$ which was actually partly Pascal
|
|
492
|
+
code (the Ada generator was originally based on the Pascal one) and
|
|
493
|
+
didn't even compile. To fix this, Snowball string variables in Ada the
|
|
494
|
+
same way as the current string. This means they now take up more space (a
|
|
495
|
+
fixed 1KB), but a typical Snowball program has either no string variables
|
|
496
|
+
or just one so the overhead seems acceptable.
|
|
497
|
+
|
|
498
|
+
+ Fix matching of an empty string variable. This valid Snowball code would
|
|
499
|
+
trigger "failed precondition" in Ada:
|
|
500
|
+
|
|
501
|
+
externals (stem)
|
|
502
|
+
strings (s)
|
|
503
|
+
define stem as ([] ->s s)
|
|
504
|
+
|
|
505
|
+
+ Fix assumption that there's a single external called "stem".
|
|
506
|
+
|
|
507
|
+
+ Fix incorrect assumption that an among containing the empty string
|
|
508
|
+
always matched, even if the empty string had a gating function.
|
|
509
|
+
This construct is not used by any existing stemmers.
|
|
510
|
+
|
|
511
|
+
* Optimisations:
|
|
512
|
+
|
|
513
|
+
+ Avoid calling among helper when the among contains only strings which are
|
|
514
|
+
one byte long, no among functions are used, and there are no actions.
|
|
515
|
+
|
|
516
|
+
* Code quality:
|
|
517
|
+
|
|
518
|
+
+ Fix indentation of generated grouping tables.
|
|
519
|
+
|
|
520
|
+
+ Rename Context to Z in runtime code. This now matches variable naming in
|
|
521
|
+
the generated Ada code (and also the C runtime and generated C code).
|
|
522
|
+
|
|
523
|
+
+ Eliminate redundant limit check (the Skip_Utf8 helper also checks the
|
|
524
|
+
limit). Looking at the history this check is a left-over from when the
|
|
525
|
+
generated code directly incremented the cursor.
|
|
526
|
+
|
|
527
|
+
+ Emit Ada literal strings without redundant empty strings between adjacent
|
|
528
|
+
escaped bytes.
|
|
529
|
+
|
|
530
|
+
+ Generate dummy loop around `or`, which allows us to handle a sub-command
|
|
531
|
+
succeeding with Ada `exit` rather than `goto`, which seems clearer.
|
|
532
|
+
|
|
533
|
+
+ Avoid creating unused labels. This is just a cosmetic improvements - there
|
|
534
|
+
are no longer mysterious gaps in the numbering of labels in the generated
|
|
535
|
+
code.
|
|
536
|
+
|
|
537
|
+
+ Avoid generating unreachable `exit`.
|
|
538
|
+
|
|
539
|
+
* Other changes:
|
|
540
|
+
|
|
541
|
+
+ Implement support for `?` (debug command). The code we generate for this
|
|
542
|
+
case is gnat-specific, but previously the code generated didn't compile so
|
|
543
|
+
working with one implementation seems a step forwards. The `?` command can
|
|
544
|
+
now be used to debug Ada, and someone with actual Ada knowledge can now
|
|
545
|
+
more easily step in and provide a portable replacement.
|
|
546
|
+
|
|
547
|
+
C/C++
|
|
548
|
+
-----
|
|
549
|
+
|
|
550
|
+
* Bug fixes:
|
|
551
|
+
|
|
552
|
+
+ Maintain invariant that the C variable corresponding to a Snowball string
|
|
553
|
+
variable is non-NULL. Previously we would release and NULL out the entry
|
|
554
|
+
in some error cases, but elsewhere the code was assuming the value was
|
|
555
|
+
non-NULL.
|
|
556
|
+
|
|
557
|
+
+ Fix invalid code generated for `setlimit`. This doesn't happen for
|
|
558
|
+
`setlimit tomark` (which is the only way `setlimit` is used in the stemmers
|
|
559
|
+
we currently ship. Bug introduced in v3.0.0.
|
|
560
|
+
|
|
561
|
+
+ Fix codegen for `hop` with constant argument. We were relying on the
|
|
562
|
+
cursor being restored on failure by the code which handled that failure,
|
|
563
|
+
but if that code is a repeat or atleast command that it has an optimisation
|
|
564
|
+
which assumes `hop` won't do this. This means we generated incorrect C
|
|
565
|
+
code for some cases where `hop` was used inside `repeat` or `atleast`.
|
|
566
|
+
This doesn't functionally affect any of the stemmers we currently ship.
|
|
567
|
+
Bug introduced in v2.1.0.
|
|
568
|
+
|
|
569
|
+
+ Fix bug in code generated when `-vprefix` is specified, introduced in
|
|
570
|
+
Snowball 2.1.0.
|
|
571
|
+
|
|
572
|
+
+ Fix incorrect assumption that an among containing the empty string
|
|
573
|
+
always matched, even if the empty string had a gating function.
|
|
574
|
+
This construct is not used by any existing stemmers.
|
|
575
|
+
|
|
576
|
+
* Optimisations:
|
|
577
|
+
|
|
578
|
+
+ Rework how non-localised variables are stored, which eliminates an
|
|
579
|
+
indirection on every access to such a variable, and also avoids some extra
|
|
580
|
+
allocations (one if a stemmer has any non-localised integer or boolean
|
|
581
|
+
variables, and another if the stemmer has any string variables). So it
|
|
582
|
+
uses a bit less memory, it makes creating and destroying a stemmer faster,
|
|
583
|
+
and it also makes stemming a bit faster (though only by ~0.1% for the
|
|
584
|
+
English stemmer on our sample vocabulary). The `-vprefix` option now
|
|
585
|
+
generates getter functions rather than using macro magic, which means the
|
|
586
|
+
syntax for accessing Snowball variables from C has changed.
|
|
587
|
+
|
|
588
|
+
+ We now maintain the invariant that SN_env's p member is non-NULL, which
|
|
589
|
+
simplifies the runtime code.
|
|
590
|
+
|
|
591
|
+
+ We now have a specialised implementation of the slice_del() runtime helper.
|
|
592
|
+
Deleting the slice is a fairly common operation, and can be done more
|
|
593
|
+
simply than via a generic replace_s() with an empty replacement string.
|
|
594
|
+
This speeds up the English stemmer by about 1% on our test vocabulary.
|
|
595
|
+
|
|
596
|
+
+ Avoid calling among helper when the among contains only strings which are
|
|
597
|
+
one byte long, no among functions are used, and there are no actions.
|
|
598
|
+
|
|
599
|
+
+ Only fetch SIZE() in replace_s() if we need it.
|
|
600
|
+
|
|
601
|
+
+ Don't return adjustment from replace_s() runtime helper since calculating
|
|
602
|
+
the adjustment in the one caller where we actually want it is just one
|
|
603
|
+
integer addition and one integer subtraction, and that turns out to be
|
|
604
|
+
slightly more efficient as well as simpler.
|
|
605
|
+
|
|
606
|
+
+ Move check for negative hop from runtime to generated code. This means we
|
|
607
|
+
can omit it for hop with a constant argument, which is all uses of hop in
|
|
608
|
+
the stemmers we currently ship.
|
|
609
|
+
|
|
610
|
+
* Code quality:
|
|
611
|
+
|
|
612
|
+
+ The generated header is now included from the generated C/C++ source file
|
|
613
|
+
(which seems cleaner than the previous approach of generating the same
|
|
614
|
+
prototypes in the header and source file).
|
|
615
|
+
|
|
616
|
+
+ The implementation of among functions has changed. Previously we stored
|
|
617
|
+
a function pointer in struct among, but that requires relocation when the
|
|
618
|
+
code is in a dynamic library, which adds load-time overhead and means
|
|
619
|
+
the among structures can't be put in a read-only section.
|
|
620
|
+
|
|
621
|
+
We now store an integer index instead, and pass in a pointer to
|
|
622
|
+
a dispatcher function when calling the find_among()/find_among_b() helper
|
|
623
|
+
which gets called when this index is non-zero. The value of the index is
|
|
624
|
+
stored in z->af so the dispatcher function can use it.
|
|
625
|
+
|
|
626
|
+
If only one unique function is used in an among, we can just pass this to
|
|
627
|
+
find_among() as the dispatcher which reduces the overhead for this common
|
|
628
|
+
case.
|
|
629
|
+
|
|
630
|
+
Profiling with cachegrind suggests this change adds a small overhead
|
|
631
|
+
to algorithms which use among functions - currently finnish and hindi
|
|
632
|
+
(and also lovins, but that's really only of academic interest and is not
|
|
633
|
+
enabled by default).
|
|
634
|
+
|
|
635
|
+
+ Avoid long string in C source. C90 only guarantees support for literal
|
|
636
|
+
strings up to 509 characters. Fixes GCC -Woverlength-strings warning.
|
|
637
|
+
|
|
638
|
+
+ Avoid C23 feature in C runtime code, introduced in Snowball 3.0.0.
|
|
639
|
+
Initialising with empty braces was only standardised in C23 (though
|
|
640
|
+
seems to be widely supported as an extension).
|
|
641
|
+
|
|
642
|
+
+ Fix code generated for `setlimit` to be C90. Bug introduced in v3.0.0, but
|
|
643
|
+
isn't triggered by any of the stemmers we currently ship.
|
|
644
|
+
|
|
645
|
+
+ Fix -Wshadow warning for nested string-$ use. We were generating code
|
|
646
|
+
using a C variable with the fixed name `failure` - now an integer suffix is
|
|
647
|
+
appended, and we only emit the variable in cases where the subcommand
|
|
648
|
+
signal isn't known at compile time.
|
|
649
|
+
|
|
650
|
+
+ Generate `do {`...`} while (0)` around `or` code, which allows us to handle
|
|
651
|
+
a sub-command succeeding with `break;` rather than having to use `goto`,
|
|
652
|
+
which reduces the number of labels used and makes the generated code a bit
|
|
653
|
+
easier to follow.
|
|
654
|
+
|
|
655
|
+
+ C comments are now generated for `(` and `do` when `-comments` is used.
|
|
656
|
+
|
|
657
|
+
+ We now generate `+=` or `-=` for `hop <constant>` (instead of something
|
|
658
|
+
like `z->c = z->c + 2`). The C compiler should treat both the same, but it
|
|
659
|
+
arguably makes the generated code a little clearer.
|
|
660
|
+
|
|
661
|
+
* Other changes:
|
|
662
|
+
|
|
663
|
+
+ C++: The `-c++` option used to generate exactly the same code as for C,
|
|
664
|
+
except with extension `.cc` instead of `.c` but now:
|
|
665
|
+
|
|
666
|
+
- C++ classes are generated.
|
|
667
|
+
- C++ `bool` is used for Snowball booleans.
|
|
668
|
+
- Loop variables are declared inside `for (`...`)`.
|
|
669
|
+
- Allocation failures and internal errors (e.g. slice_check() failing)
|
|
670
|
+
throw a C++ exception - this is a bit simpler and more efficient that the
|
|
671
|
+
C code approach of returning -1 which then has to be checked for and
|
|
672
|
+
propagated through the generated code.
|
|
673
|
+
|
|
674
|
+
+ Snowball's debug command (`?`) now works out of the box (previously you
|
|
675
|
+
have to adjust a `#if 0` preprocessor conditional in the runtime code).
|
|
676
|
+
|
|
677
|
+
+ Rename `runtime/header.h` (which really seems too generic, and is also easy
|
|
678
|
+
to confuse with `compiler/header.h`) to `runtime/snowball_runtime.h`. We
|
|
679
|
+
expect most users will be using the C stemmers through libstemmer and so
|
|
680
|
+
won't be affected by this.
|
|
681
|
+
|
|
682
|
+
C#
|
|
683
|
+
--
|
|
684
|
+
|
|
685
|
+
* Bug fixes:
|
|
686
|
+
|
|
687
|
+
+ Fix code generated for `<-s`. This is not used by any of the stemmers we
|
|
688
|
+
currently ship. Test case based on one from ajroetker in #270.
|
|
689
|
+
|
|
690
|
+
+ Fix code generated for string-$. This feature is not used by any of the
|
|
691
|
+
stemmers we currently ship.
|
|
692
|
+
|
|
693
|
+
+ Fix assumption that there's a single external called "stem".
|
|
694
|
+
|
|
695
|
+
* Optimisations:
|
|
696
|
+
|
|
697
|
+
+ Use Debug.Assert() in slice_check() runtime helper. Previously the runtime
|
|
698
|
+
code wrote a diagnostic message and continued if one of these checks
|
|
699
|
+
failed, but failures should only happen with a Snowball program containing
|
|
700
|
+
logic errors, or for bugs in the Snowball compiler or its runtime (or
|
|
701
|
+
possibly in the C# compiler, runtime, OS, hardware, etc). Therefore an
|
|
702
|
+
assertion seems an appropriate choice, and means the check is not enabled
|
|
703
|
+
for a production build, which seems more helpful overall. See #242.
|
|
704
|
+
|
|
705
|
+
* Code quality:
|
|
706
|
+
|
|
707
|
+
+ Eliminate duplicates from groupings. We currently implement these for C#
|
|
708
|
+
with a linear string search, and a side-effect of this change is that the
|
|
709
|
+
grouping string is now sorted, which will affect the time taken to look
|
|
710
|
+
up different characters in an arbitrary way (none of the Snowball sources
|
|
711
|
+
seem to try to list characters in frequency order). Really C# should be
|
|
712
|
+
fixed to use an O(1) lookup like other target languages.
|
|
713
|
+
|
|
714
|
+
+ The implementation of among functions has changed. We now store an integer
|
|
715
|
+
index in the Among class, and pass a dispatcher function to the among
|
|
716
|
+
helper method.
|
|
717
|
+
|
|
718
|
+
If only one unique function is used in an among, we can just pass this to
|
|
719
|
+
the helper method as the dispatcher which reduces the overhead for this
|
|
720
|
+
common case.
|
|
721
|
+
|
|
722
|
+
Crude profiling with `time make check_csharp` suggests this doesn't harm
|
|
723
|
+
performance (perhaps a little faster, but maybe just within the noise).
|
|
724
|
+
|
|
725
|
+
The main benefit is all Among arrays can now be static, which previously
|
|
726
|
+
we wasn't possible for those which used among functions (#146).
|
|
727
|
+
|
|
728
|
+
+ Remove unused return value from Stemmer.Replace() runtime helper.
|
|
729
|
+
|
|
730
|
+
+ Fix inaccurate doc comments on runtime functions.
|
|
731
|
+
|
|
732
|
+
* Other changes:
|
|
733
|
+
|
|
734
|
+
+ csharp_stemwords: Speed up output to stdout.
|
|
735
|
+
|
|
736
|
+
+ csharp_stemwords: Don't write the chosen stemmer to stdout. This is not
|
|
737
|
+
really useful information, and breaks sending the stemmed words to stdout
|
|
738
|
+
because they're preceded by extra output.
|
|
739
|
+
|
|
740
|
+
+ csharp_stemwords: Try to open input before output so we don't leave an
|
|
741
|
+
empty output file behind if we can't open the input file.
|
|
742
|
+
|
|
743
|
+
Go
|
|
744
|
+
--
|
|
745
|
+
|
|
746
|
+
* Bug fixes:
|
|
747
|
+
|
|
748
|
+
+ Fix code generated for non-constant hop. A non-constant integer expression
|
|
749
|
+
has type `int` in the generated Go code, but the hop helpers expected
|
|
750
|
+
`int32`. For a constant hop this worked because Go integer literals are
|
|
751
|
+
untyped, so will convert to `int32`. To fix this, the helpers now take
|
|
752
|
+
`int` instead of `int32`.
|
|
753
|
+
|
|
754
|
+
+ Fix code generated if `minint` or `maxint` is used. In this case we were
|
|
755
|
+
generating `use std::usize;` near the start of the Go code, but that's
|
|
756
|
+
actually Rust code and a hangover from the Go backend being originally
|
|
757
|
+
based on the Rust one.
|
|
758
|
+
|
|
759
|
+
+ The Go code generated for `->` was incorrectly signalling `f` if the
|
|
760
|
+
slice was empty. Luckily this case is not exercised by any current
|
|
761
|
+
algorithms. See #242.
|
|
762
|
+
|
|
763
|
+
+ Fix code generated for string-$ (which isn't used by any of the algorithms
|
|
764
|
+
we currently ship).
|
|
765
|
+
|
|
766
|
+
+ A snowball `external` could not previously be called from within the
|
|
767
|
+
Snowball program. This is allowed by the Snowball language, but none of
|
|
768
|
+
the shipped stemmers do this, and it's unlikely any stemmer would. Perhaps
|
|
769
|
+
it's useful if you use Snowball for other string-processing tasks.
|
|
770
|
+
|
|
771
|
+
+ Fix handling of `minint` and `maxint` - we were generating some code copied
|
|
772
|
+
verbatim from the Rust generator for this case which was not valid Go.
|
|
773
|
+
(These are not used by any of the algorithms we currently ship.)
|
|
774
|
+
|
|
775
|
+
* Optimisations:
|
|
776
|
+
|
|
777
|
+
+ Reuse `env` in stemwords which is measurably faster than creating a new one
|
|
778
|
+
for every word.
|
|
779
|
+
|
|
780
|
+
* Code quality:
|
|
781
|
+
|
|
782
|
+
+ Eliminate unnecessary semicolons from generated code.
|
|
783
|
+
|
|
784
|
+
+ Fix formatting of generated code. The code gets run through gofmt which
|
|
785
|
+
was fixing up these issues, but better to generate the code cleanly to
|
|
786
|
+
start with. The only things which gofmt now changes are that it indents
|
|
787
|
+
variable names to align in adjacent variable declarations, and a couple of
|
|
788
|
+
things which are apparently for compatibility with older versions of Go.
|
|
789
|
+
|
|
790
|
+
+ Runtime helpers SliceDel() and SliceFrom() always returned true, but the
|
|
791
|
+
generated code included failure checks in case false was returned.
|
|
792
|
+
These helpers no longer return anything, and the checks are gone.
|
|
793
|
+
|
|
794
|
+
* Documentation:
|
|
795
|
+
|
|
796
|
+
+ Recommend that users reuse an `env` since this is measurably faster than
|
|
797
|
+
creating a new one for every word.
|
|
798
|
+
|
|
799
|
+
* Other changes:
|
|
800
|
+
|
|
801
|
+
+ Remove `-gopackage` option from compiler. Use `-package`/`-P` instead
|
|
802
|
+
(`-gopackage` has just been an alias for these since Snowball 2.0.0).
|
|
803
|
+
|
|
804
|
+
Java
|
|
805
|
+
----
|
|
806
|
+
|
|
807
|
+
* Bug fixes:
|
|
808
|
+
|
|
809
|
+
+ Generate correct Java code for ASCII control chars in string literals.
|
|
810
|
+
|
|
811
|
+
+ Fix code generated for string-$. As part of this fix, we now use char[]
|
|
812
|
+
for string variables as well as the current string, which makes it much
|
|
813
|
+
simpler to switch to working on a string variable and back. Fixes #252.
|
|
814
|
+
|
|
815
|
+
+ Fix assumption that there's a single external called "stem".
|
|
816
|
+
|
|
817
|
+
* Other changes:
|
|
818
|
+
|
|
819
|
+
+ The generated Java classes no longer implement Serializable. This support
|
|
820
|
+
was added in 2016, but in 2026 this approach to serialization in Java is
|
|
821
|
+
apparently no longer used due to security problems. Fixes #255.
|
|
822
|
+
|
|
823
|
+
Javascript
|
|
824
|
+
----------
|
|
825
|
+
|
|
826
|
+
* Bug fixes:
|
|
827
|
+
|
|
828
|
+
+ Fix `->` to work when the slice is empty - previously it incorrectly
|
|
829
|
+
signalled `f` for this case. Luckily this case is not exercised by any
|
|
830
|
+
current algorithms (#242)
|
|
831
|
+
|
|
832
|
+
+ Generate public functions for all externals. Patch from simlrh (#258).
|
|
833
|
+
|
|
834
|
+
+ Fix code generated for string-$
|
|
835
|
+
|
|
836
|
+
* Optimisations:
|
|
837
|
+
|
|
838
|
+
+ Use startsWith()/endsWith() in eq_s()/eq_s_b(). This is quite a bit faster
|
|
839
|
+
as it avoids slice() creating a temporary string (e.g. measured a reduction
|
|
840
|
+
of ~17% wallclock time for tamil on the test vocabulary, taking the fastest
|
|
841
|
+
of 5 runs before and after).
|
|
842
|
+
|
|
843
|
+
+ Optimise among when all actions are `<-` with a literal string. We now
|
|
844
|
+
generate a single call to slice_from() with the argument obtained by
|
|
845
|
+
indexing into an array of literal strings. This is perhaps faster, albeit
|
|
846
|
+
not by much, but it definitely results in smaller code, which is helpful
|
|
847
|
+
for in browser use. See #227.
|
|
848
|
+
|
|
849
|
+
+ The substring_i member in the Among class is now an offset from the current
|
|
850
|
+
index, and now zero in the common case where there's not another string
|
|
851
|
+
which is a sub-prefix/sub-suffix. We've also swapped the order of elements
|
|
852
|
+
so we can omit this in the common case when it is zero and there's no among
|
|
853
|
+
function). This reduces the size of the generated Javascript code (even
|
|
854
|
+
after minification). Fixes #236.
|
|
855
|
+
|
|
856
|
+
+ Change slice_check() to assert its conditions. In C we must not perform
|
|
857
|
+
string slicing if slice_check() fails because that could result in writing
|
|
858
|
+
outside of the allocated buffer, but it's not problematic in this way for
|
|
859
|
+
Javascript, and the situations which slice_check() checks for should only
|
|
860
|
+
happen with a Snowball program containing logic errors, or for bugs in the
|
|
861
|
+
Snowball compiler or its runtime (or possibly in the Javascript
|
|
862
|
+
interpreter, OS, hardware, etc). Therefore assert() seems an appropriate
|
|
863
|
+
choice.
|
|
864
|
+
|
|
865
|
+
* Code quality:
|
|
866
|
+
|
|
867
|
+
+ Convert to using Javascript modules and classes. The way among functions
|
|
868
|
+
are called has been reworked to allow this, copying the approach now used
|
|
869
|
+
for C and C# (#234, #240). Patches from Adam Turner and Titus Ng.
|
|
870
|
+
|
|
871
|
+
+ Adjust generated code to work with deno, and suppress a few deno warnings
|
|
872
|
+
which are hard to avoid in generated code.
|
|
873
|
+
|
|
874
|
+
+ Avoid generating blocks around failure handling. The failure handle code
|
|
875
|
+
is always a single statement (and if we ever needed more than a statement
|
|
876
|
+
for some situation then we could arrange to add a block for just those
|
|
877
|
+
situations). This significantly reduces the size of the generated JS code.
|
|
878
|
+
|
|
879
|
+
+ Always inline code for `=>`. The code is not much longer than the
|
|
880
|
+
call to a helper function in BaseStemmer. Also in 3.0.0 we deprecated `=>`
|
|
881
|
+
and nothing we ship contains this command, so removing it from BaseStemmer
|
|
882
|
+
reduces the total code size a little.
|
|
883
|
+
|
|
884
|
+
+ Rename BaseStemmer's internal `cursor` property to `c`. Unfortunately,
|
|
885
|
+
`cursor` is a DOM property, so Javascript minifiers are cautious about
|
|
886
|
+
renaming it to avoid breaking code. The name `c` matches the naming we use
|
|
887
|
+
for C, Ada and Pascal.
|
|
888
|
+
|
|
889
|
+
+ Generate smaller code for hop by constant. All current uses of hop in the
|
|
890
|
+
stemmers we ship have a constant argument, so avoid using a temporary
|
|
891
|
+
variable in these cases.
|
|
892
|
+
|
|
893
|
+
+ Optimise `+=1` to `++`, `-=1` to `--`. These are a byte shorter, and it
|
|
894
|
+
seems Javascript minifiers don't do this for us because it's not a safe
|
|
895
|
+
transformation unless the minifier can deduce that the variable can't hold
|
|
896
|
+
a string.
|
|
897
|
+
|
|
898
|
+
+ Improve temporary var naming and use. These variables don't need unique
|
|
899
|
+
generated names now we're declaring them as `const` which has more sensible
|
|
900
|
+
scoping rules than `var`.
|
|
901
|
+
|
|
902
|
+
+ Generate smaller code for `insert` and string-`=`. In some cases we know
|
|
903
|
+
we have the value of member variable `this.cursor` in local `const c` so
|
|
904
|
+
use the latter instead.
|
|
905
|
+
|
|
906
|
+
+ Use triple equality for Javascript. Patch from Adam Turner.
|
|
907
|
+
|
|
908
|
+
+ Fix position of grouping type comment which is now placed consistently with
|
|
909
|
+
other type comments.
|
|
910
|
+
|
|
911
|
+
+ Use `a` instead of `among_var` in generated code. This reduces the size of
|
|
912
|
+
the generated code, which is helpful if a minification step isn't being
|
|
913
|
+
used.
|
|
914
|
+
|
|
915
|
+
+ Consistently cuddle braces in runtime code. The style wasn't entirely
|
|
916
|
+
consistent before, and cuddling braces matches the generated Javascript
|
|
917
|
+
code and the Snowball C code.
|
|
918
|
+
|
|
919
|
+
+ Generate block around case to bound the scope of `const` and `let` within
|
|
920
|
+
the case.
|
|
921
|
+
|
|
922
|
+
+ Use `let` in README example.
|
|
923
|
+
|
|
924
|
+
+ Use `let` consistently in stemwords.js.
|
|
925
|
+
|
|
926
|
+
+ Initialise integer Snowball variables - we annotate them as being type
|
|
927
|
+
"number" so we shouldn't let them have value undefined. Patch from Adam
|
|
928
|
+
Turner.
|
|
929
|
+
|
|
930
|
+
+ Improve/fix typescript annotations in runtime and generated code.
|
|
931
|
+
|
|
932
|
+
+ Annotate runtime with @ts-expect-error. It doesn't seem to be possible to
|
|
933
|
+
express the types fully in some places, but the invariants we require are
|
|
934
|
+
ensured by the Snowball compiler. Annotating the expected errors allows
|
|
935
|
+
unexpected type checking errors to be be more easily seen, and they are
|
|
936
|
+
now fatal is CI.
|
|
937
|
+
|
|
938
|
+
+ Use `===` and `!==` in stemwords.js. Patch from Adam Turner.
|
|
939
|
+
|
|
940
|
+
* Other changes:
|
|
941
|
+
|
|
942
|
+
+ Make stemmer subclasses anonymous and export them by default. This makes
|
|
943
|
+
creating a stemmer object easier as you only need to build the filename of
|
|
944
|
+
the stemmer subclass, and not also its class name.
|
|
945
|
+
|
|
946
|
+
+ Adjust interpretation of `-parentclassname` option. We supply the JS
|
|
947
|
+
snowball runtime so being able to specify a different base class name
|
|
948
|
+
doesn't seem very useful, so instead interpret this as the name to import
|
|
949
|
+
the base class as in generated stemmers. It now defaults to just `B` which
|
|
950
|
+
reduces the size of the generated stemmer code a little (even after running
|
|
951
|
+
it through most Javascript minification tools).
|
|
952
|
+
|
|
953
|
+
+ Improve stemwords.js option parsing. Make `-i` and `-o` optional to match
|
|
954
|
+
other target language versions of stemwords. Eliminate the check that
|
|
955
|
+
there are at least 3 command line arguments as we don't require any now.
|
|
956
|
+
If we encounter an argument we don't understand, we now report it and show
|
|
957
|
+
the usage message (previously we silently ignored it). We now exit with
|
|
958
|
+
status 1 if there's a problem parsing the command line.
|
|
959
|
+
|
|
960
|
+
+ stemwords.js: Emit help message in one console.log. Patch from Titus Ng
|
|
961
|
+
(#221).
|
|
962
|
+
|
|
963
|
+
Pascal
|
|
964
|
+
------
|
|
965
|
+
|
|
966
|
+
* Bug fixes:
|
|
967
|
+
|
|
968
|
+
+ We were generating invalid Pascal code when tail-calling or calling a
|
|
969
|
+
routine which always fails. Neither case is currently exercised by any
|
|
970
|
+
stemmers we ship and generate Pascal code for (the Pascal generator
|
|
971
|
+
currently only supports iso-8859-1).
|
|
972
|
+
|
|
973
|
+
+ Fix code generated for string-$ (which isn't used by any of the algorithms
|
|
974
|
+
we currently ship).
|
|
975
|
+
|
|
976
|
+
+ Fix assumption that there's a single external called "stem".
|
|
977
|
+
|
|
978
|
+
* Code quality:
|
|
979
|
+
|
|
980
|
+
+ Merge EqS and EqV runtime functions. We can get the length of a Pascal
|
|
981
|
+
AnsiString `s` cheaply with `Length(s)` so there isn't a need to pass in
|
|
982
|
+
the length in the string literal case.
|
|
983
|
+
|
|
984
|
+
+ Eliminate `While` in code generated for `repeat`/`atleast`. Pascal lacks
|
|
985
|
+
`Continue` (at least as a standard feature) and this loop only exists so we
|
|
986
|
+
can jump back to its start with `continue` in other languages - we have a
|
|
987
|
+
`Break;` at its end so it doesn't loop in the normal way. In Pascal we
|
|
988
|
+
generate a label before the loop and use `goto` to continue iterating, so
|
|
989
|
+
we can get rid of the Pascal loop entirely.
|
|
990
|
+
|
|
991
|
+
+ Use `Break` instead of `Goto` in code generated for `go`/`gopast`.
|
|
992
|
+
|
|
993
|
+
+ Generate dummy loop around `or` so we can handle a sub-command succeeding
|
|
994
|
+
with Pascal `Break` rather than `Goto`, which seems clearer.
|
|
995
|
+
|
|
996
|
+
+ Avoid generating `Repeat` ... `Until True` dummy loops which are not
|
|
997
|
+
actually needed.
|
|
998
|
+
|
|
999
|
+
+ Fix problem introduced in v3.0.0 with formatting of code generated for
|
|
1000
|
+
`go`/`gopast` applied to a grouping.
|
|
1001
|
+
|
|
1002
|
+
+ Switch to a simpler name mangling system. Pascal variable names are
|
|
1003
|
+
case-insensitive but Snowball names are case-sensitive. We used to address
|
|
1004
|
+
this by encoding the case of letters into a prefix on the name but that can
|
|
1005
|
+
generate long and ugly names in some cases (e.g. integer Foo_Bar ->
|
|
1006
|
+
IUllU_Foo_Bar). We now avoid collisions by adding a counter after the
|
|
1007
|
+
type code for the second and subsequent names that differ only by case
|
|
1008
|
+
(so Foo_Bar is only mangled if there's another integer which differs
|
|
1009
|
+
only by case which is declared before it, and even then just becomes
|
|
1010
|
+
something like I2_Foo_Bar).
|
|
1011
|
+
|
|
1012
|
+
+ Emit Pascal literal strings without redundant empty strings between
|
|
1013
|
+
adjacent escaped bytes.
|
|
1014
|
+
|
|
1015
|
+
+ The -comments option now includes the values of string literals, so has
|
|
1016
|
+
been changed to generate "rest of line" comments (starting `//`) rather
|
|
1017
|
+
than block comments (delimited by `{` ... `}`) so that string literals
|
|
1018
|
+
containing `}` don't need escaping. We were already using `//` comments in
|
|
1019
|
+
the Pascal runtime so this shouldn't harm portability.
|
|
1020
|
+
|
|
1021
|
+
Python
|
|
1022
|
+
------
|
|
1023
|
+
|
|
1024
|
+
* Bug fixes:
|
|
1025
|
+
|
|
1026
|
+
+ Fix `algorithms()` when forwarding to PyStemmer. It looks like this has
|
|
1027
|
+
never worked as the code has been like this since it was merged, and we
|
|
1028
|
+
were forwarding to a method which PyStemmer doesn't provide and never seems
|
|
1029
|
+
to have provided.
|
|
1030
|
+
|
|
1031
|
+
+ stemwords.py: Make -i and -o optional. The command syntax already
|
|
1032
|
+
suggested they were, but actually we gave an error if they were omitted.
|
|
1033
|
+
|
|
1034
|
+
+ Fix code generated for string-$ (which isn't used by any of the algorithms
|
|
1035
|
+
we currently ship).
|
|
1036
|
+
|
|
1037
|
+
+ Fix `->` to work when the slice is empty - previously it incorrectly
|
|
1038
|
+
signalled `f` for this case. Luckily this case is not exercised by any
|
|
1039
|
+
current algorithms (#242)
|
|
1040
|
+
|
|
1041
|
+
+ Remove deprecated licence classifier which now triggers a deprecation
|
|
1042
|
+
warning from Python's setuptools. We already specify the licensing in the
|
|
1043
|
+
now preferred way via `license=` with a SPDX licence expression.
|
|
1044
|
+
|
|
1045
|
+
* Optimisations:
|
|
1046
|
+
|
|
1047
|
+
+ Optimise single-character string literal checks in the same way we already
|
|
1048
|
+
do for C. This seems to be measurably faster (tested with Turkish which
|
|
1049
|
+
has lots of single character literal tests).
|
|
1050
|
+
|
|
1051
|
+
+ Groupings are now implemented via a Python set, or a string for small
|
|
1052
|
+
groupings.
|
|
1053
|
+
|
|
1054
|
+
+ Eliminate use of exception in code generated for `or`. We can instead wrap
|
|
1055
|
+
the code in a loop and use `break`.
|
|
1056
|
+
|
|
1057
|
+
+ Eliminate use of exception in `goto` and `gopast`. We can just use `break`
|
|
1058
|
+
here to exit the `while` loop we're also inside and move the `except` from
|
|
1059
|
+
the previous `try` onto the `while`.
|
|
1060
|
+
|
|
1061
|
+
+ Avoid using a temporary for `hop` with a constant argument as benchmarking
|
|
1062
|
+
with timeit shows this is faster.
|
|
1063
|
+
|
|
1064
|
+
+ Optimise string test by using startswith()/endswith() with suitable
|
|
1065
|
+
start/end parameters which avoids creating a temporary substring and avoids
|
|
1066
|
+
an explicit limit check. This speeds up artificial testcases consisting of
|
|
1067
|
+
`goto 'the'` by 10%.
|
|
1068
|
+
|
|
1069
|
+
+ Optimise among when all actions are `<-` with a literal string. We now
|
|
1070
|
+
generate a single call to slice_from() with the argument obtained by
|
|
1071
|
+
indexing into an array of literal strings. See #227.
|
|
1072
|
+
|
|
1073
|
+
+ Reduce overhead of code to forward to PyStemmer, both when forwarding and
|
|
1074
|
+
when using the pure Python stemmers.
|
|
1075
|
+
|
|
1076
|
+
+ Reuse exception classes much more. This reduces the number of labN classes
|
|
1077
|
+
we need by 142 over all the current stemmers.
|
|
1078
|
+
|
|
1079
|
+
+ Change slice_check() to assert its conditions. In C we must not perform
|
|
1080
|
+
string slicing if slice_check() fails because that could result in writing
|
|
1081
|
+
outside of the allocated buffer, but it's not problematic in this way for
|
|
1082
|
+
Python, and the situations which slice_check() checks for should only
|
|
1083
|
+
happen with a Snowball program containing logic errors, or for bugs in the
|
|
1084
|
+
Snowball compiler or its runtime (or possibly in the Python interpreter,
|
|
1085
|
+
OS, hardware, etc). Therefore assert() seems an appropriate choice.
|
|
1086
|
+
|
|
1087
|
+
* Code quality:
|
|
1088
|
+
|
|
1089
|
+
+ Use _ as dummy loop variable. We don't use the loop variable's value, and
|
|
1090
|
+
the loop itself tracks the current iteration so generating nested loops
|
|
1091
|
+
using `_` as the loop variable works correctly.
|
|
1092
|
+
|
|
1093
|
+
+ Avoid mysterious gaps in the numbering of variables in the generated code.
|
|
1094
|
+
This was already done for the other languages, but I missed Python it
|
|
1095
|
+
seems.
|
|
1096
|
+
|
|
1097
|
+
+ Avoid generating unused lab0 class for a Snowball program which doesn't use
|
|
1098
|
+
any failure labels.
|
|
1099
|
+
|
|
1100
|
+
+ Avoid generating a blank line at start of the body of a Snowball `loop`.
|
|
1101
|
+
|
|
1102
|
+
+ stemwords.py: Replace deprecated `codecs.open()` with built-in `open()`.
|
|
1103
|
+
Patch from Dmitry Shachnev.
|
|
1104
|
+
|
|
1105
|
+
* Documentation:
|
|
1106
|
+
|
|
1107
|
+
+ Remove unnecessary semicolons from Python code in docs.
|
|
1108
|
+
|
|
1109
|
+
* Other changes:
|
|
1110
|
+
|
|
1111
|
+
+ Remove Python 2 support. We stopped officially supporting it in Snowball
|
|
1112
|
+
2.1.0, but now we've actually stripped out support. Versions of Python ≥
|
|
1113
|
+
3.3 continue to be supported. Patch from Dmitry Shachnev (#212).
|
|
1114
|
+
|
|
1115
|
+
Rust
|
|
1116
|
+
----
|
|
1117
|
+
|
|
1118
|
+
* Bug fixes:
|
|
1119
|
+
|
|
1120
|
+
+ Fix code generated for string-$ (which isn't used by any of the algorithms
|
|
1121
|
+
we currently ship).
|
|
1122
|
+
|
|
1123
|
+
+ A snowball `external` could not previously be called from within the
|
|
1124
|
+
Snowball program. This is allowed by the Snowball language, but none of
|
|
1125
|
+
the shipped stemmers do this, and it's unlikely any stemmer would, but
|
|
1126
|
+
perhaps it's useful if you use Snowball for other string-processing tasks.
|
|
1127
|
+
|
|
1128
|
+
+ The generated code previously treated an empty string returned by
|
|
1129
|
+
slice_to() as an error, but this was buggy since if the slice is empty
|
|
1130
|
+
the return value will be an empty string. The helper doesn't try
|
|
1131
|
+
to signal an error with an empty string so we can just drop this
|
|
1132
|
+
check. Luckily this case is not exercised by any current algorithms.
|
|
1133
|
+
See #242.
|
|
1134
|
+
|
|
1135
|
+
+ Fix incorrect assumption that an among containing the empty string
|
|
1136
|
+
always matched, even if the empty string had a gating function.
|
|
1137
|
+
This construct is not used by any existing stemmers.
|
|
1138
|
+
|
|
1139
|
+
* Optimisations:
|
|
1140
|
+
|
|
1141
|
+
+ Avoid calling among helper when the among contains only strings which are
|
|
1142
|
+
one byte long, no among functions are used, and there are no actions.
|
|
1143
|
+
|
|
1144
|
+
* Code quality:
|
|
1145
|
+
|
|
1146
|
+
+ Fix formatting of code generated for `goto`/`gopast` applied to a grouping
|
|
1147
|
+
or inverted grouping. This is just a cosmetic problem - functionally it
|
|
1148
|
+
was correct. The poor formatting was introduced in v3.0.0.
|
|
1149
|
+
|
|
1150
|
+
+ Runtime helpers slice_del() and slice_from() always returned true, but
|
|
1151
|
+
the generated code included failure checks in case false was returned.
|
|
1152
|
+
These helpers no longer return anything, and the checks are gone.
|
|
1153
|
+
|
|
1154
|
+
+ Generate space after condition in integer test (purely cosmetic).
|
|
1155
|
+
|
|
1156
|
+
New Code Generators
|
|
1157
|
+
-------------------
|
|
1158
|
+
|
|
1159
|
+
* Add Dart generator from Ryan Heise (#156, #250).
|
|
1160
|
+
|
|
1161
|
+
* Add PHP generator from Tim Whitlock and Olly Betts (#243). Requires PHP 8.3
|
|
1162
|
+
or later, which allows us to use typed class constants.
|
|
1163
|
+
|
|
1164
|
+
* Add Zig backend from AJ Roetker. Requires Zig 0.16.0 or later.
|
|
1165
|
+
|
|
1166
|
+
Snowball Language Changes
|
|
1167
|
+
-------------------------
|
|
1168
|
+
|
|
1169
|
+
* `delete` and `<-` now update the slice end. The manual said that after
|
|
1170
|
+
`[` and `]` "the slice ends will retain the same values until altered",
|
|
1171
|
+
which doesn't make it clear what happens for operations which modify the
|
|
1172
|
+
text the slice ends are in.
|
|
1173
|
+
|
|
1174
|
+
The existing handling here was inconsistent between commands: `delete`
|
|
1175
|
+
and `<-` left the slice ends on the same numeric positions, while
|
|
1176
|
+
`attach` and `insert` adjusted the slice ends to leave the slice marking
|
|
1177
|
+
the equivalent substring of the updated string. When working in UTF-8
|
|
1178
|
+
the slice end could end up in the middle of a multi-byte character after
|
|
1179
|
+
`delete` or `<-`, which seems especially undesirable.
|
|
1180
|
+
|
|
1181
|
+
I talked this over with Martin Porter and we've agreed that it makes
|
|
1182
|
+
sense for `delete` and `<-` to also update the slice ends (in fact only
|
|
1183
|
+
the right end needs adjusting) and I've clarified the wording in the
|
|
1184
|
+
manual.
|
|
1185
|
+
|
|
1186
|
+
Existing algorithms we ship don't rely on what the slice is set to after
|
|
1187
|
+
these commands.
|
|
1188
|
+
|
|
1189
|
+
* The slice is now specified to be unset after `=S` (so the same state as at
|
|
1190
|
+
the start of the program). Previously Snowball attempted to adjust the slice
|
|
1191
|
+
after `=S`, but there isn't an obvious adjustment in general because it can
|
|
1192
|
+
replace part of the content of the slice. Martin said he'd not thought of
|
|
1193
|
+
this case, and we've concluded it's best to adjust the Snowball language
|
|
1194
|
+
definition.
|
|
1195
|
+
|
|
1196
|
+
New stemming algorithms
|
|
1197
|
+
-----------------------
|
|
1198
|
+
|
|
1199
|
+
* Add Czech stemmer from Olly Betts and Jim O’Regan (#151).
|
|
1200
|
+
|
|
1201
|
+
* Add Persian (Farsi) stemmer from Saeid Darvish (#181).
|
|
1202
|
+
|
|
1203
|
+
* Add Polish stemmer from Dmitry Shachnev (#245).
|
|
1204
|
+
|
|
1205
|
+
* Add Sesotho stemmer from Kamohelo Lebjane (#260).
|
|
1206
|
+
|
|
1207
|
+
Behavioural changes to existing algorithms
|
|
1208
|
+
------------------------------------------
|
|
1209
|
+
|
|
1210
|
+
* Danish:
|
|
1211
|
+
|
|
1212
|
+
+ Adjust to handle apostrophe (#187).
|
|
1213
|
+
|
|
1214
|
+
+ Restrict undoubling to valid cases. Coverage showed that a number of the
|
|
1215
|
+
consonants we would undouble never occur in our Danish vocabulary. Testing
|
|
1216
|
+
a larger list didn't find any matches for Danish words either, so restrict
|
|
1217
|
+
the undoubling which reduces the potential for damage to foreign words and
|
|
1218
|
+
should be a little more efficient.
|
|
1219
|
+
|
|
1220
|
+
* English:
|
|
1221
|
+
|
|
1222
|
+
+ Restore exception for `skis` so it stems to `ski`. This reverts a
|
|
1223
|
+
change made erroneously in Snowball 3.0.0.
|
|
1224
|
+
|
|
1225
|
+
+ Improve the stemming of some words starting `inter`:
|
|
1226
|
+
- We now avoid conflating intern, internal, international and
|
|
1227
|
+
internment.
|
|
1228
|
+
- We now conflate interfere/interferes/interference with
|
|
1229
|
+
interfered/interfering.
|
|
1230
|
+
- The stem of `interval` is now `interval` rather than `interv`, which
|
|
1231
|
+
is mostly a cosmetic change as no unrelated words stem to `interv`.
|
|
1232
|
+
|
|
1233
|
+
* Estonian:
|
|
1234
|
+
|
|
1235
|
+
+ Handle apostrophe (#187).
|
|
1236
|
+
|
|
1237
|
+
* Finnish:
|
|
1238
|
+
|
|
1239
|
+
+ Handle apostrophe (#187).
|
|
1240
|
+
|
|
1241
|
+
+ Improve fallback from illative rules. If a word ends -han, -hen, -hin,
|
|
1242
|
+
-hon, -hän or -hön but the vowel before does not match we were not removing
|
|
1243
|
+
a suffix in case_ending, we now fallback to handling as a genitive and
|
|
1244
|
+
remove -n. This changes how we handle about 90 words - almost all for the
|
|
1245
|
+
better, most of the rest seem neutral changes.
|
|
1246
|
+
|
|
1247
|
+
+ Allow "ø" to match with -hön as this is seen with Norwegian place names,
|
|
1248
|
+
e.g. Bodøhön.
|
|
1249
|
+
|
|
1250
|
+
+ Remove illative form -hun. This improves the stemming of 14 words in
|
|
1251
|
+
our test vocabulary.
|
|
1252
|
+
|
|
1253
|
+
* German:
|
|
1254
|
+
|
|
1255
|
+
+ Handle apostrophe (#187).
|
|
1256
|
+
|
|
1257
|
+
* Italian:
|
|
1258
|
+
|
|
1259
|
+
+ Handle elisions (#187).
|
|
1260
|
+
|
|
1261
|
+
* Lithuanian:
|
|
1262
|
+
|
|
1263
|
+
+ Don't remove -er- before normal suffixes. These aren't real grammatic
|
|
1264
|
+
suffixes and seem to have been included mainly to try to conflate ancient
|
|
1265
|
+
forms of the Lithuanian word for "sister" (e.g. "sesers") with modern forms
|
|
1266
|
+
(e.g. "sesė"). We weren't even doing a complete job there however as
|
|
1267
|
+
"seserimis" and "seseris" were not handled. Removing these suffixes
|
|
1268
|
+
entirely means we no longer try to conflate the ancient and modern forms
|
|
1269
|
+
here, but at least all the forms of the old word get grouped, as do all
|
|
1270
|
+
forms of the new word. The stemming for ~150 other words is also improved,
|
|
1271
|
+
without obvious downsides. Patch from Justas Sakalauskas (#263).
|
|
1272
|
+
|
|
1273
|
+
+ Remove trailing apostrophe as final step - an apostrophe is sometimes used
|
|
1274
|
+
to separate a Lithuanian ending on an international word (#187).
|
|
1275
|
+
|
|
1276
|
+
* Norwegian:
|
|
1277
|
+
|
|
1278
|
+
+ Adjust to handle apostrophe (#187).
|
|
1279
|
+
|
|
1280
|
+
* Polish:
|
|
1281
|
+
|
|
1282
|
+
+ Remove optional apostrophe after removing suffix. Polish uses an
|
|
1283
|
+
apostrophe to separate loanwords from native suffixes. (The correct use is
|
|
1284
|
+
to mark the elision of the final sound of a loanword before a Polish
|
|
1285
|
+
inflectional endings, but it's also often used with any loanword) (#187).
|
|
1286
|
+
|
|
1287
|
+
Optimisations to existing algorithms
|
|
1288
|
+
------------------------------------
|
|
1289
|
+
|
|
1290
|
+
* English:
|
|
1291
|
+
|
|
1292
|
+
+ Optimise -eed, -eedly handling by performing the much cheaper R1 check
|
|
1293
|
+
before the among of exceptional cases.
|
|
1294
|
+
|
|
1295
|
+
* Esperanto:
|
|
1296
|
+
|
|
1297
|
+
+ Eliminate use of among functions. It's easy to avoid them, and they come
|
|
1298
|
+
with a performance overhead in some target languages. For C, the new
|
|
1299
|
+
version is 0.09% faster (from cachegrind estimated cycle count).
|
|
1300
|
+
|
|
1301
|
+
* Indonesian:
|
|
1302
|
+
|
|
1303
|
+
+ Avoid use of among functions, which gives a 1.9% speed up for C (from
|
|
1304
|
+
cachegrind estimated cycle count).
|
|
1305
|
+
|
|
1306
|
+
* Lithuanian:
|
|
1307
|
+
|
|
1308
|
+
+ Minor simplification/optimisation by relying on Snowball restoring the
|
|
1309
|
+
cursor on failure.
|
|
1310
|
+
|
|
1311
|
+
* Turkish:
|
|
1312
|
+
|
|
1313
|
+
+ Simplify `not test C` to just `not C`. If C succeeds, then the `not` fails
|
|
1314
|
+
and the cursor will get restored by whatever handles that signal.
|
|
1315
|
+
|
|
1316
|
+
Code clarity improvements to existing algorithms
|
|
1317
|
+
------------------------------------------------
|
|
1318
|
+
|
|
1319
|
+
* Finnish: Rename `V1` and `LONG` to match the names used in the algorithm
|
|
1320
|
+
description on the website.
|
|
1321
|
+
|
|
1322
|
+
* Italian: Eliminate use of legacy among starter.
|
|
1323
|
+
|
|
1324
|
+
Build system
|
|
1325
|
+
------------
|
|
1326
|
+
|
|
1327
|
+
* The default flags used with `ar` are now `-cr` instead of `-cru`. Many
|
|
1328
|
+
Linux distros configure `ar` to use option `D` (deterministic mode) by
|
|
1329
|
+
default, which was triggering a warning that option `u` is ignored.
|
|
1330
|
+
Option `u` is just a minor optimisation for the case where the archive
|
|
1331
|
+
already exist and only some object files have change, so it seems best
|
|
1332
|
+
to just not try to use it and avoid the warning.
|
|
1333
|
+
|
|
1334
|
+
Make variable `ARFLAGS` can now be used to specify flags to use with `ar`,
|
|
1335
|
+
so if you want to continue using `-cru`, you can use:
|
|
1336
|
+
|
|
1337
|
+
make ARFLAGS=-cru
|
|
1338
|
+
|
|
1339
|
+
If `D` is on by default in your `ar`, you'll actually want:
|
|
1340
|
+
|
|
1341
|
+
make ARFLAGS=-cruU
|
|
1342
|
+
|
|
1343
|
+
* Add comment documenting how to use iconv.py (simple pure-Python alternative
|
|
1344
|
+
which allows running the testsuite without iconv installed).
|
|
1345
|
+
|
|
1346
|
+
* `make clean` now removes all built files for all target languages, and is
|
|
1347
|
+
now tested by CI to ensure this doesn't regress.
|
|
1348
|
+
|
|
1349
|
+
* Make "make check_utf8" parallel-safe by avoiding writing the stemmed output
|
|
1350
|
+
to disk by default (except for Arabic). To get the output saved as tmp.txt
|
|
1351
|
+
on error for debugging you can now use: `make SAVETMP=1 check_utf8`. Patch
|
|
1352
|
+
from Adam Turner (#237, #238).
|
|
1353
|
+
|
|
1354
|
+
* Ada: Fix parallel build by adding missing dependency from .adb to the
|
|
1355
|
+
corresponding .sbl file (#237, #238).
|
|
1356
|
+
|
|
1357
|
+
* Go: Use `$(go)` for `go generate` as well.
|
|
1358
|
+
|
|
1359
|
+
* Python: Omit output "(THIN_FACTOR=)" if set empty.
|
|
1360
|
+
|
|
1361
|
+
* Add SNOWBALL_FLAGS, intended to allow passing options such as `-comments`
|
|
1362
|
+
and `-coverage` during development and debugging.
|
|
1363
|
+
|
|
1364
|
+
* Add make targets to assist comparing generated code before and after a
|
|
1365
|
+
compiler change: `baseline-create`, `generate` and `baseline-diff`.
|
|
1366
|
+
|
|
1367
|
+
* We now have CI testing that the Snowball compiler builds as C99 (we were
|
|
1368
|
+
already testing that the generated C code builds as C90). Fixes #283,
|
|
1369
|
+
reported by Domingo Alvarez Duarte.
|
|
1370
|
+
|
|
1371
|
+
Testsuite
|
|
1372
|
+
---------
|
|
1373
|
+
|
|
1374
|
+
* New testsuite for the Snowball compiler which tests parsing, errors and
|
|
1375
|
+
warnings.
|
|
1376
|
+
|
|
1377
|
+
* New runtime testsuite which tests the implementation of Snowball language
|
|
1378
|
+
features in each supported target language. These provide something much
|
|
1379
|
+
more like a proper set of unit tests rather than relying on checking all the
|
|
1380
|
+
algorithms produce the expected output to validate all the target language
|
|
1381
|
+
generators. These tests are run with -comments on to provide some test
|
|
1382
|
+
coverage for this option. Fixes #157.
|
|
1383
|
+
|
|
1384
|
+
* stemtest: Add more number testcases, relocated to here from finnish/voc.txt.
|
|
1385
|
+
They're better by stemtest as we want to avoid any stemmer damaging numbers,
|
|
1386
|
+
and testcases here can easily be run for all stemmers.
|
|
1387
|
+
|
|
1
1388
|
Snowball 3.0.1 (2025-05-09)
|
|
2
1389
|
===========================
|
|
3
1390
|
|
|
@@ -323,15 +1710,15 @@ Behavioural changes to existing algorithms
|
|
|
323
1710
|
contain other accented characters and it seems better for the stemmer to
|
|
324
1711
|
handle such words the same way regardless of the encoding in use.
|
|
325
1712
|
|
|
326
|
-
* English: Replace '-ogist' with '-og' to conflate "geologist" and "geology",
|
|
327
|
-
Suggested by Marc Schipperheijn on snowball-discuss.
|
|
1713
|
+
* English: Replace '-ogist' with '-og' to conflate "geologist" and "geology",
|
|
1714
|
+
etc. Suggested by Marc Schipperheijn on snowball-discuss.
|
|
328
1715
|
|
|
329
1716
|
* English: Add extra condition to undoubling. We no longer undouble if the
|
|
330
1717
|
double consonant is preceded by exactly "a", "e" or "o" to avoid conflating
|
|
331
1718
|
"add"/"ad", "egg"/"eg", "off"/"of", etc. Fixes #182, reported by Ed Page.
|
|
332
1719
|
|
|
333
|
-
* English: Avoid conflating 'emerge' and 'emergency'. Reported by Frederick
|
|
334
|
-
on snowball-discuss.
|
|
1720
|
+
* English: Avoid conflating 'emerge' and 'emergency'. Reported by Frederick
|
|
1721
|
+
Ross on snowball-discuss.
|
|
335
1722
|
|
|
336
1723
|
* English: Avoid conflating 'evening' and 'even'. Reported by Ann B on
|
|
337
1724
|
snowball-discuss.
|
|
@@ -429,7 +1816,7 @@ Behavioural changes to existing algorithms
|
|
|
429
1816
|
|
|
430
1817
|
* Spanish: Handle -acion like -ación and -ucion like -ución. It's apparently
|
|
431
1818
|
common to miss off accents in Spanish, and there are examples in our test
|
|
432
|
-
vocabulary that these
|
|
1819
|
+
vocabulary that these changes help. Proposed by Damian Janowski.
|
|
433
1820
|
|
|
434
1821
|
* Swedish: Replace suffix "öst" with "ös" when preceded by any of 'iklnprtuv'
|
|
435
1822
|
rather than just 'l'. The new rule only requires the "öst" to be in R1
|
|
@@ -653,7 +2040,7 @@ Compiler
|
|
|
653
2040
|
in the case where we were in a stringdef (but correct if we weren't).
|
|
654
2041
|
|
|
655
2042
|
* Eliminate special handling for among starter. We now convert the starter
|
|
656
|
-
to be a command before the among, adding an
|
|
2043
|
+
to be a command before the among, adding an explicit substring if there
|
|
657
2044
|
isn't one.
|
|
658
2045
|
|
|
659
2046
|
* We now warn if the body of a `repeat` or `atleast` loop always signals
|
|
@@ -831,7 +2218,7 @@ Code generation improvements
|
|
|
831
2218
|
+ Constant numeric subexpressions and constant numeric tests are now
|
|
832
2219
|
evaluated at Snowball compile time.
|
|
833
2220
|
|
|
834
|
-
+ Simplify the following
|
|
2221
|
+
+ Simplify the following degenerate `loop` and `atleast` constructs where
|
|
835
2222
|
N is a compile-time constant:
|
|
836
2223
|
|
|
837
2224
|
- loop N C where N <= 0 is a no-op.
|