mittens 0.3.0 → 0.3.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (174) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +8 -0
  3. data/Gemfile +2 -1
  4. data/README.md +2 -2
  5. data/Rakefile +12 -5
  6. data/ext/mittens/extconf.rb +3 -1
  7. data/lib/mittens/version.rb +1 -1
  8. data/vendor/snowball/.github/workflows/ci.yml +144 -17
  9. data/vendor/snowball/.github/workflows/coverage.yml +117 -0
  10. data/vendor/snowball/.github/workflows/runtime-tests.yml +26 -0
  11. data/vendor/snowball/.gitignore +33 -7
  12. data/vendor/snowball/CONTRIBUTING.rst +7 -0
  13. data/vendor/snowball/COPYING +1 -1
  14. data/vendor/snowball/GNUmakefile +736 -298
  15. data/vendor/snowball/NEWS +1394 -7
  16. data/vendor/snowball/README.rst +8 -8
  17. data/vendor/snowball/ada/generate/generate.adb +5 -5
  18. data/vendor/snowball/ada/src/stemmer.adb +177 -188
  19. data/vendor/snowball/ada/src/stemmer.ads +86 -89
  20. data/vendor/snowball/ada/src/stemwords.adb +21 -5
  21. data/vendor/snowball/algorithms/czech.sbl +255 -0
  22. data/vendor/snowball/algorithms/danish.sbl +22 -10
  23. data/vendor/snowball/algorithms/english.sbl +7 -4
  24. data/vendor/snowball/algorithms/esperanto.sbl +6 -7
  25. data/vendor/snowball/algorithms/estonian.sbl +9 -4
  26. data/vendor/snowball/algorithms/finnish.sbl +33 -21
  27. data/vendor/snowball/algorithms/german.sbl +5 -0
  28. data/vendor/snowball/algorithms/indonesian.sbl +115 -96
  29. data/vendor/snowball/algorithms/italian.sbl +26 -1
  30. data/vendor/snowball/algorithms/lithuanian.sbl +13 -13
  31. data/vendor/snowball/algorithms/norwegian.sbl +17 -6
  32. data/vendor/snowball/algorithms/persian.sbl +387 -0
  33. data/vendor/snowball/algorithms/polish.sbl +177 -0
  34. data/vendor/snowball/algorithms/sesotho.sbl +115 -0
  35. data/vendor/snowball/algorithms/turkish.sbl +4 -4
  36. data/vendor/snowball/compiler/analyser.c +2002 -711
  37. data/vendor/snowball/compiler/driver.c +460 -298
  38. data/vendor/snowball/compiler/generator.c +439 -1965
  39. data/vendor/snowball/compiler/generator_ada.c +605 -430
  40. data/vendor/snowball/compiler/generator_c.c +2227 -0
  41. data/vendor/snowball/compiler/generator_csharp.c +365 -274
  42. data/vendor/snowball/compiler/generator_dart.c +1476 -0
  43. data/vendor/snowball/compiler/generator_go.c +348 -270
  44. data/vendor/snowball/compiler/generator_java.c +330 -233
  45. data/vendor/snowball/compiler/generator_js.c +534 -385
  46. data/vendor/snowball/compiler/generator_pascal.c +361 -320
  47. data/vendor/snowball/compiler/generator_php.c +1445 -0
  48. data/vendor/snowball/compiler/generator_python.c +399 -299
  49. data/vendor/snowball/compiler/generator_rust.c +370 -261
  50. data/vendor/snowball/compiler/generator_zig.c +1474 -0
  51. data/vendor/snowball/compiler/header.h +153 -86
  52. data/vendor/snowball/compiler/space.c +121 -63
  53. data/vendor/snowball/compiler/tokeniser.c +286 -182
  54. data/vendor/snowball/compiler/tokens.h +71 -0
  55. data/vendor/snowball/csharp/Snowball/Among.cs +14 -26
  56. data/vendor/snowball/csharp/Snowball/AssemblyInfo.cs +3 -3
  57. data/vendor/snowball/csharp/Snowball/Stemmer.cs +41 -45
  58. data/vendor/snowball/csharp/Stemwords/Program.cs +20 -14
  59. data/vendor/snowball/cxx/.gitignore +4 -0
  60. data/vendor/snowball/cxx/generate_algorithms.pl +87 -0
  61. data/vendor/snowball/cxx/stemmer.h +12 -0
  62. data/vendor/snowball/cxx/stemwords.cxx +176 -0
  63. data/vendor/snowball/cxx/utilities.cxx +2 -0
  64. data/vendor/snowball/dart/.gitignore +4 -0
  65. data/vendor/snowball/dart/analysis_options.yaml +1 -0
  66. data/vendor/snowball/dart/example/test_app.dart +65 -0
  67. data/vendor/snowball/dart/generate_algorithms.pl +21 -0
  68. data/vendor/snowball/dart/lib/snowball.dart +18 -0
  69. data/vendor/snowball/dart/lib/src/snowball.dart +339 -0
  70. data/vendor/snowball/dart/pubspec.yaml +17 -0
  71. data/vendor/snowball/doc/libstemmer_dart_README +37 -0
  72. data/vendor/snowball/doc/libstemmer_js_README +2 -2
  73. data/vendor/snowball/doc/libstemmer_php_README +38 -0
  74. data/vendor/snowball/doc/libstemmer_python_README +3 -3
  75. data/vendor/snowball/examples/stemwords.c +18 -3
  76. data/vendor/snowball/go/README.md +12 -4
  77. data/vendor/snowball/go/env.go +12 -14
  78. data/vendor/snowball/go/stemwords/main.go +2 -1
  79. data/vendor/snowball/java/org/tartarus/snowball/CharArraySequence.java +36 -0
  80. data/vendor/snowball/java/org/tartarus/snowball/SnowballProgram.java +17 -34
  81. data/vendor/snowball/java/org/tartarus/snowball/SnowballStemmer.java +0 -2
  82. data/vendor/snowball/javascript/base-stemmer.js +219 -252
  83. data/vendor/snowball/javascript/stemwords.js +86 -58
  84. data/vendor/snowball/libstemmer/mkmodules.pl +1 -1
  85. data/vendor/snowball/libstemmer/modules.txt +4 -0
  86. data/vendor/snowball/pascal/SnowballProgram.pas +18 -31
  87. data/vendor/snowball/php/base-stemmer.php +453 -0
  88. data/vendor/snowball/php/stemwords.php +25 -0
  89. data/vendor/snowball/python/create_init.py +16 -21
  90. data/vendor/snowball/python/pyproject.toml +3 -0
  91. data/vendor/snowball/python/setup.py +5 -7
  92. data/vendor/snowball/python/snowballstemmer/among.py +1 -2
  93. data/vendor/snowball/python/snowballstemmer/basestemmer.py +24 -33
  94. data/vendor/snowball/python/stemwords.py +72 -63
  95. data/vendor/snowball/runtime/api.c +11 -42
  96. data/vendor/snowball/runtime/api.h +11 -9
  97. data/vendor/snowball/runtime/snowball_runtime.h +110 -0
  98. data/vendor/snowball/runtime/utilities.c +282 -106
  99. data/vendor/snowball/rust/src/main.rs +2 -2
  100. data/vendor/snowball/rust/src/snowball/snowball_env.rs +7 -9
  101. data/vendor/snowball/tests/compilertest +62 -0
  102. data/vendor/snowball/tests/errors/ae-errors.sbl +53 -0
  103. data/vendor/snowball/tests/errors/ae-errors.stderr +8 -0
  104. data/vendor/snowball/tests/errors/bad-dollar.sbl +14 -0
  105. data/vendor/snowball/tests/errors/bad-dollar.stderr +4 -0
  106. data/vendor/snowball/tests/errors/bad-grouping-definition.sbl +9 -0
  107. data/vendor/snowball/tests/errors/bad-grouping-definition.stderr +6 -0
  108. data/vendor/snowball/tests/errors/missing-bra.sbl +6 -0
  109. data/vendor/snowball/tests/errors/missing-bra.stderr +3 -0
  110. data/vendor/snowball/tests/errors/missing-command.sbl +9 -0
  111. data/vendor/snowball/tests/errors/missing-command.stderr +1 -0
  112. data/vendor/snowball/tests/errors/missing-ket-backwardmode.sbl +3 -0
  113. data/vendor/snowball/tests/errors/missing-ket-backwardmode.stderr +1 -0
  114. data/vendor/snowball/tests/errors/missing-ket.sbl +7 -0
  115. data/vendor/snowball/tests/errors/missing-ket.stderr +6 -0
  116. data/vendor/snowball/tests/errors/notdefined.sbl +4 -0
  117. data/vendor/snowball/tests/errors/notdefined.stderr +2 -0
  118. data/vendor/snowball/tests/errors/string-omitted.sbl +5 -0
  119. data/vendor/snowball/tests/errors/string-omitted.stderr +1 -0
  120. data/vendor/snowball/tests/errors/undeclared.sbl +20 -0
  121. data/vendor/snowball/tests/errors/undeclared.stderr +9 -0
  122. data/vendor/snowball/tests/errors/unexpected-token.sbl +24 -0
  123. data/vendor/snowball/tests/errors/unexpected-token.stderr +17 -0
  124. data/vendor/snowball/tests/errors/wrongdirection.sbl +14 -0
  125. data/vendor/snowball/tests/errors/wrongdirection.stderr +1 -0
  126. data/vendor/snowball/tests/runtime/among.sbl +19 -0
  127. data/vendor/snowball/tests/runtime/arithmeticexpr.sbl +86 -0
  128. data/vendor/snowball/tests/runtime/attachinsert.sbl +47 -0
  129. data/vendor/snowball/tests/runtime/booleans.sbl +19 -0
  130. data/vendor/snowball/tests/runtime/externals.sbl +22 -0
  131. data/vendor/snowball/tests/runtime/hop.sbl +12 -0
  132. data/vendor/snowball/tests/runtime/integertests.sbl +60 -0
  133. data/vendor/snowball/tests/runtime/intlimits.sbl +25 -0
  134. data/vendor/snowball/tests/runtime/naming.sbl +20 -0
  135. data/vendor/snowball/tests/runtime/not.sbl +30 -0
  136. data/vendor/snowball/tests/runtime/or.sbl +17 -0
  137. data/vendor/snowball/tests/runtime/repeat.sbl +16 -0
  138. data/vendor/snowball/tests/runtime/setlimit.sbl +24 -0
  139. data/vendor/snowball/tests/runtime/sizelen.sbl +56 -0
  140. data/vendor/snowball/tests/runtime/slice.sbl +27 -0
  141. data/vendor/snowball/tests/runtime/stringdollar.sbl +39 -0
  142. data/vendor/snowball/tests/runtime/strings.sbl +66 -0
  143. data/vendor/snowball/tests/runtime/test.sbl +19 -0
  144. data/vendor/snowball/tests/stemtest.c +41 -1
  145. data/vendor/snowball/tests/syntax/canon.sbl +32 -0
  146. data/vendor/snowball/tests/syntax/canon.stderr +0 -0
  147. data/vendor/snowball/tests/syntax/canon.syntax +57 -0
  148. data/vendor/snowball/tests/syntax/emptyprogram.sbl +2 -0
  149. data/vendor/snowball/tests/syntax/emptyprogram.syntax +0 -0
  150. data/vendor/snowball/tests/syntax/groupings.sbl +11 -0
  151. data/vendor/snowball/tests/syntax/inline.sbl +144 -0
  152. data/vendor/snowball/tests/syntax/inline.stderr +0 -0
  153. data/vendor/snowball/tests/syntax/inline.syntax +121 -0
  154. data/vendor/snowball/tests/syntax/legacy.sbl +17 -0
  155. data/vendor/snowball/tests/syntax/legacy.stderr +5 -0
  156. data/vendor/snowball/tests/syntax/legacy.syntax +19 -0
  157. data/vendor/snowball/tests/syntax/loops.sbl +14 -0
  158. data/vendor/snowball/tests/syntax/loops.stderr +9 -0
  159. data/vendor/snowball/tests/syntax/loops.syntax +29 -0
  160. data/vendor/snowball/tests/syntax/noops.sbl +76 -0
  161. data/vendor/snowball/tests/syntax/noops.stderr +27 -0
  162. data/vendor/snowball/tests/syntax/noops.syntax +135 -0
  163. data/vendor/snowball/tests/syntax/simplifyae.sbl +21 -0
  164. data/vendor/snowball/tests/syntax/simplifyae.stderr +0 -0
  165. data/vendor/snowball/tests/syntax/simplifyae.syntax +39 -0
  166. data/vendor/snowball/tests/syntax/unused.sbl +18 -0
  167. data/vendor/snowball/tests/syntax/unused.stderr +7 -0
  168. data/vendor/snowball/tests/syntax/unused.syntax +3 -0
  169. data/vendor/snowball/zig/env.zig +599 -0
  170. data/vendor/snowball/zig/generate_algorithms.pl +19 -0
  171. data/vendor/snowball/zig/stemwords.zig +123 -0
  172. metadata +102 -4
  173. data/vendor/snowball/compiler/syswords.h +0 -86
  174. data/vendor/snowball/runtime/header.h +0 -62
data/vendor/snowball/NEWS CHANGED
@@ -1,3 +1,1390 @@
1
+ Snowball 3.1.1 (2026-06-03)
2
+ ===========================
3
+
4
+ Compiler changes
5
+ ----------------
6
+
7
+ * Bug fixes:
8
+
9
+ + Fix a segmentation fault after reporting an error for a string command
10
+ not followed by a string variable name or string literal. Bug introduced
11
+ in 3.1.0. Patch from Jerry James (#287).
12
+
13
+ * Compiler command-line options:
14
+
15
+ + Emit an error for `-o -`/`-output -`. Output to stdout is not supported
16
+ because we need to generate multiple files for some target languages.
17
+ We were interpreting `-` as a base filename to append extensions to, so
18
+ we'd create `-.c` and `-.h` for C, but creating filenames that start with
19
+ `-` seems unhelpful.
20
+
21
+ Generic code generation changes
22
+ -------------------------------
23
+
24
+ * Bug fixes:
25
+
26
+ + Variable localisation was failing to check the expression on the RHS of an
27
+ integer test for uses of a variable, so could incorrectly localise an
28
+ integer variable whose value should have persisted between calls to a
29
+ function. This bug won't realistically manifest in real world Snowball
30
+ code.
31
+
32
+ * Optimisations:
33
+
34
+ + Inline some routines which are only used once. This is done for routines
35
+ consisting of a single non-compound command (or cases such as `not
36
+ <boolean>` and `goto <grouping>` which we internally synthesise a
37
+ non-compound command for). Localisation of variables happens after
38
+ inlining, so variables can now be localised in more cases.
39
+
40
+ + `test next` and `not next` are both now simplified to a comparison between
41
+ `cursor` and `limit` (like `not atlimit` and `atlimit`). We already
42
+ normalise `hop 1` to `next`, so `test hop 1` and `not hop 1` are also
43
+ simplified in this way.
44
+
45
+ + Simplify `not` applied to an integer test by removing the `not` and
46
+ flipping the sense of the test (e.g. `not $(x > y)` becomes `$(x <= y)`)
47
+ which results in simpler generated code. More usefully in real world code,
48
+ this also results in simpler generated code for `not atlimit` (since
49
+ `atlimit` is converted `$(cursor >= limit)` or `$(cursor <= limit)`
50
+ (depending on the current direction).
51
+
52
+ + Canonicalise `delete` to `<-''`. 3.1.0 added a canonicalisation of `<-''`
53
+ to `delete`, but if we instead make `<-''` the canonical representation
54
+ then the generators can easily handle an empty string here specially (and
55
+ if they don't then the results are still correct), we don't have to handle
56
+ the delete token after tokenisation, and the optimisation of among when
57
+ all actions are `<-` plus a literal string can also fire when some actions
58
+ are `delete`. (This case occurs in the stemmers for arabic, czech,
59
+ estonian, hungarian, irish, italian, polish, porter and tamil.)
60
+
61
+ C++
62
+ ---
63
+
64
+ * Code quality:
65
+
66
+ + C++ runtime support is now fully hooked up and can be tested using
67
+ `make check_cxx`.
68
+
69
+ + Don't use `extern "C"` for functions which can throw because MSVC seems to
70
+ treat such functions as if they were marked `noexcept`.
71
+
72
+ Javascript
73
+ ----------
74
+
75
+ * Code quality:
76
+
77
+ + Fix formatting of code generated by `among` of `<-` optimisation where we
78
+ were emitting a literal `+` rather than increasing the margin size. The
79
+ `+` was actually harmless in practice, but the rest of the file would be
80
+ misindented.
81
+
82
+ Python
83
+ ------
84
+
85
+ * Other changes:
86
+
87
+ + Skip classifier for Sesotho which isn't yet in the official list of
88
+ trove classifiers. Patch from Dmitry Shachnev (#289).
89
+
90
+ + Add classifier to indicate support for Python 3.14.
91
+
92
+ Build system
93
+ ------------
94
+
95
+ * The `generate` target added in 3.1.0 now overrides `FPC` and `MCS` to do
96
+ nothing, to avoid compiling Pascal and C# code.
97
+
98
+ * Provide way to run tests under valgrind:
99
+
100
+ make RUN_STEMWORDS='valgrind ./stemwords' check
101
+
102
+ Testsuite
103
+ ---------
104
+
105
+ * compilertest: Require syntax tests have stderr output - we are testing errors
106
+ here, so we should really be checking the errors we get are the ones we
107
+ expect. We expect the compiler to fail, so testing stderr means we will
108
+ catch unrelated failures. This provides a regression test for #287.
109
+
110
+ Documentation
111
+ -------------
112
+
113
+ * NEWS: Fix formatting/wording of a few existing entries.
114
+
115
+ * README: List C++ as target language.
116
+
117
+ Snowball 3.1.0 (2026-05-22)
118
+ ===========================
119
+
120
+ Compiler changes
121
+ ----------------
122
+
123
+ * Bug fixes:
124
+
125
+ + Fix segmentation fault if -syntax is used on a program with no code.
126
+
127
+ + Fix segmentation fault on some assignment syntax errors.
128
+
129
+ + Fix bug introduced in v3.0.0 with conversion of `among` starter. If there
130
+ were any commands after the among in the same command list then the among
131
+ itself would get lost. Not triggered by any current algorithms.
132
+
133
+ + Clear name field when removing dead assignments. This is visible in the
134
+ syntax tree shown when command line option -syntax is used, but probably
135
+ doesn't affect anything otherwise.
136
+
137
+ * Compiler command-line options:
138
+
139
+ + Using `-` for the Snowball source file is now interpreted as stdin.
140
+
141
+ + Improve comments generated by `-comments` to show more details of the
142
+ corresponding Snowball code (e.g. variable names, arithmetic expressions,
143
+ and literal strings).
144
+
145
+ + Add `-coverage` option which enables a code coverage feature. So far this
146
+ tracks which among strings and functions are exercised, and which grouping
147
+ characters are exercised. !
148
+
149
+ + Support `-eprefix` for all target languages. This is easy to do and
150
+ provides a way to deal with externals which collide with keywords in the
151
+ target language. Our build system now uses `-eprefix _` for Python to make
152
+ the `stem` external non-public (it is called by BaseStemmer method
153
+ `stemWord()`) and we no longer hard-code prefixing Python externals with
154
+ `_`.
155
+
156
+ + Describe more options in `--help` output.
157
+
158
+ + Sort target language options in `--help` output.
159
+
160
+ + The `-o` option is now optional. If not specified we now write output(s)
161
+ to the same filename as the first source, but with a different extension
162
+ (e.g. path/to/english.sbl -> path/to/english.c and path/to/english.h).
163
+
164
+ + The `-o` option can now optionally include an extension so you can now
165
+ write `-c++ -o path/to/foo.cxx` instead of `-c++ -o path/to/foo`, which can
166
+ be more convenient (e.g. in `make` rules) and also provides an easy way to
167
+ specify an alternative extension (for example, `.cxx`, `.cc` and `.cpp` are
168
+ all extensions commonly used for C++ source code).
169
+
170
+ + Reject `-vprefix` option for target languages which don't support it (it is
171
+ currently only implemented for C/C++).
172
+
173
+ * Diagnostics:
174
+
175
+ + Clean up and improve error reporting.
176
+
177
+ + Improve line numbers reported for some errors and warnings by using the
178
+ line number of an appropriate token rather than the current line number
179
+ of the tokeniser (which is often the line after the command being warned
180
+ about).
181
+
182
+ + Improve recovery after various errors, trying to resynchronise based on
183
+ what's more likely, and eliminating some additional irrelevant errors
184
+ (including reporting the exact same error twice in some situations).
185
+
186
+ + Emit warnings for uses of legacy Snowball language features.
187
+
188
+ + The Snowball manual describes `integers (x)` as a declaration of `x` so we
189
+ now warn:
190
+
191
+ integer 'x' declared but not used
192
+
193
+ rather than:
194
+
195
+ integer 'x' defined but not used
196
+
197
+ + 3.0.0 added a warning if the body of a `repeat` or `atleast` loop always
198
+ signals `t` (meaning it will loop forever which is very undesirable for a
199
+ stemming algorithm) or always signals `f` (meaning it will never loop,
200
+ which seems unlikely to be what was intended). This warning was added to
201
+ the C generator, but has been moved to generic code so it is now issued
202
+ regardless of the current target language.
203
+
204
+ + Improve the wording of the warning if the body of a `repeat` or `atleast`
205
+ loop always signals 't' to explicitly say this means the loop is infinite.
206
+
207
+ + Improve warning message for unreachable code after `not`.
208
+
209
+ + `$x = x + 1` cleared the initialised status of x (rather than just not
210
+ setting it) which could lead to bogus warnings that `x` is never
211
+ initialised.
212
+
213
+ + The compiler no longer exits immediately after reporting a division by zero
214
+ error in the Snowball code.
215
+
216
+ + We now report a division by zero error for `$x /= 0` (this was meant to be
217
+ already implemented but wasn't working due to a code typo).
218
+
219
+ + More consistent wording of "is a no-op" warnings.
220
+
221
+ + Warn that `insert ''` and `attach ''` are no-ops (and don't generate code
222
+ for them).
223
+
224
+ + Warn if a string used to define a grouping repeats characters. There's no
225
+ reason to do this, so it seems likely to be a typo.
226
+
227
+ + Avoid sometimes reporting "-1 blocks unfreed".
228
+
229
+ * Optimisations:
230
+
231
+ + Speed up processing larger Snowball programs by growing large string
232
+ buffers exponentially to avoid a huge number of reallocations. For
233
+ example, this reduced the time to compile serbian.sbl to C by about 80%!
234
+
235
+ + Optimise reading of input file when it is seekable (which it is in typical
236
+ usage). Non-seekable input files are still supported.
237
+
238
+ + Optimise writing integers when generating code. 72% of integers we write
239
+ are 0 to 9 and these are now written as a character. Other values are now
240
+ handled without a temporary buffer, avoiding a copy. This reduced the time
241
+ to compile serbian.sbl to C by about 8%, for example.
242
+
243
+ + Optimise comparing among actions to find and merge equivalent actions.
244
+ The comparison function used for this was carefully returning a full order,
245
+ but actually we only need to know if the actions are equivalent or not
246
+ which can be tested more efficiently. For example, this reduced the time
247
+ to compile serbian.sbl to C by about 2%.
248
+
249
+ + We now precompute the possible signals from each command which means this
250
+ is now done exactly once per command, whereas previously we could end up
251
+ doing it many times for some commands in some cases. The only functional
252
+ change should we no longer make a pessimistic assumption if the function
253
+ call depth reaches 100. This is cleaner but is unlikely to make a
254
+ difference for any real-world Snowball programs.
255
+
256
+ + Handle possible_signals for string-$ which just passes on signals from its
257
+ subcommand. This doesn't affect code generation for any algorithms we
258
+ currently ship.
259
+
260
+ + We now only generate function bodies to a temporary buffer for target
261
+ languages where we need to. This makes the code a bit clearer and reduces
262
+ the amount of copying of data so will make the Snowball compiler a little
263
+ faster. This change produces identical output for all current algorithms.
264
+
265
+ + Tokenisation now decodes symbol tokens using switch statements. We don't
266
+ know the length of these tokens in advance, so the old approach of binary
267
+ chop on a sorted list required searching the list multiple times with
268
+ different possible lengths. Alphabetical tokens are still decoded by
269
+ binary chop.
270
+
271
+ * Code quality:
272
+
273
+ + Remove unused routines and groupings from the program during the analysis
274
+ phase, which avoids each generator having to have duplicate code to skip
275
+ them.
276
+
277
+ + Fix small memory leak if all uses of a name are eliminated.
278
+
279
+ + Always use `snprintf()` instead of `sprintf()`. If the buffer passed was
280
+ too small we now emit an error rather than quietly using truncated output.
281
+
282
+ + Fix GCC -Wcast-qual warnings in compiler and enable this warning by
283
+ default.
284
+
285
+ + Switch to using the standard C `bool` type in the code of the compiler.
286
+ (The generated code still aims to require only C90.)
287
+
288
+ * Other changes:
289
+
290
+ + Provide a simpler way to build a cut-down Snowball compiler. The
291
+ motivation here was to have a way to more quickly build a smaller Snowball
292
+ compiler which only targets C. Rather than have a DISABLE_xxx macro for
293
+ each language, just check if TARGET_C_ONLY is defined, and only turn off
294
+ the code to actually call the other generators which greatly reduces the
295
+ amount of conditionalisation required.
296
+
297
+ Generic code generation changes
298
+ -------------------------------
299
+
300
+ * Bug fixes:
301
+
302
+ + Fix code generated for `setlimit tomark` for all target languages to
303
+ restore the limit correctly afterwards. The bug was not triggered by any
304
+ of the existing stemmers.
305
+
306
+ + We no longer optimise repeat/atleast applied to goto/gopast on a (non)
307
+ grouping. This optimisation was flawed - it requires that the code in the
308
+ loop preserves the cursor's value on failure, but the target language
309
+ helper functions used here don't currently do that (they probably easily
310
+ could so there's scope to reinstate this optimisation).
311
+
312
+ Looking at the stemmers we ship, this affects the code generated for one
313
+ loop in indonesian.sbl, but it happens the cursor value is overwritten
314
+ immediately after this loop anyway. The bug could affect non-shipped
315
+ Snowball code though so isn't purely latent. This bug was introduced in
316
+ Snowball 3.0.0.
317
+
318
+ + When generating target language literal strings we now always escape
319
+ characters which can be problematic when viewing the generated source code.
320
+ We always escape control characters U+007F to U+009F, non-breaking space
321
+ U+00A0 (visually identical to a space), and U+0590 and above (as a crude
322
+ way to avoid literal LTR characters in sources which can result in
323
+ confusing rendering).
324
+
325
+ + Fix line numbers given to various tokens (the line numbers previously given
326
+ were at least of lines in the same command or the line after it). These
327
+ lone numbers can be seem in the target language comments generated when
328
+ `-comments` is used.
329
+
330
+ + Fix warning and simplification of code when `not` is applied to a command
331
+ which always signals `t`. Bug introduced in v3.0.0. Fixes #271.
332
+
333
+ + Warn and simplify `not` applied to a command which always signals `f`.
334
+
335
+ * Optimisations:
336
+
337
+ + Add machinery to generate a Snowball variable as a local variable in the
338
+ target language instead of it being "global" (typically a private class
339
+ member in the target language). This reduces the amount of state in
340
+ stemmer objects, and typically reduces the overhead of accessing these
341
+ variables a little.
342
+
343
+ We now do this for integers and booleans in all target languages, and
344
+ for strings in target languages where benchmarking seems to show it
345
+ is faster (Dart, Go, JS, Pascal, PHP, Python).
346
+
347
+ It's done for a Snowball variable which is only used in one routine, that
348
+ routine doesn't (directly or indirectly) call itself, and the variable is
349
+ set by any code path which leads to a use of the variable. The mechanism
350
+ which traces the code paths errs on being too conservative in some cases,
351
+ but it's good enough for all instances in the code we ship, and is likely
352
+ to handle the vast majority of real-world cases. We issue an "info"
353
+ diagnostic to report when a variable which is only used in one routine
354
+ can't be localised - please report if you see this in real world code and
355
+ we can try to enhance the code path tracing.
356
+
357
+ + Tail-calling and similar optimisations can now work for non-trivial
358
+ routines (previously they only worked for routines consisting of a
359
+ single command and not enclosed in parentheses).
360
+
361
+ + A grouping test at the end of a routine now generates simpler code.
362
+
363
+ + A string test at the end of a routine now generates simpler code.
364
+
365
+ + Optimise testing a boolean (optionally preceded by `not`) when used at the
366
+ end of a routine.
367
+
368
+ + Optimise an `among` with no commands at the end of a routine.
369
+
370
+ + Generate simpler code for `not` applied to testing a boolean variable.
371
+
372
+ + A `not` only needs to restore the cursor when its subcommand fails, so we
373
+ now consider whether its subcommand can modify the cursor on failure
374
+ (rather than whether it can modify the cursor at all). Related to #226.
375
+
376
+ + An `or` only needs to restore the cursor when a subcommand fails, so we
377
+ now consider whether its subcommand can modify the cursor on failure
378
+ (rather than whether it can modify the cursor at all). We also now
379
+ consider each subcommand individually, and only emit the cursor restore for
380
+ those subcommands which need it. (#226)
381
+
382
+ + Both `and` and `or` only need to restore the cursor between sub-commands
383
+ so no longer consider if the final subcommand might change the cursor.
384
+ This makes some small improvements to the generated code for a few
385
+ of the currently shipped algorithms. (#226)
386
+
387
+ + Handle more commands when checking if the cursor needs restoring - this
388
+ improves the generated code for tamil.sbl a bit.
389
+
390
+ + Single case amongs are now refactored to eliminate the `among` and so no
391
+ longer call the among machinery. Sometimes a single case among is the
392
+ natural way to express a single rule in Snowball code as it can show
393
+ commonality with rulesets with multiple rules, but it's inefficient to
394
+ actually generate as an among. Of the stemmers we currently ship, this
395
+ improves code generation for arabic, estonian, greek and lithuanian.
396
+
397
+ + Avoid unnecessary cursor update in among helpers. We only need to update
398
+ the cursor on success, but were unconditionally doing so after calling an
399
+ among function.
400
+
401
+ + Handle more commands in repeat_score(). None of these help code generation
402
+ for any currently shipped algorithms, but they are valid to optimise here.
403
+
404
+ + Canonicalise `<-''` to `delete`.
405
+
406
+ + Simplify some cases of compound assignment operators when the argument is
407
+ (or can be simplified to) a constant integer, like we already do for
408
+ arithmetic expressions. For example, `$x += len '{a"}' - len 'a'` is a
409
+ no-op when using a fixed-width encoding.
410
+
411
+ + Canonicalise `fail C` to `false` in some cases where `C` has no
412
+ side-effects.
413
+
414
+ + Removing unreachable code could leave single-entry `and`/`or` nodes
415
+ which could result in generating target language code with unused
416
+ variables. These nodes are now replaced with their subnode.
417
+
418
+ + Eliminate `true` below `and`/`false` below `or`. These are unlikely to
419
+ appear verbatim in real programs, but can be created by optimisations, and
420
+ also can appear in runtime tests, leading to the generated target language
421
+ code having unused variables and/or unreachable code.
422
+
423
+ + Canonicalise setmark, atmark and atlimit by converting `setmark x` to
424
+ `$x = cursor`, `atmark x` to `$(cursor == x)` and `atlimit` to either
425
+ `$(cursor >= limit)` or `$(cursor <= limit)` (depending on whether we're
426
+ in backwardmode or not). This means the target language generators have
427
+ three fewer commands to handle, and also gives us tail-calling of `atlimit`
428
+ and `atmark x` (there's a tail-callable use of `atlimit` in the Turkish
429
+ stemmer).
430
+
431
+ + Remerge among actions after optimisation. It seems hard to fully move the
432
+ code to merge them later, but we can check for actions which have become
433
+ equivalent to `true` or to other actions after optimisation but before we
434
+ generate code.
435
+
436
+ * Code quality:
437
+
438
+ + Find Snowball routines which are not reachable by calling an external. We
439
+ no longer generate code for such routines, nor for variables and groupings
440
+ which are only used in them, which helps to avoid "unused" warnings in the
441
+ generated target language code.
442
+
443
+ + If the sub-command of `repeat`/`atleast` always signals `t` or always
444
+ signals `f` we now prune the rest of the current command list, and simplify
445
+ the command for always `f` (c_repeat -> c_do; c_atleast -> c_bra). These
446
+ changes help to avoid generating redundant target language code which can
447
+ trigger errors or warnings.
448
+
449
+ * Other changes:
450
+
451
+ + `delete` and `<-` now update the slice end (see the "Snowball Language
452
+ Changes" section).
453
+
454
+ Ada
455
+ ---
456
+
457
+ * Bug fixes:
458
+
459
+ + Ada variable names are case-insensitive, so if two Snowball names of the
460
+ same type differed only by case we would generate Ada code with a name
461
+ collision. We now avoid such collisions by adding a counter after the type
462
+ code for the second and subsequent names that differ only by case.
463
+
464
+ + Ada stemmer names are now prefixed with `S_` so `or.sbl` now generates
465
+ stemmer `S_or`, avoiding a name clash with an Ada keyword.
466
+
467
+ + Fix Ada code generated `for setlimit tomark p`. This affected the
468
+ generated code for the Lithuanian stemmer, but it appears by luck in this
469
+ case the bug didn't actually affect the stemmer's output for any input.
470
+
471
+ + Fix `setlimit` ... `repeat` bug. The generated Ada code was running the
472
+ code to recover from a failure inside a `repeat` loop twice due to a
473
+ missing line of code compared to other generators. In `backwardmode`, the
474
+ failure code happens to be idempotent so running it twice doesn't cause a
475
+ problem, but in forwards mode this results in the cursor getting double
476
+ adjusted if the length of the stem has changed due to insertions, deletions
477
+ or substitutions. None of the existing algorithms use `setlimit` in
478
+ forwards mode, so they're unaffected by this bug. Fixes #275.
479
+
480
+ + Fix overcopying in string replacement code. The code to move the tail
481
+ up/down was copying one byte too many. We're working in a 1024 byte
482
+ fixed-length string buffer, and the maximum allowed input word is one byte
483
+ shorter, so it seems this was harmless in practice.
484
+
485
+ + Allow characters <32 and 127 in string literals.
486
+
487
+ + `=S` can no longer result in the slice ends becoming negative and
488
+ triggering a CONSTRAINT_ERROR (the slice is now specified to be unset
489
+ after `=S` - see the "Snowball Language Changes" section).
490
+
491
+ + Fix Ada code generated for string-$ which was actually partly Pascal
492
+ code (the Ada generator was originally based on the Pascal one) and
493
+ didn't even compile. To fix this, Snowball string variables in Ada the
494
+ same way as the current string. This means they now take up more space (a
495
+ fixed 1KB), but a typical Snowball program has either no string variables
496
+ or just one so the overhead seems acceptable.
497
+
498
+ + Fix matching of an empty string variable. This valid Snowball code would
499
+ trigger "failed precondition" in Ada:
500
+
501
+ externals (stem)
502
+ strings (s)
503
+ define stem as ([] ->s s)
504
+
505
+ + Fix assumption that there's a single external called "stem".
506
+
507
+ + Fix incorrect assumption that an among containing the empty string
508
+ always matched, even if the empty string had a gating function.
509
+ This construct is not used by any existing stemmers.
510
+
511
+ * Optimisations:
512
+
513
+ + Avoid calling among helper when the among contains only strings which are
514
+ one byte long, no among functions are used, and there are no actions.
515
+
516
+ * Code quality:
517
+
518
+ + Fix indentation of generated grouping tables.
519
+
520
+ + Rename Context to Z in runtime code. This now matches variable naming in
521
+ the generated Ada code (and also the C runtime and generated C code).
522
+
523
+ + Eliminate redundant limit check (the Skip_Utf8 helper also checks the
524
+ limit). Looking at the history this check is a left-over from when the
525
+ generated code directly incremented the cursor.
526
+
527
+ + Emit Ada literal strings without redundant empty strings between adjacent
528
+ escaped bytes.
529
+
530
+ + Generate dummy loop around `or`, which allows us to handle a sub-command
531
+ succeeding with Ada `exit` rather than `goto`, which seems clearer.
532
+
533
+ + Avoid creating unused labels. This is just a cosmetic improvements - there
534
+ are no longer mysterious gaps in the numbering of labels in the generated
535
+ code.
536
+
537
+ + Avoid generating unreachable `exit`.
538
+
539
+ * Other changes:
540
+
541
+ + Implement support for `?` (debug command). The code we generate for this
542
+ case is gnat-specific, but previously the code generated didn't compile so
543
+ working with one implementation seems a step forwards. The `?` command can
544
+ now be used to debug Ada, and someone with actual Ada knowledge can now
545
+ more easily step in and provide a portable replacement.
546
+
547
+ C/C++
548
+ -----
549
+
550
+ * Bug fixes:
551
+
552
+ + Maintain invariant that the C variable corresponding to a Snowball string
553
+ variable is non-NULL. Previously we would release and NULL out the entry
554
+ in some error cases, but elsewhere the code was assuming the value was
555
+ non-NULL.
556
+
557
+ + Fix invalid code generated for `setlimit`. This doesn't happen for
558
+ `setlimit tomark` (which is the only way `setlimit` is used in the stemmers
559
+ we currently ship. Bug introduced in v3.0.0.
560
+
561
+ + Fix codegen for `hop` with constant argument. We were relying on the
562
+ cursor being restored on failure by the code which handled that failure,
563
+ but if that code is a repeat or atleast command that it has an optimisation
564
+ which assumes `hop` won't do this. This means we generated incorrect C
565
+ code for some cases where `hop` was used inside `repeat` or `atleast`.
566
+ This doesn't functionally affect any of the stemmers we currently ship.
567
+ Bug introduced in v2.1.0.
568
+
569
+ + Fix bug in code generated when `-vprefix` is specified, introduced in
570
+ Snowball 2.1.0.
571
+
572
+ + Fix incorrect assumption that an among containing the empty string
573
+ always matched, even if the empty string had a gating function.
574
+ This construct is not used by any existing stemmers.
575
+
576
+ * Optimisations:
577
+
578
+ + Rework how non-localised variables are stored, which eliminates an
579
+ indirection on every access to such a variable, and also avoids some extra
580
+ allocations (one if a stemmer has any non-localised integer or boolean
581
+ variables, and another if the stemmer has any string variables). So it
582
+ uses a bit less memory, it makes creating and destroying a stemmer faster,
583
+ and it also makes stemming a bit faster (though only by ~0.1% for the
584
+ English stemmer on our sample vocabulary). The `-vprefix` option now
585
+ generates getter functions rather than using macro magic, which means the
586
+ syntax for accessing Snowball variables from C has changed.
587
+
588
+ + We now maintain the invariant that SN_env's p member is non-NULL, which
589
+ simplifies the runtime code.
590
+
591
+ + We now have a specialised implementation of the slice_del() runtime helper.
592
+ Deleting the slice is a fairly common operation, and can be done more
593
+ simply than via a generic replace_s() with an empty replacement string.
594
+ This speeds up the English stemmer by about 1% on our test vocabulary.
595
+
596
+ + Avoid calling among helper when the among contains only strings which are
597
+ one byte long, no among functions are used, and there are no actions.
598
+
599
+ + Only fetch SIZE() in replace_s() if we need it.
600
+
601
+ + Don't return adjustment from replace_s() runtime helper since calculating
602
+ the adjustment in the one caller where we actually want it is just one
603
+ integer addition and one integer subtraction, and that turns out to be
604
+ slightly more efficient as well as simpler.
605
+
606
+ + Move check for negative hop from runtime to generated code. This means we
607
+ can omit it for hop with a constant argument, which is all uses of hop in
608
+ the stemmers we currently ship.
609
+
610
+ * Code quality:
611
+
612
+ + The generated header is now included from the generated C/C++ source file
613
+ (which seems cleaner than the previous approach of generating the same
614
+ prototypes in the header and source file).
615
+
616
+ + The implementation of among functions has changed. Previously we stored
617
+ a function pointer in struct among, but that requires relocation when the
618
+ code is in a dynamic library, which adds load-time overhead and means
619
+ the among structures can't be put in a read-only section.
620
+
621
+ We now store an integer index instead, and pass in a pointer to
622
+ a dispatcher function when calling the find_among()/find_among_b() helper
623
+ which gets called when this index is non-zero. The value of the index is
624
+ stored in z->af so the dispatcher function can use it.
625
+
626
+ If only one unique function is used in an among, we can just pass this to
627
+ find_among() as the dispatcher which reduces the overhead for this common
628
+ case.
629
+
630
+ Profiling with cachegrind suggests this change adds a small overhead
631
+ to algorithms which use among functions - currently finnish and hindi
632
+ (and also lovins, but that's really only of academic interest and is not
633
+ enabled by default).
634
+
635
+ + Avoid long string in C source. C90 only guarantees support for literal
636
+ strings up to 509 characters. Fixes GCC -Woverlength-strings warning.
637
+
638
+ + Avoid C23 feature in C runtime code, introduced in Snowball 3.0.0.
639
+ Initialising with empty braces was only standardised in C23 (though
640
+ seems to be widely supported as an extension).
641
+
642
+ + Fix code generated for `setlimit` to be C90. Bug introduced in v3.0.0, but
643
+ isn't triggered by any of the stemmers we currently ship.
644
+
645
+ + Fix -Wshadow warning for nested string-$ use. We were generating code
646
+ using a C variable with the fixed name `failure` - now an integer suffix is
647
+ appended, and we only emit the variable in cases where the subcommand
648
+ signal isn't known at compile time.
649
+
650
+ + Generate `do {`...`} while (0)` around `or` code, which allows us to handle
651
+ a sub-command succeeding with `break;` rather than having to use `goto`,
652
+ which reduces the number of labels used and makes the generated code a bit
653
+ easier to follow.
654
+
655
+ + C comments are now generated for `(` and `do` when `-comments` is used.
656
+
657
+ + We now generate `+=` or `-=` for `hop <constant>` (instead of something
658
+ like `z->c = z->c + 2`). The C compiler should treat both the same, but it
659
+ arguably makes the generated code a little clearer.
660
+
661
+ * Other changes:
662
+
663
+ + C++: The `-c++` option used to generate exactly the same code as for C,
664
+ except with extension `.cc` instead of `.c` but now:
665
+
666
+ - C++ classes are generated.
667
+ - C++ `bool` is used for Snowball booleans.
668
+ - Loop variables are declared inside `for (`...`)`.
669
+ - Allocation failures and internal errors (e.g. slice_check() failing)
670
+ throw a C++ exception - this is a bit simpler and more efficient that the
671
+ C code approach of returning -1 which then has to be checked for and
672
+ propagated through the generated code.
673
+
674
+ + Snowball's debug command (`?`) now works out of the box (previously you
675
+ have to adjust a `#if 0` preprocessor conditional in the runtime code).
676
+
677
+ + Rename `runtime/header.h` (which really seems too generic, and is also easy
678
+ to confuse with `compiler/header.h`) to `runtime/snowball_runtime.h`. We
679
+ expect most users will be using the C stemmers through libstemmer and so
680
+ won't be affected by this.
681
+
682
+ C#
683
+ --
684
+
685
+ * Bug fixes:
686
+
687
+ + Fix code generated for `<-s`. This is not used by any of the stemmers we
688
+ currently ship. Test case based on one from ajroetker in #270.
689
+
690
+ + Fix code generated for string-$. This feature is not used by any of the
691
+ stemmers we currently ship.
692
+
693
+ + Fix assumption that there's a single external called "stem".
694
+
695
+ * Optimisations:
696
+
697
+ + Use Debug.Assert() in slice_check() runtime helper. Previously the runtime
698
+ code wrote a diagnostic message and continued if one of these checks
699
+ failed, but failures should only happen with a Snowball program containing
700
+ logic errors, or for bugs in the Snowball compiler or its runtime (or
701
+ possibly in the C# compiler, runtime, OS, hardware, etc). Therefore an
702
+ assertion seems an appropriate choice, and means the check is not enabled
703
+ for a production build, which seems more helpful overall. See #242.
704
+
705
+ * Code quality:
706
+
707
+ + Eliminate duplicates from groupings. We currently implement these for C#
708
+ with a linear string search, and a side-effect of this change is that the
709
+ grouping string is now sorted, which will affect the time taken to look
710
+ up different characters in an arbitrary way (none of the Snowball sources
711
+ seem to try to list characters in frequency order). Really C# should be
712
+ fixed to use an O(1) lookup like other target languages.
713
+
714
+ + The implementation of among functions has changed. We now store an integer
715
+ index in the Among class, and pass a dispatcher function to the among
716
+ helper method.
717
+
718
+ If only one unique function is used in an among, we can just pass this to
719
+ the helper method as the dispatcher which reduces the overhead for this
720
+ common case.
721
+
722
+ Crude profiling with `time make check_csharp` suggests this doesn't harm
723
+ performance (perhaps a little faster, but maybe just within the noise).
724
+
725
+ The main benefit is all Among arrays can now be static, which previously
726
+ we wasn't possible for those which used among functions (#146).
727
+
728
+ + Remove unused return value from Stemmer.Replace() runtime helper.
729
+
730
+ + Fix inaccurate doc comments on runtime functions.
731
+
732
+ * Other changes:
733
+
734
+ + csharp_stemwords: Speed up output to stdout.
735
+
736
+ + csharp_stemwords: Don't write the chosen stemmer to stdout. This is not
737
+ really useful information, and breaks sending the stemmed words to stdout
738
+ because they're preceded by extra output.
739
+
740
+ + csharp_stemwords: Try to open input before output so we don't leave an
741
+ empty output file behind if we can't open the input file.
742
+
743
+ Go
744
+ --
745
+
746
+ * Bug fixes:
747
+
748
+ + Fix code generated for non-constant hop. A non-constant integer expression
749
+ has type `int` in the generated Go code, but the hop helpers expected
750
+ `int32`. For a constant hop this worked because Go integer literals are
751
+ untyped, so will convert to `int32`. To fix this, the helpers now take
752
+ `int` instead of `int32`.
753
+
754
+ + Fix code generated if `minint` or `maxint` is used. In this case we were
755
+ generating `use std::usize;` near the start of the Go code, but that's
756
+ actually Rust code and a hangover from the Go backend being originally
757
+ based on the Rust one.
758
+
759
+ + The Go code generated for `->` was incorrectly signalling `f` if the
760
+ slice was empty. Luckily this case is not exercised by any current
761
+ algorithms. See #242.
762
+
763
+ + Fix code generated for string-$ (which isn't used by any of the algorithms
764
+ we currently ship).
765
+
766
+ + A snowball `external` could not previously be called from within the
767
+ Snowball program. This is allowed by the Snowball language, but none of
768
+ the shipped stemmers do this, and it's unlikely any stemmer would. Perhaps
769
+ it's useful if you use Snowball for other string-processing tasks.
770
+
771
+ + Fix handling of `minint` and `maxint` - we were generating some code copied
772
+ verbatim from the Rust generator for this case which was not valid Go.
773
+ (These are not used by any of the algorithms we currently ship.)
774
+
775
+ * Optimisations:
776
+
777
+ + Reuse `env` in stemwords which is measurably faster than creating a new one
778
+ for every word.
779
+
780
+ * Code quality:
781
+
782
+ + Eliminate unnecessary semicolons from generated code.
783
+
784
+ + Fix formatting of generated code. The code gets run through gofmt which
785
+ was fixing up these issues, but better to generate the code cleanly to
786
+ start with. The only things which gofmt now changes are that it indents
787
+ variable names to align in adjacent variable declarations, and a couple of
788
+ things which are apparently for compatibility with older versions of Go.
789
+
790
+ + Runtime helpers SliceDel() and SliceFrom() always returned true, but the
791
+ generated code included failure checks in case false was returned.
792
+ These helpers no longer return anything, and the checks are gone.
793
+
794
+ * Documentation:
795
+
796
+ + Recommend that users reuse an `env` since this is measurably faster than
797
+ creating a new one for every word.
798
+
799
+ * Other changes:
800
+
801
+ + Remove `-gopackage` option from compiler. Use `-package`/`-P` instead
802
+ (`-gopackage` has just been an alias for these since Snowball 2.0.0).
803
+
804
+ Java
805
+ ----
806
+
807
+ * Bug fixes:
808
+
809
+ + Generate correct Java code for ASCII control chars in string literals.
810
+
811
+ + Fix code generated for string-$. As part of this fix, we now use char[]
812
+ for string variables as well as the current string, which makes it much
813
+ simpler to switch to working on a string variable and back. Fixes #252.
814
+
815
+ + Fix assumption that there's a single external called "stem".
816
+
817
+ * Other changes:
818
+
819
+ + The generated Java classes no longer implement Serializable. This support
820
+ was added in 2016, but in 2026 this approach to serialization in Java is
821
+ apparently no longer used due to security problems. Fixes #255.
822
+
823
+ Javascript
824
+ ----------
825
+
826
+ * Bug fixes:
827
+
828
+ + Fix `->` to work when the slice is empty - previously it incorrectly
829
+ signalled `f` for this case. Luckily this case is not exercised by any
830
+ current algorithms (#242)
831
+
832
+ + Generate public functions for all externals. Patch from simlrh (#258).
833
+
834
+ + Fix code generated for string-$
835
+
836
+ * Optimisations:
837
+
838
+ + Use startsWith()/endsWith() in eq_s()/eq_s_b(). This is quite a bit faster
839
+ as it avoids slice() creating a temporary string (e.g. measured a reduction
840
+ of ~17% wallclock time for tamil on the test vocabulary, taking the fastest
841
+ of 5 runs before and after).
842
+
843
+ + Optimise among when all actions are `<-` with a literal string. We now
844
+ generate a single call to slice_from() with the argument obtained by
845
+ indexing into an array of literal strings. This is perhaps faster, albeit
846
+ not by much, but it definitely results in smaller code, which is helpful
847
+ for in browser use. See #227.
848
+
849
+ + The substring_i member in the Among class is now an offset from the current
850
+ index, and now zero in the common case where there's not another string
851
+ which is a sub-prefix/sub-suffix. We've also swapped the order of elements
852
+ so we can omit this in the common case when it is zero and there's no among
853
+ function). This reduces the size of the generated Javascript code (even
854
+ after minification). Fixes #236.
855
+
856
+ + Change slice_check() to assert its conditions. In C we must not perform
857
+ string slicing if slice_check() fails because that could result in writing
858
+ outside of the allocated buffer, but it's not problematic in this way for
859
+ Javascript, and the situations which slice_check() checks for should only
860
+ happen with a Snowball program containing logic errors, or for bugs in the
861
+ Snowball compiler or its runtime (or possibly in the Javascript
862
+ interpreter, OS, hardware, etc). Therefore assert() seems an appropriate
863
+ choice.
864
+
865
+ * Code quality:
866
+
867
+ + Convert to using Javascript modules and classes. The way among functions
868
+ are called has been reworked to allow this, copying the approach now used
869
+ for C and C# (#234, #240). Patches from Adam Turner and Titus Ng.
870
+
871
+ + Adjust generated code to work with deno, and suppress a few deno warnings
872
+ which are hard to avoid in generated code.
873
+
874
+ + Avoid generating blocks around failure handling. The failure handle code
875
+ is always a single statement (and if we ever needed more than a statement
876
+ for some situation then we could arrange to add a block for just those
877
+ situations). This significantly reduces the size of the generated JS code.
878
+
879
+ + Always inline code for `=>`. The code is not much longer than the
880
+ call to a helper function in BaseStemmer. Also in 3.0.0 we deprecated `=>`
881
+ and nothing we ship contains this command, so removing it from BaseStemmer
882
+ reduces the total code size a little.
883
+
884
+ + Rename BaseStemmer's internal `cursor` property to `c`. Unfortunately,
885
+ `cursor` is a DOM property, so Javascript minifiers are cautious about
886
+ renaming it to avoid breaking code. The name `c` matches the naming we use
887
+ for C, Ada and Pascal.
888
+
889
+ + Generate smaller code for hop by constant. All current uses of hop in the
890
+ stemmers we ship have a constant argument, so avoid using a temporary
891
+ variable in these cases.
892
+
893
+ + Optimise `+=1` to `++`, `-=1` to `--`. These are a byte shorter, and it
894
+ seems Javascript minifiers don't do this for us because it's not a safe
895
+ transformation unless the minifier can deduce that the variable can't hold
896
+ a string.
897
+
898
+ + Improve temporary var naming and use. These variables don't need unique
899
+ generated names now we're declaring them as `const` which has more sensible
900
+ scoping rules than `var`.
901
+
902
+ + Generate smaller code for `insert` and string-`=`. In some cases we know
903
+ we have the value of member variable `this.cursor` in local `const c` so
904
+ use the latter instead.
905
+
906
+ + Use triple equality for Javascript. Patch from Adam Turner.
907
+
908
+ + Fix position of grouping type comment which is now placed consistently with
909
+ other type comments.
910
+
911
+ + Use `a` instead of `among_var` in generated code. This reduces the size of
912
+ the generated code, which is helpful if a minification step isn't being
913
+ used.
914
+
915
+ + Consistently cuddle braces in runtime code. The style wasn't entirely
916
+ consistent before, and cuddling braces matches the generated Javascript
917
+ code and the Snowball C code.
918
+
919
+ + Generate block around case to bound the scope of `const` and `let` within
920
+ the case.
921
+
922
+ + Use `let` in README example.
923
+
924
+ + Use `let` consistently in stemwords.js.
925
+
926
+ + Initialise integer Snowball variables - we annotate them as being type
927
+ "number" so we shouldn't let them have value undefined. Patch from Adam
928
+ Turner.
929
+
930
+ + Improve/fix typescript annotations in runtime and generated code.
931
+
932
+ + Annotate runtime with @ts-expect-error. It doesn't seem to be possible to
933
+ express the types fully in some places, but the invariants we require are
934
+ ensured by the Snowball compiler. Annotating the expected errors allows
935
+ unexpected type checking errors to be be more easily seen, and they are
936
+ now fatal is CI.
937
+
938
+ + Use `===` and `!==` in stemwords.js. Patch from Adam Turner.
939
+
940
+ * Other changes:
941
+
942
+ + Make stemmer subclasses anonymous and export them by default. This makes
943
+ creating a stemmer object easier as you only need to build the filename of
944
+ the stemmer subclass, and not also its class name.
945
+
946
+ + Adjust interpretation of `-parentclassname` option. We supply the JS
947
+ snowball runtime so being able to specify a different base class name
948
+ doesn't seem very useful, so instead interpret this as the name to import
949
+ the base class as in generated stemmers. It now defaults to just `B` which
950
+ reduces the size of the generated stemmer code a little (even after running
951
+ it through most Javascript minification tools).
952
+
953
+ + Improve stemwords.js option parsing. Make `-i` and `-o` optional to match
954
+ other target language versions of stemwords. Eliminate the check that
955
+ there are at least 3 command line arguments as we don't require any now.
956
+ If we encounter an argument we don't understand, we now report it and show
957
+ the usage message (previously we silently ignored it). We now exit with
958
+ status 1 if there's a problem parsing the command line.
959
+
960
+ + stemwords.js: Emit help message in one console.log. Patch from Titus Ng
961
+ (#221).
962
+
963
+ Pascal
964
+ ------
965
+
966
+ * Bug fixes:
967
+
968
+ + We were generating invalid Pascal code when tail-calling or calling a
969
+ routine which always fails. Neither case is currently exercised by any
970
+ stemmers we ship and generate Pascal code for (the Pascal generator
971
+ currently only supports iso-8859-1).
972
+
973
+ + Fix code generated for string-$ (which isn't used by any of the algorithms
974
+ we currently ship).
975
+
976
+ + Fix assumption that there's a single external called "stem".
977
+
978
+ * Code quality:
979
+
980
+ + Merge EqS and EqV runtime functions. We can get the length of a Pascal
981
+ AnsiString `s` cheaply with `Length(s)` so there isn't a need to pass in
982
+ the length in the string literal case.
983
+
984
+ + Eliminate `While` in code generated for `repeat`/`atleast`. Pascal lacks
985
+ `Continue` (at least as a standard feature) and this loop only exists so we
986
+ can jump back to its start with `continue` in other languages - we have a
987
+ `Break;` at its end so it doesn't loop in the normal way. In Pascal we
988
+ generate a label before the loop and use `goto` to continue iterating, so
989
+ we can get rid of the Pascal loop entirely.
990
+
991
+ + Use `Break` instead of `Goto` in code generated for `go`/`gopast`.
992
+
993
+ + Generate dummy loop around `or` so we can handle a sub-command succeeding
994
+ with Pascal `Break` rather than `Goto`, which seems clearer.
995
+
996
+ + Avoid generating `Repeat` ... `Until True` dummy loops which are not
997
+ actually needed.
998
+
999
+ + Fix problem introduced in v3.0.0 with formatting of code generated for
1000
+ `go`/`gopast` applied to a grouping.
1001
+
1002
+ + Switch to a simpler name mangling system. Pascal variable names are
1003
+ case-insensitive but Snowball names are case-sensitive. We used to address
1004
+ this by encoding the case of letters into a prefix on the name but that can
1005
+ generate long and ugly names in some cases (e.g. integer Foo_Bar ->
1006
+ IUllU_Foo_Bar). We now avoid collisions by adding a counter after the
1007
+ type code for the second and subsequent names that differ only by case
1008
+ (so Foo_Bar is only mangled if there's another integer which differs
1009
+ only by case which is declared before it, and even then just becomes
1010
+ something like I2_Foo_Bar).
1011
+
1012
+ + Emit Pascal literal strings without redundant empty strings between
1013
+ adjacent escaped bytes.
1014
+
1015
+ + The -comments option now includes the values of string literals, so has
1016
+ been changed to generate "rest of line" comments (starting `//`) rather
1017
+ than block comments (delimited by `{` ... `}`) so that string literals
1018
+ containing `}` don't need escaping. We were already using `//` comments in
1019
+ the Pascal runtime so this shouldn't harm portability.
1020
+
1021
+ Python
1022
+ ------
1023
+
1024
+ * Bug fixes:
1025
+
1026
+ + Fix `algorithms()` when forwarding to PyStemmer. It looks like this has
1027
+ never worked as the code has been like this since it was merged, and we
1028
+ were forwarding to a method which PyStemmer doesn't provide and never seems
1029
+ to have provided.
1030
+
1031
+ + stemwords.py: Make -i and -o optional. The command syntax already
1032
+ suggested they were, but actually we gave an error if they were omitted.
1033
+
1034
+ + Fix code generated for string-$ (which isn't used by any of the algorithms
1035
+ we currently ship).
1036
+
1037
+ + Fix `->` to work when the slice is empty - previously it incorrectly
1038
+ signalled `f` for this case. Luckily this case is not exercised by any
1039
+ current algorithms (#242)
1040
+
1041
+ + Remove deprecated licence classifier which now triggers a deprecation
1042
+ warning from Python's setuptools. We already specify the licensing in the
1043
+ now preferred way via `license=` with a SPDX licence expression.
1044
+
1045
+ * Optimisations:
1046
+
1047
+ + Optimise single-character string literal checks in the same way we already
1048
+ do for C. This seems to be measurably faster (tested with Turkish which
1049
+ has lots of single character literal tests).
1050
+
1051
+ + Groupings are now implemented via a Python set, or a string for small
1052
+ groupings.
1053
+
1054
+ + Eliminate use of exception in code generated for `or`. We can instead wrap
1055
+ the code in a loop and use `break`.
1056
+
1057
+ + Eliminate use of exception in `goto` and `gopast`. We can just use `break`
1058
+ here to exit the `while` loop we're also inside and move the `except` from
1059
+ the previous `try` onto the `while`.
1060
+
1061
+ + Avoid using a temporary for `hop` with a constant argument as benchmarking
1062
+ with timeit shows this is faster.
1063
+
1064
+ + Optimise string test by using startswith()/endswith() with suitable
1065
+ start/end parameters which avoids creating a temporary substring and avoids
1066
+ an explicit limit check. This speeds up artificial testcases consisting of
1067
+ `goto 'the'` by 10%.
1068
+
1069
+ + Optimise among when all actions are `<-` with a literal string. We now
1070
+ generate a single call to slice_from() with the argument obtained by
1071
+ indexing into an array of literal strings. See #227.
1072
+
1073
+ + Reduce overhead of code to forward to PyStemmer, both when forwarding and
1074
+ when using the pure Python stemmers.
1075
+
1076
+ + Reuse exception classes much more. This reduces the number of labN classes
1077
+ we need by 142 over all the current stemmers.
1078
+
1079
+ + Change slice_check() to assert its conditions. In C we must not perform
1080
+ string slicing if slice_check() fails because that could result in writing
1081
+ outside of the allocated buffer, but it's not problematic in this way for
1082
+ Python, and the situations which slice_check() checks for should only
1083
+ happen with a Snowball program containing logic errors, or for bugs in the
1084
+ Snowball compiler or its runtime (or possibly in the Python interpreter,
1085
+ OS, hardware, etc). Therefore assert() seems an appropriate choice.
1086
+
1087
+ * Code quality:
1088
+
1089
+ + Use _ as dummy loop variable. We don't use the loop variable's value, and
1090
+ the loop itself tracks the current iteration so generating nested loops
1091
+ using `_` as the loop variable works correctly.
1092
+
1093
+ + Avoid mysterious gaps in the numbering of variables in the generated code.
1094
+ This was already done for the other languages, but I missed Python it
1095
+ seems.
1096
+
1097
+ + Avoid generating unused lab0 class for a Snowball program which doesn't use
1098
+ any failure labels.
1099
+
1100
+ + Avoid generating a blank line at start of the body of a Snowball `loop`.
1101
+
1102
+ + stemwords.py: Replace deprecated `codecs.open()` with built-in `open()`.
1103
+ Patch from Dmitry Shachnev.
1104
+
1105
+ * Documentation:
1106
+
1107
+ + Remove unnecessary semicolons from Python code in docs.
1108
+
1109
+ * Other changes:
1110
+
1111
+ + Remove Python 2 support. We stopped officially supporting it in Snowball
1112
+ 2.1.0, but now we've actually stripped out support. Versions of Python ≥
1113
+ 3.3 continue to be supported. Patch from Dmitry Shachnev (#212).
1114
+
1115
+ Rust
1116
+ ----
1117
+
1118
+ * Bug fixes:
1119
+
1120
+ + Fix code generated for string-$ (which isn't used by any of the algorithms
1121
+ we currently ship).
1122
+
1123
+ + A snowball `external` could not previously be called from within the
1124
+ Snowball program. This is allowed by the Snowball language, but none of
1125
+ the shipped stemmers do this, and it's unlikely any stemmer would, but
1126
+ perhaps it's useful if you use Snowball for other string-processing tasks.
1127
+
1128
+ + The generated code previously treated an empty string returned by
1129
+ slice_to() as an error, but this was buggy since if the slice is empty
1130
+ the return value will be an empty string. The helper doesn't try
1131
+ to signal an error with an empty string so we can just drop this
1132
+ check. Luckily this case is not exercised by any current algorithms.
1133
+ See #242.
1134
+
1135
+ + Fix incorrect assumption that an among containing the empty string
1136
+ always matched, even if the empty string had a gating function.
1137
+ This construct is not used by any existing stemmers.
1138
+
1139
+ * Optimisations:
1140
+
1141
+ + Avoid calling among helper when the among contains only strings which are
1142
+ one byte long, no among functions are used, and there are no actions.
1143
+
1144
+ * Code quality:
1145
+
1146
+ + Fix formatting of code generated for `goto`/`gopast` applied to a grouping
1147
+ or inverted grouping. This is just a cosmetic problem - functionally it
1148
+ was correct. The poor formatting was introduced in v3.0.0.
1149
+
1150
+ + Runtime helpers slice_del() and slice_from() always returned true, but
1151
+ the generated code included failure checks in case false was returned.
1152
+ These helpers no longer return anything, and the checks are gone.
1153
+
1154
+ + Generate space after condition in integer test (purely cosmetic).
1155
+
1156
+ New Code Generators
1157
+ -------------------
1158
+
1159
+ * Add Dart generator from Ryan Heise (#156, #250).
1160
+
1161
+ * Add PHP generator from Tim Whitlock and Olly Betts (#243). Requires PHP 8.3
1162
+ or later, which allows us to use typed class constants.
1163
+
1164
+ * Add Zig backend from AJ Roetker. Requires Zig 0.16.0 or later.
1165
+
1166
+ Snowball Language Changes
1167
+ -------------------------
1168
+
1169
+ * `delete` and `<-` now update the slice end. The manual said that after
1170
+ `[` and `]` "the slice ends will retain the same values until altered",
1171
+ which doesn't make it clear what happens for operations which modify the
1172
+ text the slice ends are in.
1173
+
1174
+ The existing handling here was inconsistent between commands: `delete`
1175
+ and `<-` left the slice ends on the same numeric positions, while
1176
+ `attach` and `insert` adjusted the slice ends to leave the slice marking
1177
+ the equivalent substring of the updated string. When working in UTF-8
1178
+ the slice end could end up in the middle of a multi-byte character after
1179
+ `delete` or `<-`, which seems especially undesirable.
1180
+
1181
+ I talked this over with Martin Porter and we've agreed that it makes
1182
+ sense for `delete` and `<-` to also update the slice ends (in fact only
1183
+ the right end needs adjusting) and I've clarified the wording in the
1184
+ manual.
1185
+
1186
+ Existing algorithms we ship don't rely on what the slice is set to after
1187
+ these commands.
1188
+
1189
+ * The slice is now specified to be unset after `=S` (so the same state as at
1190
+ the start of the program). Previously Snowball attempted to adjust the slice
1191
+ after `=S`, but there isn't an obvious adjustment in general because it can
1192
+ replace part of the content of the slice. Martin said he'd not thought of
1193
+ this case, and we've concluded it's best to adjust the Snowball language
1194
+ definition.
1195
+
1196
+ New stemming algorithms
1197
+ -----------------------
1198
+
1199
+ * Add Czech stemmer from Olly Betts and Jim O’Regan (#151).
1200
+
1201
+ * Add Persian (Farsi) stemmer from Saeid Darvish (#181).
1202
+
1203
+ * Add Polish stemmer from Dmitry Shachnev (#245).
1204
+
1205
+ * Add Sesotho stemmer from Kamohelo Lebjane (#260).
1206
+
1207
+ Behavioural changes to existing algorithms
1208
+ ------------------------------------------
1209
+
1210
+ * Danish:
1211
+
1212
+ + Adjust to handle apostrophe (#187).
1213
+
1214
+ + Restrict undoubling to valid cases. Coverage showed that a number of the
1215
+ consonants we would undouble never occur in our Danish vocabulary. Testing
1216
+ a larger list didn't find any matches for Danish words either, so restrict
1217
+ the undoubling which reduces the potential for damage to foreign words and
1218
+ should be a little more efficient.
1219
+
1220
+ * English:
1221
+
1222
+ + Restore exception for `skis` so it stems to `ski`. This reverts a
1223
+ change made erroneously in Snowball 3.0.0.
1224
+
1225
+ + Improve the stemming of some words starting `inter`:
1226
+ - We now avoid conflating intern, internal, international and
1227
+ internment.
1228
+ - We now conflate interfere/interferes/interference with
1229
+ interfered/interfering.
1230
+ - The stem of `interval` is now `interval` rather than `interv`, which
1231
+ is mostly a cosmetic change as no unrelated words stem to `interv`.
1232
+
1233
+ * Estonian:
1234
+
1235
+ + Handle apostrophe (#187).
1236
+
1237
+ * Finnish:
1238
+
1239
+ + Handle apostrophe (#187).
1240
+
1241
+ + Improve fallback from illative rules. If a word ends -han, -hen, -hin,
1242
+ -hon, -hän or -hön but the vowel before does not match we were not removing
1243
+ a suffix in case_ending, we now fallback to handling as a genitive and
1244
+ remove -n. This changes how we handle about 90 words - almost all for the
1245
+ better, most of the rest seem neutral changes.
1246
+
1247
+ + Allow "ø" to match with -hön as this is seen with Norwegian place names,
1248
+ e.g. Bodøhön.
1249
+
1250
+ + Remove illative form -hun. This improves the stemming of 14 words in
1251
+ our test vocabulary.
1252
+
1253
+ * German:
1254
+
1255
+ + Handle apostrophe (#187).
1256
+
1257
+ * Italian:
1258
+
1259
+ + Handle elisions (#187).
1260
+
1261
+ * Lithuanian:
1262
+
1263
+ + Don't remove -er- before normal suffixes. These aren't real grammatic
1264
+ suffixes and seem to have been included mainly to try to conflate ancient
1265
+ forms of the Lithuanian word for "sister" (e.g. "sesers") with modern forms
1266
+ (e.g. "sesė"). We weren't even doing a complete job there however as
1267
+ "seserimis" and "seseris" were not handled. Removing these suffixes
1268
+ entirely means we no longer try to conflate the ancient and modern forms
1269
+ here, but at least all the forms of the old word get grouped, as do all
1270
+ forms of the new word. The stemming for ~150 other words is also improved,
1271
+ without obvious downsides. Patch from Justas Sakalauskas (#263).
1272
+
1273
+ + Remove trailing apostrophe as final step - an apostrophe is sometimes used
1274
+ to separate a Lithuanian ending on an international word (#187).
1275
+
1276
+ * Norwegian:
1277
+
1278
+ + Adjust to handle apostrophe (#187).
1279
+
1280
+ * Polish:
1281
+
1282
+ + Remove optional apostrophe after removing suffix. Polish uses an
1283
+ apostrophe to separate loanwords from native suffixes. (The correct use is
1284
+ to mark the elision of the final sound of a loanword before a Polish
1285
+ inflectional endings, but it's also often used with any loanword) (#187).
1286
+
1287
+ Optimisations to existing algorithms
1288
+ ------------------------------------
1289
+
1290
+ * English:
1291
+
1292
+ + Optimise -eed, -eedly handling by performing the much cheaper R1 check
1293
+ before the among of exceptional cases.
1294
+
1295
+ * Esperanto:
1296
+
1297
+ + Eliminate use of among functions. It's easy to avoid them, and they come
1298
+ with a performance overhead in some target languages. For C, the new
1299
+ version is 0.09% faster (from cachegrind estimated cycle count).
1300
+
1301
+ * Indonesian:
1302
+
1303
+ + Avoid use of among functions, which gives a 1.9% speed up for C (from
1304
+ cachegrind estimated cycle count).
1305
+
1306
+ * Lithuanian:
1307
+
1308
+ + Minor simplification/optimisation by relying on Snowball restoring the
1309
+ cursor on failure.
1310
+
1311
+ * Turkish:
1312
+
1313
+ + Simplify `not test C` to just `not C`. If C succeeds, then the `not` fails
1314
+ and the cursor will get restored by whatever handles that signal.
1315
+
1316
+ Code clarity improvements to existing algorithms
1317
+ ------------------------------------------------
1318
+
1319
+ * Finnish: Rename `V1` and `LONG` to match the names used in the algorithm
1320
+ description on the website.
1321
+
1322
+ * Italian: Eliminate use of legacy among starter.
1323
+
1324
+ Build system
1325
+ ------------
1326
+
1327
+ * The default flags used with `ar` are now `-cr` instead of `-cru`. Many
1328
+ Linux distros configure `ar` to use option `D` (deterministic mode) by
1329
+ default, which was triggering a warning that option `u` is ignored.
1330
+ Option `u` is just a minor optimisation for the case where the archive
1331
+ already exist and only some object files have change, so it seems best
1332
+ to just not try to use it and avoid the warning.
1333
+
1334
+ Make variable `ARFLAGS` can now be used to specify flags to use with `ar`,
1335
+ so if you want to continue using `-cru`, you can use:
1336
+
1337
+ make ARFLAGS=-cru
1338
+
1339
+ If `D` is on by default in your `ar`, you'll actually want:
1340
+
1341
+ make ARFLAGS=-cruU
1342
+
1343
+ * Add comment documenting how to use iconv.py (simple pure-Python alternative
1344
+ which allows running the testsuite without iconv installed).
1345
+
1346
+ * `make clean` now removes all built files for all target languages, and is
1347
+ now tested by CI to ensure this doesn't regress.
1348
+
1349
+ * Make "make check_utf8" parallel-safe by avoiding writing the stemmed output
1350
+ to disk by default (except for Arabic). To get the output saved as tmp.txt
1351
+ on error for debugging you can now use: `make SAVETMP=1 check_utf8`. Patch
1352
+ from Adam Turner (#237, #238).
1353
+
1354
+ * Ada: Fix parallel build by adding missing dependency from .adb to the
1355
+ corresponding .sbl file (#237, #238).
1356
+
1357
+ * Go: Use `$(go)` for `go generate` as well.
1358
+
1359
+ * Python: Omit output "(THIN_FACTOR=)" if set empty.
1360
+
1361
+ * Add SNOWBALL_FLAGS, intended to allow passing options such as `-comments`
1362
+ and `-coverage` during development and debugging.
1363
+
1364
+ * Add make targets to assist comparing generated code before and after a
1365
+ compiler change: `baseline-create`, `generate` and `baseline-diff`.
1366
+
1367
+ * We now have CI testing that the Snowball compiler builds as C99 (we were
1368
+ already testing that the generated C code builds as C90). Fixes #283,
1369
+ reported by Domingo Alvarez Duarte.
1370
+
1371
+ Testsuite
1372
+ ---------
1373
+
1374
+ * New testsuite for the Snowball compiler which tests parsing, errors and
1375
+ warnings.
1376
+
1377
+ * New runtime testsuite which tests the implementation of Snowball language
1378
+ features in each supported target language. These provide something much
1379
+ more like a proper set of unit tests rather than relying on checking all the
1380
+ algorithms produce the expected output to validate all the target language
1381
+ generators. These tests are run with -comments on to provide some test
1382
+ coverage for this option. Fixes #157.
1383
+
1384
+ * stemtest: Add more number testcases, relocated to here from finnish/voc.txt.
1385
+ They're better by stemtest as we want to avoid any stemmer damaging numbers,
1386
+ and testcases here can easily be run for all stemmers.
1387
+
1
1388
  Snowball 3.0.1 (2025-05-09)
2
1389
  ===========================
3
1390
 
@@ -323,15 +1710,15 @@ Behavioural changes to existing algorithms
323
1710
  contain other accented characters and it seems better for the stemmer to
324
1711
  handle such words the same way regardless of the encoding in use.
325
1712
 
326
- * English: Replace '-ogist' with '-og' to conflate "geologist" and "geology", etc.
327
- Suggested by Marc Schipperheijn on snowball-discuss.
1713
+ * English: Replace '-ogist' with '-og' to conflate "geologist" and "geology",
1714
+ etc. Suggested by Marc Schipperheijn on snowball-discuss.
328
1715
 
329
1716
  * English: Add extra condition to undoubling. We no longer undouble if the
330
1717
  double consonant is preceded by exactly "a", "e" or "o" to avoid conflating
331
1718
  "add"/"ad", "egg"/"eg", "off"/"of", etc. Fixes #182, reported by Ed Page.
332
1719
 
333
- * English: Avoid conflating 'emerge' and 'emergency'. Reported by Frederick Ross
334
- on snowball-discuss.
1720
+ * English: Avoid conflating 'emerge' and 'emergency'. Reported by Frederick
1721
+ Ross on snowball-discuss.
335
1722
 
336
1723
  * English: Avoid conflating 'evening' and 'even'. Reported by Ann B on
337
1724
  snowball-discuss.
@@ -429,7 +1816,7 @@ Behavioural changes to existing algorithms
429
1816
 
430
1817
  * Spanish: Handle -acion like -ación and -ucion like -ución. It's apparently
431
1818
  common to miss off accents in Spanish, and there are examples in our test
432
- vocabulary that these change helps. Proposed by Damian Janowski.
1819
+ vocabulary that these changes help. Proposed by Damian Janowski.
433
1820
 
434
1821
  * Swedish: Replace suffix "öst" with "ös" when preceded by any of 'iklnprtuv'
435
1822
  rather than just 'l'. The new rule only requires the "öst" to be in R1
@@ -653,7 +2040,7 @@ Compiler
653
2040
  in the case where we were in a stringdef (but correct if we weren't).
654
2041
 
655
2042
  * Eliminate special handling for among starter. We now convert the starter
656
- to be a command before the among, adding an explict substring if there
2043
+ to be a command before the among, adding an explicit substring if there
657
2044
  isn't one.
658
2045
 
659
2046
  * We now warn if the body of a `repeat` or `atleast` loop always signals
@@ -831,7 +2218,7 @@ Code generation improvements
831
2218
  + Constant numeric subexpressions and constant numeric tests are now
832
2219
  evaluated at Snowball compile time.
833
2220
 
834
- + Simplify the following degnerate `loop` and `atleast` constructs where
2221
+ + Simplify the following degenerate `loop` and `atleast` constructs where
835
2222
  N is a compile-time constant:
836
2223
 
837
2224
  - loop N C where N <= 0 is a no-op.