mittens 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d5cbb7de0a88546460ef6f8dc512817d97aa2664157c7226ab280b8bf5f22a0a
-  data.tar.gz: 4778ddc8fc0088ba76dea22e03530db785edaa6faa7ddd91df9a87f5f56ee92b
+  metadata.gz: 94a898f3a58eb04ef1ec07954623b14eea4161f21d330a3340ce98916c867755
+  data.tar.gz: 1fd6926ef1e62564b39ba3ea1295831493c4e52ea9a4316c2d06918b3d8d6a50
 SHA512:
-  metadata.gz: c86517350ec8e55685ac3b4fa93f4ff8de0e279997cae68336cb7dbeb49bac18bb791190648ca9c7660689bb51e88bec7413e6b5e2f0f33521f6fe110959bf80
-  data.tar.gz: f37a446b8205c9dcdea4cad7ad2811780db5e39bf8f89f070847f6df9ebb0d9bf48518042252da427e3b4b09c88031da7632086c9e58286da830f945e4d8e109
+  metadata.gz: ffcad206d69a72e22890032b19041af7f11abf4ef5ec7f51e6dce96ae1f2b2b9091e24cdf8a301202354508fd5cde1bd0421163b1351ec4ed133f46e120bd8f8
+  data.tar.gz: 0ece86c59547de3722aa5bf90afbb87e3f6a9b12c2db6323e56260e7dc3b95061b7118dd732d81ba33e7407b057c695a92aa60d4b66ea6811d6a933983037f26

data/CHANGELOG.md

@@ -1,3 +1,7 @@
+## 0.3.0 (2025-09-12)
+
+- Updated Snowball to 3.0.1
+
 ## 0.2.0 (2025-04-03)
 
 - Dropped support for Ruby < 3.2

data/README.md

@@ -2,7 +2,7 @@
 
 Stemming for Ruby, powered by [Snowball](https://github.com/snowballstem/snowball)
 
-:snowflake: Supports 28 languages
+:snowflake: Supports 30 languages
 
 [![Build Status](https://github.com/ankane/mittens/actions/workflows/build.yml/badge.svg)](https://github.com/ankane/mittens/actions)
 
@@ -25,7 +25,7 @@ stemmer = Mittens::Stemmer.new
 Stem a word
 
 ```ruby
-stemmer.stem("tomatos") # "tomato"
+stemmer.stem("tomatoes") # "tomato"
 ```
 
 ## Languages
@@ -36,7 +36,7 @@ Specify the language
 stemmer = Mittens::Stemmer.new(language: "french")
 ```
 
-Supports `arabic`, `armenian`, `basque`, `catalan`, `danish`, `dutch`, `english`, `finnish`, `french`, `german`, `greek`, `hindi`, `hungarian`, `indonesian`, `irish`, `italian`, `lithuanian`, `nepali`, `norwegian`, `porter`, `portuguese`, `romanian`, `russian`, `serbian`, `spanish`, `swedish`, `tamil`, `turkish`, and `yiddish`
+Supports `arabic`, `armenian`, `basque`, `catalan`, `danish`, `dutch`, `dutch_porter`, `english`, `esperanto`, `estonian`, `finnish`, `french`, `german`, `greek`, `hindi`, `hungarian`, `indonesian`, `irish`, `italian`, `lithuanian`, `nepali`, `norwegian`, `porter`, `portuguese`, `romanian`, `russian`, `serbian`, `spanish`, `swedish`, `tamil`, `turkish`, and `yiddish`
 
 ## History
 

data/lib/mittens/version.rb

@@ -1,3 +1,3 @@
 module Mittens
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

data/vendor/snowball/.github/workflows/ci.yml

@@ -0,0 +1,216 @@
+name: CI
+
+# Use bash by default on all platforms.
+defaults:
+  run:
+    shell: bash
+
+on:
+  push:
+    paths-ignore:
+      - '*.rst'
+      - NEWS
+  pull_request:
+    branches: master
+    paths-ignore:
+      - '*.rst'
+      - NEWS
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  build:
+    strategy:
+      matrix:
+        include:
+        - name: "C distribution build"
+          CFLAGS_DIST_BUILD: '-O2 -Wall -W -std=c90 -Wmissing-prototypes -Wmissing-declarations -Wshadow -Wdeclaration-after-statement -Werror'
+        - name: "C distribution build (clang)"
+          CFLAGS_DIST_BUILD: '-O2 -Wall -W -std=c90 -Wmissing-prototypes -Wmissing-declarations -Wshadow -Wdeclaration-after-statement -Werror'
+          CC: clang
+        - name: "C"
+          c_tests: y
+          WERROR: '-std=c99 -Werror'
+        - name: "C (clang)"
+          c_tests: y
+          WERROR: '-std=c99 -Werror'
+          CC: clang
+        - name: java
+          JAVA: java
+          JAVAC: javac
+          JAVACFLAGS: '-Xlint:all -Werror'
+        - name: go_old
+          os: 'ubuntu-22.04'
+          apt_packages: 'golang-1.13'
+          GO: go
+        - name: go_new
+          os: 'ubuntu-24.04'
+          apt_packages: 'golang-1.22'
+          GO: go
+        - name: javascript_node
+          JSRUN: node
+          apt_packages: 'nodejs'
+        - name: rust
+          RUST: rust
+          apt_packages: 'rustc'
+        - name: csharp
+          MCS: mcs
+          apt_packages: 'mono-devel'
+        - name: Pascal
+          FPC: fpc
+          apt_packages: 'fpc'
+        - name: Python 3.8
+          PYTHON_VERSION: 3.8
+          os: 'ubuntu-22.04'
+          # The pure Python versions run slowly - when we used travis for CI
+          # we used to need to thin the testdata for languages such as Arabic
+          # where there's a lot to avoid the build exceeding the maximum time
+          # allowed for a CI job.  GHA allows jobs to take up to 6 hours so
+          # we should no longer need to do this.
+          THIN_FACTOR: 1
+        - name: Python 3.10
+          PYTHON: python3.10
+          os: 'ubuntu-22.04'
+          apt_packages: 'python3.10'
+          THIN_FACTOR: 1
+        - name: Python 3.12
+          PYTHON: python3.12
+          os: 'ubuntu-24.04'
+          apt_packages: 'python3.12'
+          THIN_FACTOR: 1
+        - name: Python (pypy3)
+          PYTHON: pypy3
+          apt_packages: 'pypy3'
+        - name: Ada
+          gprbuild: gprbuild
+          apt_packages: 'gnat gprbuild'
+        - name: Windows (C)
+          os: windows-latest
+          c_tests: y
+          ccache: sccache
+        - name: Windows (Go)
+          os: windows-latest
+          GO: go
+          MAKE: mingw32-make
+          mingw64_packages: 'mingw-w64-ucrt-x86_64-go'
+          ccache: sccache
+
+      fail-fast: false
+
+    runs-on: ${{ matrix.os || 'ubuntu-latest' }}
+
+    env:
+      CC: ${{ matrix.CC || 'gcc' }}
+      MAKE: ${{ matrix.MAKE || 'make' }}
+      STEMMING_DATA: 'snowball-data'
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v4
+      with:
+        show-progress: false
+
+    - name: Checkout data
+      run: |
+        # Try to check out a branch of the same name from the snowball-data
+        # repo sibling of this snowball repo, so that PRs requiring changes to
+        # both can be CI tested easily.
+        #
+        # For a PR, GHA will have merged the PR branch into upstream master so
+        # we need to similarly merge the snowball-data branch into upstream
+        # master of the snowball-data repo as there may be changes there
+        # required by snowball master.
+        #
+        # If there's no such branch (or repo) we just use the standard
+        # snowball-data repo's default branch.  If there is such a branch but
+        # the merge fails, we treat that as a fatal error.
+        UPSTREAM_REPO_URL=https://github.com/snowballstem/snowball-data.git
+        if [ -n "$GITHUB_HEAD_REF" ] ; then
+          # Pull-request.
+          GH_BRANCH=${GITHUB_HEAD_REF}
+          GH_REPO_OWNER=${GITHUB_ACTOR}
+          GH_REPO_URL=https://github.com/$GH_REPO_OWNER/snowball-data.git
+          git clone "$UPSTREAM_REPO_URL"
+          cd snowball-data
+          git remote add pr "$GH_REPO_URL"
+          git config --global user.email "ci@example.org"
+          git config --global user.name "CI"
+          echo "Trying branch $GH_BRANCH from $GH_REPO_URL"
+          if git fetch pr && git branch --track "$GH_BRANCH" pr/"$GH_BRANCH" ; then
+            git merge "$GH_BRANCH"
+          else
+            echo "Falling back to $UPSTREAM_REPO_URL"
+          fi
+        else
+          # Push.
+          GH_BRANCH=${GITHUB_REF_NAME}
+          GH_REPO_OWNER=${GITHUB_REPOSITORY_OWNER}
+          GH_REPO_URL=https://github.com/$GH_REPO_OWNER/snowball-data.git
+          echo "Trying branch $GH_BRANCH from $GH_REPO_URL"
+          if ! git clone -b "$GH_BRANCH" "$GH_REPO_URL" ; then
+            echo "Falling back to $UPSTREAM_REPO_URL"
+            git clone "$UPSTREAM_REPO_URL"
+          fi
+        fi
+    - name: Install CCache
+      uses: hendrikmuhs/ccache-action@v1
+      with:
+        key: ${{ matrix.name }}
+        variant: ${{ matrix.ccache || 'ccache' }}
+    - name: Install Ubuntu packages
+      if: matrix.apt_packages
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y ${{ matrix.apt_packages }}
+    - name: Install mingw64 packages
+      if: matrix.mingw64_packages
+      uses: msys2/setup-msys2@v2
+      with:
+        msystem: ucrt64
+        install: base-devel ${{ matrix.mingw64_packages }}
+    - name: Build
+      run: $MAKE CC="${{ matrix.ccache || 'ccache' }} $CC"
+    - name: Test C dist
+      if: matrix.CFLAGS_DIST_BUILD
+      run: |
+        pip install setuptools build
+        $MAKE dist
+        mkdir tmp
+        cd tmp
+        tar xf ../dist/libstemmer_c-*.tar.gz
+        cd libstemmer_c-*
+        $MAKE CFLAGS="${{ matrix.CFLAGS_DIST_BUILD }}"
+    - name: Test C
+      if: matrix.c_tests
+      run: $MAKE check CC="$CC"
+    - uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.PYTHON_VERSION }}
+      if: matrix.PYTHON_VERSION
+    - name: Test Python
+      if: matrix.PYTHON || matrix.PYTHON_VERSION
+      run: $MAKE check_python python="${{ matrix.PYTHON || 'python' }}" THIN_FACTOR="${{ matrix.THIN_FACTOR }}"
+    - name: Test Java
+      if: matrix.JAVA && matrix.JAVAC
+      run: $MAKE check_java JAVA="${{ matrix.JAVA }}" JAVAC="${{ matrix.JAVAC }}" JAVACFLAGS="${{ matrix.JAVACFLAGS }}"
+    - name: Test C#
+      if: matrix.MCS
+      run: $MAKE check_csharp MCS="${{ matrix.MCS }}"
+    - name: Test Javascript
+      if: matrix.JSRUN
+      run: $MAKE check_js JSRUN="${{ matrix.JSRUN }}"
+    - name: Test Rust
+      if: matrix.RUST
+      run: $MAKE check_rust RUST="${{ matrix.RUST }}"
+    - name: Test Go
+      if: matrix.GO
+      run: |
+        go mod init github.com/snowballstem/snowball
+        $MAKE check_go GO="${{ matrix.GO }}"
+    - name: Test Pascal
+      if: matrix.FPC
+      run: $MAKE check_pascal FPC="${{ matrix.FPC }}"
+    - name: Test Ada
+      if: matrix.gprbuild
+      run: $MAKE check_ada gprbuild="${{ matrix.gprbuild }}"

data/vendor/snowball/CONTRIBUTING.rst

@@ -1,41 +1,35 @@
-Adding a new stemming algorithm
+General contribution guidelines
 ===============================
 
-This needs PRs against three repositories.  Name the branch the same for
-at least `snowball` and `snowball-data`, push to the latter repo first, and the
-CI should use your new vocabulary list when running the testsuite.
-
-Some points to note about algorithm implementations:
+We don't have a formally defined coding style guide, but please strive to
+make new/changed code look like the code around it.
 
-* Avoid literal non-ASCII characters in snowball string literals - they will
-  work OK for languages that use UTF-8, but not wide-character Unicode or other
-  encodings.  Instead use ``stringdef`` like the existing stemmers do, and
-  please use the newer `U+` notation rather than the older ``hex`` or
-  ``decimal`` as this allows us to support different encodings without having
-  to modify the source files - for example::
+Use spaces-only for indentation except where there's a syntax reason (e.g.
+``GNUmakefile``) or a strong convention (e.g. Go's standard seems to be tabs,
+and ``gofmt`` reindents code using tabs).
 
-    stringdef o" {U+00F6}
-    define foo 'o{o"}'
+Avoid adding trailing whitespace on lines.  Make sure there's a newline
+character at the end of new text files.
 
-  not::
+Avoid mixing code reformatting changes with functional changes - doing so
+makes it harder to review patches.
 
-    stringdef o" hex F6
-    define foo 'o{o"}'
+Adding a new stemming algorithm
+===============================
 
-  and definitely not::
+To add a new stemming algorithm you need to submit PRs against three
+repositories.  See below for details of what's needed in each of
+these.
 
-    define foo 'oö'
-
-  It's helpful to consistently use the same ``stringdef`` codes across the
-  different stemmers - the website has `guidance on what to use
-  <https://snowballstem.org/codesets/guide.html>`_ and a `list of stringdef
-  lines for common characters to cut and paste from
-  <https://snowballstem.org/codesets/latin-stringdef-list.txt>`_.
+Name the branch the same for at least `snowball` and `snowball-data` and push
+to `snowball-data` first, then the CI should use your new vocabulary list when
+running the testsuite.
 
 snowball repo
 -------------
 
-Add `.sbl` source to algorithms subdirectory.
+This is where the implementation of the new algorithm goes.  Add the `.sbl`
+source implementing it to the `algorithms/` subdirectory.
 
 Add entry to `libstemmer/modules.txt`, maintaining the current sorted order by
 the first column.  The columns are:
@@ -53,6 +47,35 @@ the first column.  The columns are:
   "639-2/T" and (if different) "639-2/B" columns.  For example, for the `Afar`
   language you'd put `afar,aa,aar` here.
 
+Some points to note about algorithm implementations:
+
+* Avoid literal non-ASCII characters in snowball string literals - they will
+  work OK for languages that use UTF-8, but not wide-character Unicode or other
+  encodings.  Instead use ``stringdef`` like the existing stemmers do, and
+  please use the newer `U+` notation rather than the older ``hex`` or
+  ``decimal`` as this allows us to support different encodings without having
+  to modify the source files - for example::
+
+    stringdef o" {U+00F6}
+    define foo 'o{o"}'
+
+  not::
+
+    stringdef o" hex F6
+    define foo 'o{o"}'
+
+  and definitely not::
+
+    define foo 'oö'
+
+  It's OK to use UTF-8 in comments.
+
+* It's helpful to consistently use the same ``stringdef`` codes across the
+  different stemmers - for languages using the latin alphabet our website has
+  `guidance on what to use <https://snowballstem.org/codesets/guide.html>`_ and
+  a `list of stringdef lines for common characters to cut and paste from
+  <https://snowballstem.org/codesets/latin-stringdef-list.txt>`_.
+
 snowball-data repo
 ------------------
 
@@ -68,9 +91,8 @@ you may be able to use the `wikipedia-most-common-words` script to generate
 one by extracting the most frequent words from a Wikipedia dump in the
 language the stemmer is for.  You need to specify the Unicode "script" (that's
 "script" in the sense of alphabet) to use - you can find the appropriate one
-by looking in the Unicode `Scripts.txt`::
-
-https://www.unicode.org/Public/13.0.0/ucd/Scripts.txt
+by looking in the Unicode `Scripts.txt
+<https://www.unicode.org/Public/13.0.0/ucd/Scripts.txt>`_.
 
 The script name is the second column, between `;` and `#`.  The first entries
 are all "Common" which isn't what you want - scroll down to get to the entries
@@ -88,36 +110,62 @@ practice too), but at some point "more" stops being "better".
 snowball-website repo
 ---------------------
 
-Create subdirectory of `algorithms/` named after the language.
-
-Create `stemmer.tt` which describes the stemming algorithm.  This is a
-"template toolkit" template which is essentially a mix of HTML and some
-macros for adding the navigation, sample vocabulary, etc.  See the
-existing `stemmer.tt` files for other algorithms for inspiration.
-
-If it is based on an academic paper, cite the paper and describe any difference
-between your implementation and that described in the paper (for example,
-sometimes papers have ambiguities that need resolving to re-implement the
-algorithm described).
-
-If you have a stopword list, add that as `stop.txt` in your new subdirectory.
-The `generate` script checks if such a file exists and if it does a link to
-it is automatically added.
-
-Link to your new `stemmer.tt` from `algorithms/index.tt`.
-
-Add a news entry to `index.tt`.
-
-Add the new stemmer to the online demo.  Assuming you have checkouts of the
-`snowball`, `snowball-data` and `snowball-website` repos in sibling
-directories:
-
-* run `make check_js` in the `snowball` repo
-* run `./update-js`
-* add the new stemmer to git with: `git add js/*-stemmer.js`
-* if the new language is written right-to-left (RTL) then add it to the check
-  in `demo.tt` (search for `rtl` to find the place to change.)
-* `git commit`.
+This is where a description of the new algorithm goes.  Experience from
+maintaining Snowball for many years has shown us that the most important
+points to cover are **WHY** particular things are done or are not done.
+
+For example, if a particular ending isn't removed because doing so causes
+problems in other cases it's really helpful to have that recorded.  Then
+if years later we get a bug report because this ending isn't removed we
+can easily answer, and don't have to try to contact you and hope you can
+remember, or try to work out why for ourselves.
+
+The original set of Snowball stemmers each have an English prose description
+of the algorithm which focuses on **WHAT** the algorithm does.  These might be
+helpful if you want to implement the algorithm from scratch in a separate
+language, but they've not proved very useful for maintaining the Snowball
+implementations - if the prose and Snowball code disagree we know something is
+wrong, but it's hard to know which is right!  Therefore we recommend to let
+the Snowball implementation describe what the algorithm does, and only comment
+on "**WHAT**" in cases where the implementation needs explanation to help
+the reader understand it.
+
+If your algorithm is based on an academic paper, cite the paper and describe
+any differences between your implementation and that described in the paper.
+For example, sometimes papers have ambiguities that need resolving to
+re-implement the algorithm described - see the `Hindi
+<https://snowballstem.org/algorithms/hindi/stemmer.html>`_ and `Indonesian
+<https://snowballstem.org/algorithms/indonesian/stemmer.html>`_
+stemming algorithms descriptions for examples.
+
+The mechanics of adding the algorithm description are:
+
+* Create subdirectory of `algorithms/` named after the language.
+
+* Create `stemmer.tt` which describes the stemming algorithm.  This is a
+  "template toolkit" template which is essentially a mix of HTML and some
+  macros for adding the navigation, sample vocabulary, etc.  See the
+  existing `stemmer.tt` files for other algorithms for how to use these
+  macros.
+
+* If you have a stopword list, add that as `stop.txt` in your new subdirectory.
+  The `generate` script checks if such a file exists and if it does a link to
+  it is automatically added.
+
+* Link to your new `stemmer.tt` from `algorithms/index.tt`.
+
+* Add a news entry to `index.tt`.
+
+* Add the new stemmer to the online demo.  Assuming you have checkouts of the
+  `snowball`, `snowball-data` and `snowball-website` repos in sibling
+  directories:
+
+  * run `make check_js` in the `snowball` repo
+  * run `./update-js`
+  * add the new stemmer to git with: `git add js/*-stemmer.js`
+  * if the new language is written right-to-left (RTL) then add it to the check
+    in `demo.tt` (search for `rtl` to find the place to change.)
+  * `git commit`.
 
 Adding a new programming language generator
 ===========================================
@@ -150,7 +198,7 @@ to improve further in the future.
 Key problems to solve
 ---------------------
 
-* A key problem to solve is how to map the required flow of control in response
+* You need to work out how to map the required flow of control in response
   to Snowball signals.
 
   In the generated C code this is mostly done using `goto`.  If your language
@@ -172,7 +220,7 @@ Key problems to solve
   matching C - i.e. the result should be truncated (rounded towards zero).
   Some languages lack a built-in integer division operation, or have one
   which instead implements rounding towards negative infinity.  Existing
-  backends with special handling handling here which may be useful to look at
+  backends with special handling here which may be useful to look at
   include Javascript, Pascal and Python.
 
 Don't hardcode algorithm names
@@ -199,6 +247,7 @@ Mechanics of adding a new generator
 Copy an existing `compiler/generator_*.c` for your new language and modify
 away (`generator.c` has the generator for C, but also some common functions
 so if you start from this one you'll need to remove those common functions).
+
 Please resist reformatting existing C code - there's currently a lot of code
 repeated in each generator which ought to be pulled out as common code, and
 if you reformat that just makes that job harder.
@@ -211,6 +260,6 @@ Add support to `compiler/driver.c`.
 
 Add targets to `GNUmakefile` to run tests for the new language.
 
-Hook up automated testing via CI in `.travis.yml`.
+Hook up automated testing via CI in `.github/workflows/ci.yml`.
 
 Add to the list of languages in `README.rst`.