secryst 0.1.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 919071d9eb29b220b762212fafe8f7f7ace0f0605cde7c0402dbbe8eac2584c6
-  data.tar.gz: de1b20e79c10d14261c71e0043e82a196b38b531da2fcdec0db44bcb992c7239
+  metadata.gz: 185de54f6ce981b7739770b422262452752460c5559e029e4b5039f67f6483ef
+  data.tar.gz: df7df11ff4fc5a476074e8155448a3b1ab0577ab693c37e796a98c0cf7934af8
 SHA512:
-  metadata.gz: 94bfb8aff36218341cd5baa3bcb0eb48e5be147c54f4a8b072e25a82ae027c07bacc7769fafb7f657cff9f638f9a0a4a21385108f0a45384bdf1a7796c74ea67
-  data.tar.gz: a1fee058281400822908c30a2d5fcf27ceb320dfed8d78878986460323d4028e78fa89db1579146870864fd2104238ee038e9f7468710863ed16ab6c8652ce03
+  metadata.gz: 1c51c600385d5a05e9f9298925f2ba7fb1c32a49b26f0345d877c8a11da04459a50d4adb5a23aedac480212e5b83af621a917db472382194ddeb716bd2ae5c4f
+  data.tar.gz: 558249e6dcf7ec5cb927512f6487ee8edd911cb06f9f1685bd4ab1b00ec651e230940880cc2c0f3cd61a095a37c3dc0cad046ea4299b25acdd5ee2669ba35df4

data/README.adoc

@@ -1,3 +1,53 @@
+== secryst 1.0 — modernized
+
+*Secryst* is coined from *scrying* + *crystal*: gazing into an opaque
+script to reveal its hidden reading — the vowels, phonemes, and word
+boundaries that live in the reader's mind but not on the page. This gem
+is the Ruby crystal of the family (Python: `pip install secryst` at
+https://github.com/secryst/secryst-py[secryst/secryst-py]; TypeScript:
+`npm i secryst`). All crystals implement the *interscript-ml* contract —
+the `models.yaml` index, IMF v1 model zips, and shared golden sets
+(https://github.com/interscript/interscript-ml[interscript/interscript-ml]) — and
+are diffed against each other in CI. Overrides: `SECRYST_INDEX`,
+`SECRYST_CACHE` (default `~/.cache/secryst`).
+
+Secryst transliterates text using neural sequence-to-sequence models,
+with training in Python (PyTorch + HuggingFace) and inference in pure
+Ruby via https://github.com/secryst/onnxruntime[secryst/onnxruntime] —
+no libtorch, no native compilation.
+
+=== New in 1.0
+
+- *Byte-level models (ByT5 family)*: model zips now contain
+  `encoder.onnx` + `decoder.onnx` + `metadata.yaml` (`name: byt5`).
+  The tokenizer is UTF-8 bytes themselves (pad=0, EOS=1) — no vocab
+  files.
+- *Modern training stack*: ByT5/umt5 fine-tuning, Muon optimizer
+  variants, CTC/curriculum ablations, Modal pipelines (see `src/`,
+  `train_*.py`, `docs/RESULTS.md`).
+- *Legacy single-file ONNX zips* (vocabs.yaml based) still load.
+- The old Ruby trainer and libtorch/TorchScript path are removed —
+  training lives in Python now.
+
+=== Export a trained checkpoint
+
+....
+python scripts/export_onnx_byt5.py <hf_checkpoint_dir> khm-latn-byt5.zip
+....
+
+=== Use from Ruby
+
+....
+translator = Secryst::Translator.new(model_file: 'khm-latn-byt5.zip')
+translator.translate('ភាសាខ្មែរ')
+....
+
+=== CLI
+
+....
+secryst translate -f khm-latn-byt5.zip -i 'ភាសាខ្មែរ'
+secryst install <model-name>   # via Secrystfile remotes
+....
 = Secryst
 
 image:https://github.com/secryst/secryst/workflows/test/badge.svg["Build status", link="https://github.com/secryst/secryst/actions?workflow=test"]
@@ -17,22 +67,143 @@ The goal is to allow:
 * Users of the library in Ruby who only want to "use" the trained models should not require special bindings to run.
 
 
-== Status
+== Introduction
 
-Currently Secryst works with the Khmer Romanization system as cited below.
+Secryst works with a number of Romanization and transliteration systems.
+
+It is composed of two separate pieces of software (Ruby gems):
+
+* Secryst Translator: `secryst`, for users of trained models
+* Secryst Trainer: `secryst-trainer` for users who wish to train models
+
+Secryst models are platform-independent, interoperable
+and transferrable across installations.
+
+There are two types of Secryst models:
+
+Secryst ONNX model (recommended)::
+These can be run on any platform without `libtorch`
+(which requires installation of development tools).
+
+Secryst Torch model::
+These require `libtorch` to run. Secryst Torch models are trained in Ruby.
+
+Secryst Torch models can be converted to Secryst ONNX models
+via Python using instructions below.
+
+Secryst Trainer also supports checkpoint resumption.
+
+
+== Examples
+
+Under the `examples/` directory the following systems are provided
+
+* `examples/khm-latn`: Khmer Romanization
+* `examples/arm-latn`: Armenian Romanization based on Wikipedia data
 
 
 == Prerequisites
 
-* Ruby 2.7 (*MUST* - 2.6 does not work with the latest torch-rb)
 
-* `libtorch` (1.6.0)
+=== Secryst Translator
+
+==== Basic (usage of Secryst ONNX models only)
+
+If you only need to use Secryst ONNX models, there is no need
+to install `libtorch` and development tools.
+
+
+macOS and Ubuntu:
+
+[source,sh]
+----
+$ gem install bundler
+$ bundle install
+----
+
+
+On macOS:
+
+[source,sh]
+----
+$ gem install bundler
+$ bundle install
+----
+
+
+==== To use Secryst Torch models
+
+Usage of Secryst Torch models require `libtorch`, which
+requires installation of the following packages:
+
+* `libtorch` (1.8.1)
 * `fftw`
 * `gsl`
+
+And also you must explicitly add `torch` gem to your Gemfile:
+
+[source,ruby]
+----
+gem 'torch-rb', '~> 0.6'
+----
+
+On Ubuntu:
+
+[source,sh]
+----
+$ sudo apt-get -y install libfftw3-dev libgsl-dev unzip automake \
+  make gcc g++ libtorch libtorch-dev
+$ wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.9.0%2Bcpu.zip
+$ unzip libtorch-cxx11-abi-shared-with-deps-1.8.1%2Bcpu.zip
+
+$ gem install bundler
+$ bundle config build.torch-rb \
+    --with-torch-dir=$(pwd)/libtorch
+
+$ bundle install
+----
+
+
+On Fedora:
+
+[source,sh]
+----
+$ sudo dnf -y install ruby-devel gsl-devel fftw-devel rubygem-bundler unzip \
+  automake make gcc gcc-c++
+$ git clone --recurse-submodules --depth=5 -b release/1.7 https://github.com/pytorch/pytorch.git
+$ pushd pytorch
+$ python setup.py install
+$ popd
+$ TORCH_DIR=$HOME/.local/lib/python3.9/site-packages/torch # Ensure this dir exists
+$ ln -s lib $TORCH_DIR/lib64
+
+$ bundle config build.torch-rb \
+    --with-torch-dir=$TORCH_DIR
+
+$ bundle install
+----
+
+
+On macOS:
+
+[source,sh]
+----
+$ brew install libtorch gsl fftw automake gcc
+$ gem install bundler
+$ bundle config build.torch-rb \
+    --with-torch-dir=$(brew --prefix libtorch)
+$ bundle install
+----
+
+
+
+=== Secryst Trainer
+
+In order to use Secryst Trainer two additional components are necessary:
+
 * `lapack`
 * `openblas`
 
-
 On Ubuntu:
 
 [source,sh]
@@ -40,14 +211,17 @@ On Ubuntu:
 $ sudo apt-get -y install libfftw3-dev libgsl-dev libopenblas-dev \
     liblapack-dev liblapacke-dev unzip automake make gcc g++ \
     libtorch libtorch-dev
-$ wget https://download.pytorch.org/libtorch/cu102/libtorch-cxx11-abi-shared-with-deps-1.6.0.zip
-$ unzip libtorch-cxx11-abi-shared-with-deps-1.6.0.zip
+$ wget https://download.pytorch.org/libtorch/cu111/libtorch-cxx11-abi-shared-with-deps-1.8.1%2Bcu111.zip
+$ unzip libtorch-cxx11-abi-shared-with-deps-1.8.1%2Bcu111.zip
 
-$ gem install bundler -v "~> 2"
+$ gem install bundler
 $ bundle config build.torch-rb \
     --with-torch-dir=$(pwd)/libtorch
 
 $ bundle install
+
+# To enable ONNX training, you must also install the Python portions
+$ pip3 install -r requirements.txt
 ----
 
 
@@ -57,12 +231,17 @@ On macOS:
 ----
 $ brew install libtorch gsl lapack openblas fftw automake gcc
 
-$ gem install bundler -v "~> 2"
+$ gem install bundler
 $ bundle config build.numo-linalg \
-    --with-openblas-dir=/usr/local/opt/openblas \
-    --with-lapack-lib=/usr/local/opt/lapack
+    --with-openblas-dir=$(brew --prefix openblas) \
+    --with-lapack-lib=$(brew --prefix lapack)
+$ bundle config build.torch-rb \
+    --with-torch-dir=$(brew --prefix libtorch)
 
 $ bundle install
+
+# To enable ONNX training, you must also install the Python portions
+$ pip3 install -r requirements.txt
 ----
 
 
@@ -77,6 +256,223 @@ $ bundle exec gem uninstall numo-linalg
 ----
 
 
+== Usage
+
+Secryst provides a CLI for training models and re-using trained models.
+
+
+=== Using trained models
+
+You will need to install the `secryst` gem (prerequisites must be fulfilled):
+
+[source,sh]
+----
+$ gem install secryst
+----
+
+To utilize a trained model:
+
+[source,sh]
+----
+# Transform all individual lines of `--input_text_file`.
+# Specifying:
+#   - trained model zip archive at `--model-file`.
+#     Must include `metadata.yaml`, `vocabs.yaml` and
+#     an `.pth` or `.onnx` model file.
+
+secryst translate \
+  --input_text_file=examples/to-translate.txt \
+  --model-file=examples/checkpoints/checkpoint-500.zip
+----
+
+Both Secryst ONNX models and Secryst Torch models can
+be used with this command.
+
+
+=== Training models in Ruby (output: Secryst Torch model)
+
+Secryst supports training models in Ruby into the Secryst Torch model format.
+These created models can then be used by other users through the `secryst` gem.
+
+NOTE: To make a trained Secryst model available for all platforms,
+you should convert the Secryst Torch model into a Secryst ONNX model.
+
+You will need to install the `secryst-trainer` gem (prerequisites must be fulfilled):
+
+[source,sh]
+----
+$ gem install secryst-trainer
+----
+
+NOTE: The `secryst` gem will be automatically installed alongside `secryst-trainer`.
+
+
+Training a typical model:
+
+[source,sh]
+----
+# Train all individual lines of the file specified in `-i` to the
+# corresponding line in target `-t`.
+#
+# Specifying:
+#   - `max-epochs` specifies how many epochs training will be run
+#   - `log-interval` specifies how often should Secryst report on
+#     learning parameters.
+#   - `checkpoint-every` indicates how often Secryst saves a checkpoint
+#     file to `checkpoint_dir`, in the format `checkpoint-{epoch}.zip`.
+#   - `checkpoint_dir` specifies the directory to store checkpoint files. If some checkpoints are already in the directory, the training will continue from the latest
+
+secryst-trainer train \
+  -i 'data/khm-latn-small/input.csv' \
+  -t 'data/khm-latn-small/target.csv' \
+  --max-epochs=500 \
+  --log-interval=1 \
+  --checkpoint-every=50 \
+  --checkpoint_dir=examples/checkpoints
+----
+
+
+Training with all options:
+
+[source,sh]
+----
+# Train all individual lines of the file specified in `-i` to the
+# corresponding line in target `-t`.
+#
+# Specifying:
+#   - `batch-size` specifies the batch size for training
+#   - `max-epochs` specifies how many epochs training will be run
+#   - `log-interval` specifies how often should Secryst report on
+#     learning parameters.
+#   - `checkpoint-every` indicates how often Secryst saves a checkpoint
+#     file to `checkpoint_dir`, in the format `checkpoint-{epoch}.zip`.
+#   - `checkpoint_dir` specifies the directory to store checkpoint. If some checkpoints are already in the directory, the training will continue from the latest
+#   - `gamma` specifies the gamma value used
+#   - hyperparameters in a key-value pair format
+
+secryst-trainer train --model=transformer \
+  -i 'data/khm-latn-small/input.csv' \
+  -t 'data/khm-latn-small/target.csv' \
+  --batch-size=32 \
+  --max-epochs=500 \
+  --log-interval=1 \
+  --checkpoint-every=50 \
+  --checkpoint_dir=checkpoints \
+  --gamma=0.2 \
+  -h d_model:64 nhead:8 num_encoder_layers:4 num_decoder_layers:4 \
+    dim_feedforward:256 dropout:0.05 activation:relu
+----
+
+
+=== Convert Secryst Torch models to Secryst ONNX models
+
+Due to a limitation of https://pytorch.org/cppdocs/[libtorch's C++ interface]
+not being able to encode trained models in ONNX, we have to use PyTorch to
+convert Secryst Torch models into Secryst ONNX models.
+
+Secryst supports generation of Secryst ONNX models using PyTorch.
+
+First, clone this current repository.
+
+To convert a Secryst Torch model to a Secryst ONNX model, run:
+
+[source,sh]
+----
+python3 python/pth_to_onnx.py checkpoint.zip output.zip
+----
+
+The trained Secryst ONNX model can be used as usual:
+
+[source,sh]
+----
+bundle exec secryst translate --model-file output.zip -t texts.txt
+----
+
+
+
+=== Resuming training
+
+Secryst Trainer supports checkpoint resumption.
+
+It will detect whether you already have checkpoint model files in the model output
+directory (in the filename of `checkpoint-nnn.zip`), and attempt to
+resume training from there. Notice that in resuming training, you
+must use identical parameters and the identical training dataset,
+otherwise the process will throw out an error.
+
+
+=== Importing non-Secryst ONNX models
+
+You can easily utilize non-Secryst trained ONNX models in Secryst as well.
+
+You need to prepare a zip file with:
+
+* An `.onnx` model file
+* The `vocabs.yaml` file
+
+The `vocabs.yaml` file has to contain two keys - `input` and `target` (it's okay if they are the same), which contain all tokens from vocabulary placed in original order.
+
+Like this:
+
+[source,yaml]
+----
+input:
+- [UNK]
+- ...
+target:
+- [UNK]
+- ...
+----
+
+Then just utilize model usually as described above. You can find an example of this in the examples folder (`onnx_import.rb`).
+
+=== Training on GPU
+
+To allow training on CUDA drivers Secryst ships Python trainer.
+
+Install the python version 3.8, and required packages:
+
+[source,sh]
+----
+pip3 install -r requirements.txt
+----
+
+And start the training (all the options are the same as Ruby trainer):
+
+[source,sh]
+----
+python3 python/train.py -i 'data/khm-latn-small/input.csv' \
+  -t 'data/khm-latn-small/target.csv' \
+  --max-epochs=500 \
+  --log-interval=1 \
+  --checkpoint-every=50 \
+  --checkpoint-dir=examples/checkpoints
+----
+
+
+== Examples
+
+The Khmer transliteration system is implemented as an example.
+
+To run the training:
+
+[source,sh]
+----
+$ bundle exec examples/training.rb
+----
+
+To run translations through the transformer:
+
+[source,sh]
+----
+$ bundle exec examples/translating.rb
+----
+
+* Checkpoint files are generated as `examples/checkpoints/*.zip`
+* It includes `metadata.yaml`, `model.pth` and `vocabs.yaml` files
+
+
+
 
 == References
 
@@ -88,10 +484,6 @@ based on:
   Attention is all you need. 2017. In:
   _Advances in Neural Information Processing Systems_, pages 6000-6010.
 
-The sample transliteration system implemented is the Khmer system:
-
-* https://viblo.asia/p/nlp-khmer-word-segmentation-YWOZrgNNlQ0
-* https://viblo.asia/p/nlp-khmer-romanization-using-seq2seq-m68Z07OQKkG
 
 
 == Origin of name

data/bin/secryst

@@ -0,0 +1,42 @@
+#!/usr/bin/env ruby
+
+require "thor"
+require "secryst"
+
+class SecrystCli < Thor
+  def self.exit_on_failure?
+    true
+  end
+  desc "translate", "Translate input sequence using specified model"
+  method_option :source, :aliases => "-s", :desc => "Add a model index to provision the model"
+  method_option :input_text, :aliases => "-i", :desc => "String of text to translate"
+  method_option :input_text_file, :aliases => "-t", :desc => "Path to file with text to translate"
+  method_option :model_file, :aliases => "-f", :desc => "Specify model name or a checkpoint .zip file", :required => true
+
+  def translate
+    Secryst::Provisioning.add_remote(options[:source]) if options[:source]
+
+    translator = Secryst::Translator.new(
+      model_file: options[:model_file],
+    )
+
+    if options[:input_text]
+      puts translator.translate(options[:input_text])
+    elsif options[:input_text_file]
+      File.readlines(options[:input_text_file]).each do |line|
+        puts translator.translate(line)
+      end
+    end
+  end
+
+  desc "install <name>", "Installs a model locally"
+  method_option :source, :aliases => "-s", :desc => "Add a model index to provision the model"
+
+  def install(name)
+    Secryst::Provisioning.add_remote(options[:source]) if options[:source]
+
+    Secryst::Provisioning.locate(name)
+  end
+end
+
+SecrystCli.start

data/lib/secryst/byt5_onnx.rb

@@ -0,0 +1,103 @@
+require 'zip'
+require 'onnxruntime'
+require 'tmpdir'
+
+module Secryst
+  # Byte-level seq2seq (ByT5 family) inference over ONNX sessions from an
+  # IMF v1 model zip: sha256-verified graphs, greedy KV-cache decode when
+  # the zip ships decoder-kv.onnx (default), plain full-recompute
+  # fallback otherwise. Tokenization is the canonical ByT5 table
+  # (byte b -> id b+3, trailing EOS) — see Secryst::IMF.
+  class Byt5Onnx
+    def initialize(zip_path)
+      @manifest = IMF.manifest(zip_path)
+      graphs = IMF.verify_and_read(zip_path)
+      @tmpdir = Dir.mktmpdir('secryst-imf')
+      @encoder = build_session(graphs, 'encoder.onnx')
+      @kv = @manifest['decoder'] == 'kv' && graphs.key?('decoder-kv.onnx')
+      @decoder = build_session(graphs, @kv ? 'decoder-kv.onnx' : 'decoder.onnx')
+      @pasts = @kv ? zero_pasts : {}
+    end
+
+    def translate(text, max_seq_length: 256)
+      ids = IMF.encode(text)
+      return '' if ids.length == 1 # only the trailing EOS: empty input
+
+      hidden = @encoder.predict({ input_ids: [ids] })['last_hidden_state']
+      tokens = @kv ? greedy_kv(hidden, max_seq_length) : greedy_plain(hidden, max_seq_length)
+      IMF.decode(tokens)
+    end
+
+    def id
+      @manifest['id']
+    end
+
+    private
+
+    # The onnxruntime gem loads from paths, not bytes: verified bytes go
+    # to a private tmpdir.
+    def build_session(graphs, name)
+      path = File.join(@tmpdir, name)
+      File.binwrite(path, graphs.fetch(name))
+      OnnxRuntime::Model.new(path)
+    end
+
+    def zero_pasts
+      @decoder.inputs.each_with_object({}) do |meta, pasts|
+        next unless meta[:name].start_with?('past_')
+        shape = meta[:shape] # [batch, heads, past_seq, d_kv]; dynamic dims are Strings
+        heads = shape[1].is_a?(Integer) ? shape[1] : 4
+        d_kv = shape[3].is_a?(Integer) ? shape[3] : 8
+        # Numo carries the explicit [1, heads, 0, d_kv] shape through;
+        # nested Ruby arrays cannot express a zero-length dim.
+        pasts[meta[:name]] = Numo::DFloat.zeros(1, heads, 0, d_kv)
+      end
+    end
+
+    def argmax(row)
+      best = 0
+      best_val = -Float::INFINITY
+      row.each_with_index do |value, index|
+        if value > best_val
+          best_val = value
+          best = index
+        end
+      end
+      best
+    end
+
+    def greedy_kv(hidden, max_seq_length)
+      pasts = @pasts.dup
+      current = [IMF::PAD_ID]
+      generated = []
+      max_seq_length.times do
+        results = @decoder.predict(
+          { input_ids: [current], encoder_hidden_states: hidden }.merge(pasts)
+        )
+        token = argmax(results['logits'].first.last)
+        break if token == IMF::EOS_ID
+
+        generated << token
+        pasts = pasts.keys.to_h { |name| [name, results[name.sub('past_', 'present_')]] }
+        current = [token]
+      end
+      generated
+    end
+
+    def greedy_plain(hidden, max_seq_length)
+      decoder_ids = [IMF::PAD_ID]
+      generated = []
+      max_seq_length.times do
+        logits = @decoder.predict(
+          { input_ids: [decoder_ids], encoder_hidden_states: hidden }
+        )['logits']
+        token = argmax(logits.first.last)
+        break if token == IMF::EOS_ID
+
+        generated << token
+        decoder_ids = decoder_ids + [token]
+      end
+      generated
+    end
+  end
+end