interscript 0.1.0 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 19905541367b50886e3d130aac4132cbb5e9050bff1409f924b23848ea5b3a32
-  data.tar.gz: d9d6a9531009778797f2826cf51dfc2b68b27651608be5c91ed7f343d75ff75f
+  metadata.gz: 3eb5c858ad7332995b5f8ed1c7766305f6be5fb5524e91de7df1d0d3d770a3ee
+  data.tar.gz: 80be952635dd65cf1e23107f70c44557ec9711590759e610a042f2c746f67d4e
 SHA512:
-  metadata.gz: 247a535a7fd1203b2331b2c6c99fa9a9e0747c3fc5ff1f5476c034d1c72ae5bd16bbf2866cf639cadd321989dd653a461eda2b0db662db303c381b2de8b18285
-  data.tar.gz: a9c259539a46c33dc5c7de38edc9b5d330727064509dbc6cc17cdfa3baf1ae950fa3e8c645e45e9e04fa3afc213f50f3e1cfe8c44470bf8024e93f3621e32b87
+  metadata.gz: bdb63b2ea8e6337a31879e7c1331f6e9a95dd205580bc8656cdb806e624b7cf946c17d300125470494c1643332b3f5f1a3d713a040bd6ee907dcba5d189db95e
+  data.tar.gz: 1aa1b254b253f148b4a5cc316027d3e67ca1ba68e866ebedbc025dc9c5f5c78680b12c24bc52177d65b2c9a6dd262ee97e34f52b4d32f83dfc9c232dc2c2d0de

data/README.adoc

@@ -1,48 +1,263 @@
-= Interscript: Interoperable Script Conversion Systems and a Ruby implementation
+= Interscript: Interoperable Script Conversion Systems, with a Ruby implementation
 
-== Introducation
+image:https://github.com/interscript/interscript/workflows/test/badge.svg["Ruby build status", link="https://github.com/interscript/interscript/actions?workflow=test"]
+image:https://github.com/interscript/interscript/workflows/js/badge.svg["JavaScript build status", link="https://github.com/interscript/interscript/actions?workflow=js"]
 
-This repository contains a number of transliteration schemes from:
+== Introduction
 
+This repository contains interoperable transliteration schemes from:
+
+* ALA-LC
 * BGN/PCGN
 * ICAO
 * ISO
 * UN (by UNGEGN)
+* Many, many other script conversion system authorities.
 
 The goal is to achieve interoperable transliteration schemes allowing quality comparisons.
 
 
-== STATUS (work in progress!)
 
-These transliteration systems currently work:
+== Demonstration
+
+These transliteration systems are used in the demo:
 
 `bgnpcgn-rus-Cyrl-Latn-1947`:: BGN/PCGN Romanization of Russian
-`iso-rus-Cyrl-Latn-iso9`::     ISO 9 Romanization of Russian
+`iso-rus-Cyrl-Latn-9-1995`::     ISO 9 Romanization of Russian
 `icao-rus-Cyrl-Latn-9303`::    ICAO MRZ Romanization of Russian
-`bas-rus-Cyrl-Latn-bss`::      Bulgaria Academy of Science Streamlined System for Russian
+`bas-rus-Cyrl-Latn-2017-bss`::      Bulgaria Academy of Science Streamlined System for Russian
+
+image:demo/20191118-interscript-demo-cast.gif["interscript screencast"]
+
+
+== Installation
+
+=== Prerequisites
+
+Linux:
+
+[source,sh]
+----
+apt-get install swig python3-setuptools
+----
+
+Windows:
 
+[source,sh]
+----
+choco install --no-progress swig
+----
+
+Interscript depends on Python and the https://github.com/sequitur-g2p/sequitur-g2p[`sequitur-g2p`] module
+
+[source,sh]
+----
+pip3 install setuptools numpy
+curl -sSL -o sequitur-g2p.zip https://github.com/sequitur-g2p/sequitur-g2p/archive/806273f.zip
+pip3 install sequitur-g2p.zip
+----
+
+Interscript depends on Ruby. Once you manage to install Ruby, it's easy.
+
+[source,sh]
+----
+gem install interscript
+----
 
 == Usage
 
+Assume you have a file ready in the source script like this:
+
+[source,sh]
+----
+cat <<EOT > rus-Cyrl.txt
+Эх, тройка! птица тройка, кто тебя выдумал? знать, у бойкого народа ты
+могла только родиться, в той земле, что не любит шутить, а
+ровнем-гладнем разметнулась на полсвета, да и ступай считать версты,
+пока не зарябит тебе в очи. И не хитрый, кажись, дорожный снаряд, не
+железным схвачен винтом, а наскоро живьём с одним топором да долотом
+снарядил и собрал тебя ярославский расторопный мужик. Не в немецких
+ботфортах ямщик: борода да рукавицы, и сидит чёрт знает на чём; а
+привстал, да замахнулся, да затянул песню — кони вихрем, спицы в
+колесах смешались в один гладкий круг, только дрогнула дорога, да
+вскрикнул в испуге остановившийся пешеход — и вон она понеслась,
+понеслась, понеслась!
+
+Н.В. Гоголь
+EOT
+----
+
+You can run `interscript` on this text using different transliteration systems.
+
+[source,sh]
+----
+interscript rus-Cyrl.txt \
+  --system=bgnpcgn-rus-Cyrl-Latn-1947 \
+  --output=bgnpcgn-rus-Latn.txt
+
+interscript rus-Cyrl.txt \
+  --system=iso-rus-Cyrl-Latn-9-1995 \
+  --output=iso-rus-Latn.txt
+
+interscript rus-Cyrl.txt \
+  --system=icao-rus-Cyrl-Latn-9303 \
+  --output=icao-rus-Latn.txt
+
+interscript rus-Cyrl.txt \
+  --system=bas-rus-Cyrl-Latn-2017-bss \
+  --output=bas-rus-Latn.txt
+----
+
+It is then easy to see the exact differences in rendering between the systems.
+
+[source,sh]
+----
+diff bgnpcgn-rus-Latn.txt bas-rus-Latn.txt
+----
+
+== Adding transliteration system
+
+Transliteration systems stored in a `maps/` directory as YAML files.
+You can create a new file and add it to the directory.
+
+The file should be named as `<system-code>.yaml`, where `system-code`
+is in accordance with
+http://calconnect.gitlab.io/tc-localization/csd-transcription-systems[ISO/CC 24229].
+
+=== File structure
+
+[source,yaml]
+----
+authority_id: bgnpcgn
+id: 1947
+language: rus
+source_script: Cyrl
+destination_script: Latn
+name: ROMANIZATION OF RUSSIAN, BGN/PCGN 1947 System
+url: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/807920/ROMANIZATION_OF_RUSSIAN.pdf
+creation_date: 1947
+confirmation_date: 2019-06
+description: The BGN/PCGN system for Russian was adopted ...
+
+notes:
+  - The character e should be romanized ye initially, after the vowel ...
+
+tests:
+  - source: ДЛИННОЕ ПОКРЫВАЛО
+    expected: DLINNOYE POKRYVALO
+  - source: Еловая шишка
+    expected: Yelovaya shishka
+
+map:
+  rules:
+    - pattern: (?<=[АаЕеЁёИиОоУуЫыЭэЮюЯяЙйЪъЬь])\u0415 # Е after a, e, ё, и, о, у, ы, э, ю, я, й, ъ, ь
+      result: Ye
+    - pattern: \b\u0415 # Е initially
+      result: Ye
+
+  characters:
+    "\u0410": "A"
+    "\u0411": "B"
+    "\u0412": "V"
+----
+
+
+=== Rules
+
+The subsection `rules` is placed under the `map` key. All rules are applied in order they are placed before the subsection `characters` applying. Rules apply to an original text, not to a result of previous rules applying.
+
+Each rule has `pattern` and `result` elements.
+
+Pattern is a regex expression. It should be representing as a string without `//` or `%r{}` parentheses. For example `\b\u0415`. In case a rule is depend on previous or next content, lookahead or lookbehind could be used. For example a rule with the pattern `(?<=[АаЕеЁёИиОоУуЫыЭэЮюЯяЙйЪъЬь])\u0415` find every Е after upper or lower case symbols a, e, ё, и, о, у, ы, э, ю, я, й, ъ, ь.
+
+Result is a replacement a for pattern's match. It can contain a string, an Unicode characters specified by a hexadecimal number, a captured group reference. String with hexadecimal number or captured group reference should be double quoted. For example `"Y\u00eb"` or `"\\1\u00b7\\2"`. Captured group are referred by double backslash and group's number.
+
+Because rules are applied in order, multiple rules applicable to the same segment of a string can be addressed by rule ordering, and rules can be used as priority over characters. For example:
+
+[source,yaml]
+----
+map:
+  rules:
+    - pattern: \u03B3\u03B3    # γ (before Γ, Ξ, Χ)
+      result: ng
+    - pattern: (?<![Γγ])\u03B3(?=[ΕεέΗηήΙιίΥυύ])    # γ (before front vowels)
+      result: y
+----
+
+(γι maps to `yi`; but γγ maps to `ng`. In the case of γγι, the first rule takes priority, and the transliteration is `ngi`: it makes the second rule impossible.)
+
+[source,yaml]
+----
+map:
+  rules:
+    - pattern: (?<=\b)\u03BC[πΠ]  # μπ (initially)
+      result: b
+    - pattern: \u03BC[πΠ]         # μπ (medially)
+      result: mb
+----
+
+(The first rule applies at the start of a word; the second rule does not specify a context, as it applies in all other cases not covered by the first rule.)
+
+[source,yaml]
+----
+map:
+  rules:
+    - pattern: ";"
+      result: "?"
+
+  characters
+    "\u00B7": ";
+----
+
+(This guarantees that any `;` are converted to `?` before any new `;` are introduced; because all three are Latin script, they could be mixed up in ordering.)
+
+Normally rules "`bleed`" each other: once a rule applies to a segment, that segment cannot trigger other rules, because it is already converted to Roman. Exceptionally, it will be necessary to have a rule add or remove characters in the original script, rather than transliterate them, so that the same context can be invoked by two rules in succession:
+
+[source,yaml]
+----
+map:
+  rules:
+    - pattern: (?<=[АаЕеЁёИиОоУуЫыЭэЮюЯя])\u042b # Ы after any vowel character
+      result: "\u00b7Ы"
+    - pattern: \u042b(?=[АаУуЫыЭэ])              # Ы before а, у, ы, or э
+      result: "Ы\u00b7"
+----
+
+(If the result were `\u00B7Y`, the second rule could not be applied afterwards; but we want ОЫУ to transliterate as `O·Y·U`. In order to make that happen, we preserve the Ы during the rules phase, resulting in О·Ы·У; we only convert the letters to Roman script in the `characters` phase.)
+
+=== Testing transliteration systems
+
+To test all transliteration systems in the `maps/` directory, run:
 
 [source,sh]
 ----
-# Transliterating Russian Cyrillic to Latin using the Streamlined System for Russian
-interscript samples/rus-Cyrl.txt --system=bas-rus-Cyrl-Latn-bss --output=rus-Latn.txt
+bundle exec rspec
+----
+
+The command takes `source` texts from the `test` section, transforms
+them using `rules` and `charmaps` from the `map` key, and compares the
+results with `expected:` text from the `source:` section.
+
+To test a specific transliteration system, set the environment variable
+`TRANSLIT_SYSTEM` to the system code of the desired system
+(i.e. the "`basename`" of the system's YAML file):
 
-# Transliterating Russian Cyrillic to Latin using the BGN/PCGN Romanization of Russian
-interscript samples/rus-Cyrl.txt --system=bgnpcgn-rus-Cyrl-Latn-1947 --output=rus-Latn.txt
+[source,sh]
+----
+TRANSLIT_SYSTEM=bgnpcgn-rus-Cyrl-Latn-1947 bundle exec rspec
 ----
 
 
 == ISCS system codes
 
-The system code identifying a script conversion system has a few components:
+In accordance with
+http://calconnect.gitlab.io/tc-localization/csd-transcription-systems[ISO/CC 24229],
+the system code identifying a script conversion system has the following components:
 
-e.g. `bgnpcgn-rus-Cyrl-Latn-1947`
+e.g. `bgnpcgn-rus-Cyrl-Latn-1947`:
 
 `bgnpcgn`:: the authority identifier
-`rus`:: an ISO 639-2 3-letter language code that this system applies to
+`rus`:: an ISO 639-{1,2,3,5} language code that this system applies to (For 639-2, use (T) code)
 `Cyrl`:: an ISO 15924 script code, identifying the source script
 `Latn`:: an ISO 15924 script code, identifying the target script
 `1947`:: an identifier unit within the authority to identify this system
@@ -53,13 +268,31 @@ e.g. `bgnpcgn-rus-Cyrl-Latn-1947`
 Currently the schemes cover Cyrillic, Armenian, Greek, Arabic and Hebrew.
 
 
-== Sources
+== Samples to play with
 
 * `rus-Cyrl-1.txt`: Copied from the XLS output from http://www.primorsk.vybory.izbirkom.ru/region/primorsk?action=show&global=true&root=254017025&tvd=4254017212287&vrn=100100067795849&prver=0&pronetvd=0&region=25&sub_region=25&type=242&vibid=4254017212287
 
 * `rus-Cyrl-2.txt`: Copied from the XLS output from http://www.yaroslavl.vybory.izbirkom.ru/region/yaroslavl?action=show&root=764013001&tvd=4764013188704&vrn=4764013188693&prver=0&pronetvd=0&region=76&sub_region=76&type=426&vibid=4764013188704
 
 
-== Credits
+== References
+
+Reference documents are located at the
+https://github.com/interscript/interscript-references[interscript-references repository].
+Some specifications that have distribution limitations may not be reproduced there.
+
+
+== Links to system definitions
+
+* https://www.iso.org/committee/48750.html[ISO/TC 46 (see standards published by WG 3)]
+* http://geonames.nga.mil/gns/html/romanization.html[BGN/PCGN and BGN Romanization systems (BGN)]
+* https://www.gov.uk/government/publications/romanization-systems[BGN/PCGN Romanization systems (PCGN)]
+* https://www.loc.gov/catdir/cpso/roman.html[ALA-LC Romanization systems in current use]
+* http://catdir.loc.gov/catdir/cpso/roman.html[ALA-LC Romanization systems from 1997]
+* http://www.eki.ee/wgrs/[UN Romanization systems]
+* http://www.eki.ee/knab/kblatyl2.htm[EKI KNAB systems]
+
+== Copyright and license
+
+This is a Ribose project. Copyright Ribose.
 
-This is a Ribose project.

data/bin/interscript

@@ -1,22 +1,41 @@
 #!/usr/bin/env ruby
+
 require 'rubygems'
-require_relative '../lib/interscript'
-
-if ARGV.empty?
-  puts "write source file, source format, and output file"
-else
-  args = Hash[ ARGV.flat_map{|s| s.scan(/--?([^=\s]+)(?:=(\S+))?/) } ]
-  input = ARGV[0]
-  system_code = args["system"]
-  output_file = args["output"]
-
-  raise "Please enter the system code with --system={system_code}" unless system_code
-
-  if output_file
-    Interscript.instance.transliterate_file(system_code, input, output_file)
-  else
-    puts Interscript.instance.transliterate(system_code, IO.read(input))
-  end
+
+# resolve bin path, ignoring symlinks
+require 'pathname'
+bin_file = Pathname.new(__FILE__).realpath
+
+# add self to libpath
+$LOAD_PATH.unshift File.expand_path("../../lib", bin_file)
+
+# Fixes https://github.com/rubygems/rubygems/issues/1420
+require 'rubygems/specification'
+
+class Gem::Specification
+  def this; self; end
 end
 
+require 'interscript/command'
+
+if ARGV.any? && !Interscript::Command.all_tasks.key?(ARGV.first)
+  ARGV.unshift :translit
+end
+Interscript::Command.start ARGV
+
+# if ARGV.empty?
+#   puts "write source file, source format, and output file"
+# else
+#   args = Hash[ARGV.flat_map { |s| s.scan(/--?([^=\s]+)(?:=(\S+))?/) }]
+#   input = ARGV[0]
+#   system_code = args["system"]
+#   output_file = args["output"]
+
+#   raise "Please enter the system code with --system={system_code}" unless system_code
 
+#   if output_file
+#     Interscript.transliterate_file(system_code, input, output_file)
+#   else
+#     puts Interscript.transliterate(system_code, IO.read(input))
+#   end
+# end

data/bin/rspec

@@ -0,0 +1,29 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+#
+# This file was generated by Bundler.
+#
+# The application 'rspec' is installed as part of a gem, and
+# this file is here to facilitate running it.
+#
+
+require "pathname"
+ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile",
+  Pathname.new(__FILE__).realpath)
+
+bundle_binstub = File.expand_path("../bundle", __FILE__)
+
+if File.file?(bundle_binstub)
+  if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/
+    load(bundle_binstub)
+  else
+    abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run.
+Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.")
+  end
+end
+
+require "rubygems"
+require "bundler/setup"
+
+load Gem.bin_path("rspec-core", "rspec")

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/g2pwrapper.py

@@ -0,0 +1,34 @@
+import g2p, SequiturTool
+import numpy
+
+def transliterate(model, word):
+
+  class Struct:
+      def __init__(self, **entries):
+          self.__dict__.update(entries)
+
+  model_path = {
+    'pythainlp_lexicon': './lib/model-7', 
+    'wiktionary_phonemic': './lib/tha-pt-b-7'
+  }
+
+  connector_dict = {
+    'pythainlp_lexicon': '', 
+    'wiktionary_phonemic': '-'
+  }
+
+
+  modelFile = model_path[model]
+  connector = connector_dict[model]
+
+  options = Struct(**{'profile': None, 'resource_usage': None, 'psyco': None, 'tempdir': None, 'trainSample': None, 'develSample': None, 'testSample': None, 'checkpoint': None, 'resume_from_checkpoint': None, 'shouldTranspose': None, 'modelFile': modelFile , 'newModelFile': None, 'shouldTestContinuously': None, 'shouldSelfTest': None, 'lengthConstraints': None, 'shouldSuppressNewMultigrams': None, 'viterbi': None, 'shouldRampUp': None, 'shouldWipeModel': None, 'shouldInitializeWithCounts': None, 'minIterations': 20, 'maxIterations': 100, 'eager_discount_adjustment': None, 'fixed_discount': None, 'encoding': 'UTF-8', 'phoneme_to_phoneme': None, 'test_segmental': None, 'testResult': None, 'applySample': None, 'applyWord': word, 'variants_mass': None, 'variants_number': None, 'fakeTranslator': None, 'stack_limit': None})
+
+  loadSample = g2p.loadG2PSample
+
+  model = SequiturTool.procureModel(options, loadSample)
+  if not model:
+      return 1
+  translator = g2p.Translator(model)
+  del model
+
+  return connector.join(translator(tuple(word)))