interscript 0.1.7 → 2.1.0b1

data/README.adoc

@@ -1,298 +0,0 @@
-= Interscript: Interoperable Script Conversion Systems, with a Ruby implementation
-
-image:https://github.com/interscript/interscript/workflows/test/badge.svg["Ruby build status", link="https://github.com/interscript/interscript/actions?workflow=test"]
-image:https://github.com/interscript/interscript/workflows/js/badge.svg["JavaScript build status", link="https://github.com/interscript/interscript/actions?workflow=js"]
-
-== Introduction
-
-This repository contains interoperable transliteration schemes from:
-
-* ALA-LC
-* BGN/PCGN
-* ICAO
-* ISO
-* UN (by UNGEGN)
-* Many, many other script conversion system authorities.
-
-The goal is to achieve interoperable transliteration schemes allowing quality comparisons.
-
-
-
-== Demonstration
-
-These transliteration systems are used in the demo:
-
-`bgnpcgn-rus-Cyrl-Latn-1947`:: BGN/PCGN Romanization of Russian
-`iso-rus-Cyrl-Latn-9-1995`::     ISO 9 Romanization of Russian
-`icao-rus-Cyrl-Latn-9303`::    ICAO MRZ Romanization of Russian
-`bas-rus-Cyrl-Latn-2017-bss`::      Bulgaria Academy of Science Streamlined System for Russian
-
-image:demo/20191118-interscript-demo-cast.gif["interscript screencast"]
-
-
-== Installation
-
-=== Prerequisites
-
-Linux:
-
-[source,sh]
-----
-apt-get install swig python3-setuptools
-----
-
-Windows:
-
-[source,sh]
-----
-choco install --no-progress swig
-----
-
-Interscript depends on Python and the https://github.com/sequitur-g2p/sequitur-g2p[`sequitur-g2p`] module
-
-[source,sh]
-----
-pip3 install setuptools numpy
-curl -sSL -o sequitur-g2p.zip https://github.com/sequitur-g2p/sequitur-g2p/archive/806273f.zip
-pip3 install sequitur-g2p.zip
-----
-
-Interscript depends on Ruby. Once you manage to install Ruby, it's easy.
-
-[source,sh]
-----
-gem install interscript
-----
-
-== Usage
-
-Assume you have a file ready in the source script like this:
-
-[source,sh]
-----
-cat <<EOT > rus-Cyrl.txt
-Эх, тройка! птица тройка, кто тебя выдумал? знать, у бойкого народа ты
-могла только родиться, в той земле, что не любит шутить, а
-ровнем-гладнем разметнулась на полсвета, да и ступай считать версты,
-пока не зарябит тебе в очи. И не хитрый, кажись, дорожный снаряд, не
-железным схвачен винтом, а наскоро живьём с одним топором да долотом
-снарядил и собрал тебя ярославский расторопный мужик. Не в немецких
-ботфортах ямщик: борода да рукавицы, и сидит чёрт знает на чём; а
-привстал, да замахнулся, да затянул песню — кони вихрем, спицы в
-колесах смешались в один гладкий круг, только дрогнула дорога, да
-вскрикнул в испуге остановившийся пешеход — и вон она понеслась,
-понеслась, понеслась!
-
-Н.В. Гоголь
-EOT
-----
-
-You can run `interscript` on this text using different transliteration systems.
-
-[source,sh]
-----
-interscript rus-Cyrl.txt \
-  --system=bgnpcgn-rus-Cyrl-Latn-1947 \
-  --output=bgnpcgn-rus-Latn.txt
-
-interscript rus-Cyrl.txt \
-  --system=iso-rus-Cyrl-Latn-9-1995 \
-  --output=iso-rus-Latn.txt
-
-interscript rus-Cyrl.txt \
-  --system=icao-rus-Cyrl-Latn-9303 \
-  --output=icao-rus-Latn.txt
-
-interscript rus-Cyrl.txt \
-  --system=bas-rus-Cyrl-Latn-2017-bss \
-  --output=bas-rus-Latn.txt
-----
-
-It is then easy to see the exact differences in rendering between the systems.
-
-[source,sh]
-----
-diff bgnpcgn-rus-Latn.txt bas-rus-Latn.txt
-----
-
-== Adding transliteration system
-
-Transliteration systems stored in a `maps/` directory as YAML files.
-You can create a new file and add it to the directory.
-
-The file should be named as `<system-code>.yaml`, where `system-code`
-is in accordance with
-http://calconnect.gitlab.io/tc-localization/csd-transcription-systems[ISO/CC 24229].
-
-=== File structure
-
-[source,yaml]
-----
-authority_id: bgnpcgn
-id: 1947
-language: rus
-source_script: Cyrl
-destination_script: Latn
-name: ROMANIZATION OF RUSSIAN, BGN/PCGN 1947 System
-url: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/807920/ROMANIZATION_OF_RUSSIAN.pdf
-creation_date: 1947
-confirmation_date: 2019-06
-description: The BGN/PCGN system for Russian was adopted ...
-
-notes:
-  - The character e should be romanized ye initially, after the vowel ...
-
-tests:
-  - source: ДЛИННОЕ ПОКРЫВАЛО
-    expected: DLINNOYE POKRYVALO
-  - source: Еловая шишка
-    expected: Yelovaya shishka
-
-map:
-  rules:
-    - pattern: (?<=[АаЕеЁёИиОоУуЫыЭэЮюЯяЙйЪъЬь])\u0415 # Е after a, e, ё, и, о, у, ы, э, ю, я, й, ъ, ь
-      result: Ye
-    - pattern: \b\u0415 # Е initially
-      result: Ye
-
-  characters:
-    "\u0410": "A"
-    "\u0411": "B"
-    "\u0412": "V"
-----
-
-
-=== Rules
-
-The subsection `rules` is placed under the `map` key. All rules are applied in order they are placed before the subsection `characters` applying. Rules apply to an original text, not to a result of previous rules applying.
-
-Each rule has `pattern` and `result` elements.
-
-Pattern is a regex expression. It should be representing as a string without `//` or `%r{}` parentheses. For example `\b\u0415`. In case a rule is depend on previous or next content, lookahead or lookbehind could be used. For example a rule with the pattern `(?<=[АаЕеЁёИиОоУуЫыЭэЮюЯяЙйЪъЬь])\u0415` find every Е after upper or lower case symbols a, e, ё, и, о, у, ы, э, ю, я, й, ъ, ь.
-
-Result is a replacement a for pattern's match. It can contain a string, an Unicode characters specified by a hexadecimal number, a captured group reference. String with hexadecimal number or captured group reference should be double quoted. For example `"Y\u00eb"` or `"\\1\u00b7\\2"`. Captured group are referred by double backslash and group's number.
-
-Because rules are applied in order, multiple rules applicable to the same segment of a string can be addressed by rule ordering, and rules can be used as priority over characters. For example:
-
-[source,yaml]
-----
-map:
-  rules:
-    - pattern: \u03B3\u03B3    # γ (before Γ, Ξ, Χ)
-      result: ng
-    - pattern: (?<![Γγ])\u03B3(?=[ΕεέΗηήΙιίΥυύ])    # γ (before front vowels)
-      result: y
-----
-
-(γι maps to `yi`; but γγ maps to `ng`. In the case of γγι, the first rule takes priority, and the transliteration is `ngi`: it makes the second rule impossible.)
-
-[source,yaml]
-----
-map:
-  rules:
-    - pattern: (?<=\b)\u03BC[πΠ]  # μπ (initially)
-      result: b
-    - pattern: \u03BC[πΠ]         # μπ (medially)
-      result: mb
-----
-
-(The first rule applies at the start of a word; the second rule does not specify a context, as it applies in all other cases not covered by the first rule.)
-
-[source,yaml]
-----
-map:
-  rules:
-    - pattern: ";"
-      result: "?"
-
-  characters
-    "\u00B7": ";
-----
-
-(This guarantees that any `;` are converted to `?` before any new `;` are introduced; because all three are Latin script, they could be mixed up in ordering.)
-
-Normally rules "`bleed`" each other: once a rule applies to a segment, that segment cannot trigger other rules, because it is already converted to Roman. Exceptionally, it will be necessary to have a rule add or remove characters in the original script, rather than transliterate them, so that the same context can be invoked by two rules in succession:
-
-[source,yaml]
-----
-map:
-  rules:
-    - pattern: (?<=[АаЕеЁёИиОоУуЫыЭэЮюЯя])\u042b # Ы after any vowel character
-      result: "\u00b7Ы"
-    - pattern: \u042b(?=[АаУуЫыЭэ])              # Ы before а, у, ы, or э
-      result: "Ы\u00b7"
-----
-
-(If the result were `\u00B7Y`, the second rule could not be applied afterwards; but we want ОЫУ to transliterate as `O·Y·U`. In order to make that happen, we preserve the Ы during the rules phase, resulting in О·Ы·У; we only convert the letters to Roman script in the `characters` phase.)
-
-=== Testing transliteration systems
-
-To test all transliteration systems in the `maps/` directory, run:
-
-[source,sh]
-----
-bundle exec rspec
-----
-
-The command takes `source` texts from the `test` section, transforms
-them using `rules` and `charmaps` from the `map` key, and compares the
-results with `expected:` text from the `source:` section.
-
-To test a specific transliteration system, set the environment variable
-`TRANSLIT_SYSTEM` to the system code of the desired system
-(i.e. the "`basename`" of the system's YAML file):
-
-[source,sh]
-----
-TRANSLIT_SYSTEM=bgnpcgn-rus-Cyrl-Latn-1947 bundle exec rspec
-----
-
-
-== ISCS system codes
-
-In accordance with
-http://calconnect.gitlab.io/tc-localization/csd-transcription-systems[ISO/CC 24229],
-the system code identifying a script conversion system has the following components:
-
-e.g. `bgnpcgn-rus-Cyrl-Latn-1947`:
-
-`bgnpcgn`:: the authority identifier
-`rus`:: an ISO 639-{1,2,3,5} language code that this system applies to (For 639-2, use (T) code)
-`Cyrl`:: an ISO 15924 script code, identifying the source script
-`Latn`:: an ISO 15924 script code, identifying the target script
-`1947`:: an identifier unit within the authority to identify this system
-
-
-== Covered languages
-
-Currently the schemes cover Cyrillic, Armenian, Greek, Arabic and Hebrew.
-
-
-== Samples to play with
-
-* `rus-Cyrl-1.txt`: Copied from the XLS output from http://www.primorsk.vybory.izbirkom.ru/region/primorsk?action=show&global=true&root=254017025&tvd=4254017212287&vrn=100100067795849&prver=0&pronetvd=0&region=25&sub_region=25&type=242&vibid=4254017212287
-
-* `rus-Cyrl-2.txt`: Copied from the XLS output from http://www.yaroslavl.vybory.izbirkom.ru/region/yaroslavl?action=show&root=764013001&tvd=4764013188704&vrn=4764013188693&prver=0&pronetvd=0&region=76&sub_region=76&type=426&vibid=4764013188704
-
-
-== References
-
-Reference documents are located at the
-https://github.com/interscript/interscript-references[interscript-references repository].
-Some specifications that have distribution limitations may not be reproduced there.
-
-
-== Links to system definitions
-
-* https://www.iso.org/committee/48750.html[ISO/TC 46 (see standards published by WG 3)]
-* http://geonames.nga.mil/gns/html/romanization.html[BGN/PCGN and BGN Romanization systems (BGN)]
-* https://www.gov.uk/government/publications/romanization-systems[BGN/PCGN Romanization systems (PCGN)]
-* https://www.loc.gov/catdir/cpso/roman.html[ALA-LC Romanization systems in current use]
-* http://catdir.loc.gov/catdir/cpso/roman.html[ALA-LC Romanization systems from 1997]
-* http://www.eki.ee/wgrs/[UN Romanization systems]
-* http://www.eki.ee/knab/kblatyl2.htm[EKI KNAB systems]
-
-== Copyright and license
-
-This is a Ribose project. Copyright Ribose.
-

data/lib/g2pwrapper.py

@@ -1,34 +0,0 @@
-import g2p, SequiturTool
-import numpy
-
-def transliterate(model, word):
-
-  class Struct:
-      def __init__(self, **entries):
-          self.__dict__.update(entries)
-
-  model_path = {
-    'pythainlp_lexicon': './lib/model-7', 
-    'wiktionary_phonemic': './lib/tha-pt-b-7'
-  }
-
-  connector_dict = {
-    'pythainlp_lexicon': '', 
-    'wiktionary_phonemic': '-'
-  }
-
-
-  modelFile = model_path[model]
-  connector = connector_dict[model]
-
-  options = Struct(**{'profile': None, 'resource_usage': None, 'psyco': None, 'tempdir': None, 'trainSample': None, 'develSample': None, 'testSample': None, 'checkpoint': None, 'resume_from_checkpoint': None, 'shouldTranspose': None, 'modelFile': modelFile , 'newModelFile': None, 'shouldTestContinuously': None, 'shouldSelfTest': None, 'lengthConstraints': None, 'shouldSuppressNewMultigrams': None, 'viterbi': None, 'shouldRampUp': None, 'shouldWipeModel': None, 'shouldInitializeWithCounts': None, 'minIterations': 20, 'maxIterations': 100, 'eager_discount_adjustment': None, 'fixed_discount': None, 'encoding': 'UTF-8', 'phoneme_to_phoneme': None, 'test_segmental': None, 'testResult': None, 'applySample': None, 'applyWord': word, 'variants_mass': None, 'variants_number': None, 'fakeTranslator': None, 'stack_limit': None})
-
-  loadSample = g2p.loadG2PSample
-
-  model = SequiturTool.procureModel(options, loadSample)
-  if not model:
-      return 1
-  translator = g2p.Translator(model)
-  del model
-
-  return connector.join(translator(tuple(word)))

data/lib/interscript/fs.rb

@@ -1,69 +0,0 @@
-require 'pathname'
-
-module Interscript
-  module Fs
-    def sub_replace(string, pos, size, repl)
-      string[pos..pos + size - 1] = repl
-      string
-    end
-
-    def root_path
-      @root_path ||= Pathname.new(File.join(File.dirname(__dir__), ".."))
-    end
-
-    def transliterate_file(system_code, input_file, output_file, maps={})
-      input = File.read(input_file)
-      output = transliterate(system_code, input, maps)
-
-      File.open(output_file, 'w') do |f|
-        f.puts(output)
-      end
-
-      puts "Output written to: #{output_file}"
-      output_file
-    end
-
-    def import_python_modules
-      begin
-        pyimport :g2pwrapper
-      rescue
-        pyimport :sys
-        sys.path.append(root_path.to_s + "/lib/")
-        pyimport :g2pwrapper
-      end
-    end
-
-    def external_process(process_name, string)
-      import_python_modules
-
-      case process_name
-      when 'sequitur.pythainlp_lexicon'
-        return g2pwrapper.transliterate('pythainlp_lexicon', string)
-      when 'sequitur.wiktionary_phonemic'
-        return g2pwrapper.transliterate('wiktionary_phonemic', string)
-      else
-        raise ExternalProcessNotRecognizedError.new
-      end
-
-    rescue
-      raise ExternalProcessUnavailableError.new
-    end
-
-    def external_processing(mapping, string)
-      # Segmentation
-      string = external_process(mapping.segmentation, string) if mapping.segmentation
-
-      # Transliteration/Transcription
-      string = external_process(mapping.transcription, string) if mapping.transcription
-
-      string
-    end
-
-    private
-
-    def mkregexp(regexpstring)
-      /#{regexpstring}/u
-    end
-
-  end
-end

data/lib/interscript/mapping.rb

@@ -1,142 +0,0 @@
-require 'rambling-trie'
-require 'yaml' unless RUBY_ENGINE == 'opal'
-require 'json'
-
-module Interscript
-
-  class Mapping
-    attr_reader(
-      :id,
-      :url,
-      :name,
-      :notes,
-      :rules,
-      :tests,
-      :language,
-      :postrules,
-      :characters,
-      :description,
-      :authority_id,
-      :creation_date,
-      :source_script,
-      :destination_script,
-      :chain,
-      :character_separator,
-      :word_separator,
-      :title_case,
-      :downcase,
-      :dictionary,
-      :characters_hash,
-      :dictionary_hash,
-      :segmentation,
-      :transcription,
-      :dictionary_trie
-    )
-
-    def initialize(system_code, options = {})
-      @system_code = system_code
-      @depth = options.fetch(:depth, 0).to_i
-
-      unless RUBY_ENGINE == 'opal'
-        @system_path = options.fetch(:system_code, default_path)
-      end
-
-      load_and_serialize_system_mappings
-    end
-
-    def self.for(system_code, options = {})
-      new(system_code, options)
-    end
-
-    def load_and_serialize_system_mappings
-      return if depth >= 5
-
-      mappings = load_system_mappings
-      serialize_system_mappings(mappings)
-    end
-
-    private
-
-    attr_reader :depth, :system_code, :system_path
-
-    def system_code_file
-      [system_code, "yaml"].join(".")
-    end
-
-    def default_path
-      @default_path ||= Interscript.root_path.join("maps")
-    end
-
-    def load_system_mappings
-      if RUBY_ENGINE == 'opal'
-        load_opal_mappings
-      else
-        load_fs_mappings
-      end
-    end
-
-    def load_opal_mappings
-      JSON.parse(`InterscriptMaps[#{system_code}]`)
-    end
-
-    def load_fs_mappings
-      YAML.load_file(system_path.join(system_code_file))
-    rescue Errno::ENOENT
-      raise Interscript::InvalidSystemError.new("No system mappings found")
-    end
-
-    def serialize_system_mappings(mappings)
-      @id = mappings.fetch("id", nil)
-      @url = mappings.fetch("url", nil)
-      @name = mappings.fetch("name", nil)
-      @notes = mappings.fetch("notes", nil)
-      @tests = mappings.fetch("tests", [])
-      @language = mappings.fetch("language", nil)
-      @description = mappings.fetch("description", nil)
-      @authority_id = mappings.fetch("authority_id", nil)
-      @creation_date = mappings.fetch("creation_date", nil)
-      @source_script = mappings.fetch("source_script", nil)
-      @destination_script = mappings.fetch("destination_script", nil)
-      @chain = mappings.fetch("chain", [])
-      @character_separator = mappings["map"]["character_separator"] || nil
-      @word_separator = mappings["map"]["word_separator"] || nil
-      @title_case = mappings["map"]["title_case"] || false
-      @downcase = mappings["map"]["downcase"] || false
-      @rules = mappings["map"]["rules"] || []
-      @postrules = mappings["map"]["postrules"] || []
-      @characters = mappings["map"]["characters"] || {}
-      @dictionary = mappings["map"]["dictionary"] || {}
-      @segmentation = mappings["map"]["segementation"] || nil
-      @transcription = mappings["map"]["transcription"] || nil
-
-      include_inherited_mappings(mappings)
-      build_hashes
-      build_trie
-    end
-
-    def include_inherited_mappings(mappings)
-      inherit_systems = [].push(mappings["map"]["inherit"]).flatten
-
-      inherit_systems.each do |inherit_system|
-        next unless inherit_system
-
-        inherited_mapping = Mapping.for(inherit_system, depth: depth + 1)
-
-        @rules = [rules, inherited_mapping.rules].flatten
-        @postrules = [inherited_mapping.postrules, postrules].flatten
-        @characters = (inherited_mapping.characters|| {}).merge(characters)
-        @dictionary = (inherited_mapping.dictionary|| {}).merge(dictionary)
-      end
-    end
-
-    def build_hashes
-      @characters_hash = characters&.sort_by { |k, _v| k.size }&.reverse&.to_h
-      @dictionary_hash = dictionary&.sort_by { |k, _v| k.size }&.reverse&.to_h
-    end
-
-    def build_trie
-      @dictionary_trie = Rambling::Trie.create
-      dictionary_trie.concat dictionary.keys
-    end
-  end
-end