mongolian 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: b5ae047f3b8d5d37be63b5646bae637537a46f1a8402d9cb6bed1d95ea9f3135
+  data.tar.gz: 7e00183b681aa0a29119a5d3eadcb687546761bd3b9c78f8dc670f1dc869e7de
+SHA512:
+  metadata.gz: b075ed85bfd720da5169e92b2e5c5ce6ce0a279ee7fb5fa8b137137bc6bcb6debd257efdac73751f63e1ccbb28bff533f2e764d71c427ae36c099173c595ca4e
+  data.tar.gz: 9ed88cc1be818a65e58a134dafca486faa1eeeed3325c460a86c3aac60990ebcd18848802164afd27e3ac79b2c611e73e14470ea2547f38521ffca60634f6042

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## [Unreleased]
+
+## [0.1.0] - 2025-06-11
+
+- Initial release

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,132 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 hfl
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,42 @@
+# Mongolian
+
+欢迎使用 Mongolian Gem！本 Gem 打包了一些蒙古文处理常用的操作，例如分词、蒙古文转换为拉丁转写等。
+
+## 安装
+
+安装 Gem 并添加到应用的 Gemfile 可以通过执行：
+
+```bash
+bundle add mongolian
+```
+
+如果没有使用 bundler 管理依赖，可以通过：
+
+```bash
+gem install mongolian
+```
+直接安装。
+
+## 用法
+
+1. 判断一个字符或者词是否包含蒙古文字符；
+2. 蒙古文分词；
+3. 蒙古文词干提取；
+
+## 开发
+
+查看代码库，运行 `bin/setup` 安装依赖库。运行 `rake test` 进行测试。运行 `bin/console` 启用提示互动体验程序功能。
+
+本地安装 Gem，运行 `bundle exec rake install`。发布新版本，更新 `version.rb` 中的版本号，然后运行 `bundle exec rake release`，这将基于版本号创建一个 Git Tag，推送 Git 提交和创建 Tag，推送 `.gem` 文件到 [rubygems.org](https://rubygems.org)。
+
+## 贡献
+
+欢迎到 GitHub 的 https://github.com/hfl/mongolian 代码库提交问题报告和提交软件修正。
+
+## 许可
+
+本 Gem 基于[MIT 许可](https://opensource.org/licenses/MIT)分发。
+
+## Code of Conduct
+
+Everyone interacting in the Mongolian project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/hfl/mongolian/blob/main/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "minitest/test_task"
+
+Minitest::TestTask.create
+
+task default: :test

data/lib/mongolian/latin.rb

@@ -0,0 +1,107 @@
+module Mongolian
+  # 传统蒙古文很多时候需要对其进行拉丁转写，本程序对相关传统蒙古文词句进行转写转换
+  # 转换原则：
+  # 1. 仅针对传统蒙古文进行转换；
+  # 2. 传统蒙古文与拉丁转写字母一一对应；
+  # 3. 根据 Unicode 定义，转写不分阴阳形式；
+  
+  # 转换传统蒙古文为拉丁转写
+  def m2l
+    mongolian_str = self.dup.to_str
+    mongolian_str = mongolian_str.gsub(/ᠠ/, "a")
+    mongolian_str = mongolian_str.gsub(/ᠡ/, "e")
+    mongolian_str = mongolian_str.gsub(/ᠢ/, "i")
+    mongolian_str = mongolian_str.gsub(/ᠣ/, "o")
+    mongolian_str = mongolian_str.gsub(/ᠤ/, "u")
+    mongolian_str = mongolian_str.gsub(/ᠥ/, "ö")
+    mongolian_str = mongolian_str.gsub(/ᠦ/, "ü")
+    mongolian_str = mongolian_str.gsub(/ᠧ/, "ë")
+    mongolian_str = mongolian_str.gsub(/ᠨ/, "n")
+    mongolian_str = mongolian_str.gsub(/ᠩ/, "ŋ")
+    mongolian_str = mongolian_str.gsub(/ᠪ/, "b")
+    mongolian_str = mongolian_str.gsub(/ᠫ/, "p")
+    mongolian_str = mongolian_str.gsub(/ᠬ/, "x")
+    mongolian_str = mongolian_str.gsub(/ᠭ/, "g")
+    mongolian_str = mongolian_str.gsub(/ᠮ/, "m")
+    mongolian_str = mongolian_str.gsub(/ᠯ/, "l")
+    mongolian_str = mongolian_str.gsub(/ᠰ/, "s")
+    mongolian_str = mongolian_str.gsub(/ᠱ/, "š")
+    mongolian_str = mongolian_str.gsub(/ᠲ/, "t")
+    mongolian_str = mongolian_str.gsub(/ᠳ/, "d")
+    mongolian_str = mongolian_str.gsub(/ᠴ/, "č")
+    mongolian_str = mongolian_str.gsub(/ᠵ/, "ǰ")
+    mongolian_str = mongolian_str.gsub(/ᠶ/, "y")
+    mongolian_str = mongolian_str.gsub(/ᠷ/, "r")
+    mongolian_str = mongolian_str.gsub(/ᠸ/, "w")
+    mongolian_str = mongolian_str.gsub(/ᠹ/, "f")
+    mongolian_str = mongolian_str.gsub(/ᠺ/, "k")
+    mongolian_str = mongolian_str.gsub(/ᠻ/, "ḳ")
+    mongolian_str = mongolian_str.gsub(/ᠼ/, "c")
+    mongolian_str = mongolian_str.gsub(/ᠽ/, "z")
+    mongolian_str = mongolian_str.gsub(/ᠾ/, "h")
+    mongolian_str = mongolian_str.gsub(/ᠿ/, "ž")
+    mongolian_str = mongolian_str.gsub(/ᡀ/, "lh")
+    mongolian_str = mongolian_str.gsub(/ᡁ/, "ẑ")
+    mongolian_str = mongolian_str.gsub(/ᡂ/, "ĉ")
+    mongolian_str = mongolian_str.gsub(/᠀/, "&")
+    mongolian_str = mongolian_str.gsub(/᠁/, "…")
+    mongolian_str = mongolian_str.gsub(/᠂/, ",")
+    mongolian_str = mongolian_str.gsub(/᠃/, ".")
+    mongolian_str = mongolian_str.gsub(/᠇/, ":")
+    mongolian_str = mongolian_str.gsub(/᠈/, "#")
+    mongolian_str = mongolian_str.gsub(/᠊/, "‐")
+    mongolian_str = mongolian_str.gsub(/᠋/, "") #fvs1
+    mongolian_str = mongolian_str.gsub(/᠌/, "") #fvs2
+    mongolian_str = mongolian_str.gsub(/᠍/, "") #fvs3
+    mongolian_str = mongolian_str.gsub(/᠎/, "_")
+    mongolian_str = mongolian_str.gsub(/᠐/, "'0")
+    mongolian_str = mongolian_str.gsub(/᠑/, "'1")
+    mongolian_str = mongolian_str.gsub(/᠒/, "'2")
+    mongolian_str = mongolian_str.gsub(/᠓/, "'3")
+    mongolian_str = mongolian_str.gsub(/᠔/, "'4")
+    mongolian_str = mongolian_str.gsub(/᠕/, "'5")
+    mongolian_str = mongolian_str.gsub(/᠖/, "'6")
+    mongolian_str = mongolian_str.gsub(/᠗/, "'7")
+    mongolian_str = mongolian_str.gsub(/᠘/, "'8")
+    mongolian_str = mongolian_str.gsub(/᠙/, "'9")
+    mongolian_str = mongolian_str.gsub(/ᡛ/, "ń")
+    mongolian_str = mongolian_str.gsub(/ᢀ/, "ṃ")
+    mongolian_str = mongolian_str.gsub(/ᢁ/, "ḥ")
+    mongolian_str = mongolian_str.gsub(/ᢂ/, "â")
+    mongolian_str = mongolian_str.gsub(/ᢃ/, "ŏ")
+    mongolian_str = mongolian_str.gsub(/ᢄ/, "ô")
+    mongolian_str = mongolian_str.gsub(/ᢅ/, "ˑ")
+    mongolian_str = mongolian_str.gsub(/ᢆ/, "ːˑ")
+    mongolian_str = mongolian_str.gsub(/ᢇ/, "ā")
+    mongolian_str = mongolian_str.gsub(/ᢈ/, "ī")
+    mongolian_str = mongolian_str.gsub(/ᢉ/, "ḵ")
+    mongolian_str = mongolian_str.gsub(/ᢊ/, "ṉ")
+    mongolian_str = mongolian_str.gsub(/ᢋ/, "ƈ")
+    mongolian_str = mongolian_str.gsub(/ᢌ/, "ť")
+    mongolian_str = mongolian_str.gsub(/ᢍ/, "ţ")
+    mongolian_str = mongolian_str.gsub(/ᢎ/, "ḏ")
+    mongolian_str = mongolian_str.gsub(/ᢏ/, "ņ")
+    mongolian_str = mongolian_str.gsub(/ᢐ/, "ṯ")
+    mongolian_str = mongolian_str.gsub(/ᢑ/, "ḓ")
+    mongolian_str = mongolian_str.gsub(/ᢒ/, "p̄")
+    mongolian_str = mongolian_str.gsub(/ᢓ/, "ṕh")
+    mongolian_str = mongolian_str.gsub(/ᢔ/, "ş")
+    mongolian_str = mongolian_str.gsub(/ᢕ/, "ẖ")
+    mongolian_str = mongolian_str.gsub(/ᢖ/, "ẕ")
+    mongolian_str = mongolian_str.gsub(/ᢗ/, "ʒ̄")
+    mongolian_str = mongolian_str.gsub(/ᢦ/, "ŭ")
+    mongolian_str = mongolian_str.gsub(/ᢧ/, "ÿ")
+    mongolian_str = mongolian_str.gsub(/ᢩ/, "̲")
+    mongolian_str = mongolian_str.gsub(/‌/, "^")
+    mongolian_str = mongolian_str.gsub(/‍/, "*")
+    mongolian_str = mongolian_str.gsub(/ /, "-")
+    return mongolian_str
+  end
+  
+  # 对拉丁转写蒙古文恢复为蒙古文
+  def l2m
+    latin_str = self.dup.to_str
+    mongolian_str = latin_str
+    return mongolian_str
+  end
+end

data/lib/mongolian/spellchecker.rb

@@ -0,0 +1,88 @@
+module Mongolian
+  # 基于语法的拼写检查包括元音和谐律及词性检测、划分音节等
+
+  # 测试单词是否符合元音和谐律，并判断词性 参考P87
+  # 返回值 0：错误，不符合元音和谐律，外来词、双词根词或者拼写错误
+  # 返回值 1：正确，且为阳性词
+  # 返回值 10：阳性词，但有错误，第一音节出现第五元音，却在后面出现了第四元音
+  # 返回值 2：正确，阴性词
+  # 返回值 20：阴性词，但有错误，第一音节出现第七元音，却在后面出现了第六元音
+  def vowel_harmony
+    mongolian_str = self.dup.to_str
+    #vowel = ["ᠠ", "ᠡ", "ᠢ", "ᠣ", "ᠤ", "ᠥ", "ᠦ"]
+    if mongolian_str =~ /[ᠠᠣᠤ]/ and mongolian_str =~ /[ᠡᠥᠦ]/
+      return 0
+    else
+      if mongolian_str =~ /[ᠠᠣᠤ]/
+        #如果第一音节阳性元音为第 5 元音，却在后面出现第 4 元音，则错误
+        if mongolian_str.syllable[0] =~ /ᠤ/ and mongolian_str[mongolian_str.syllable[0].size..-1] =~ /ᠣ/
+          return 10
+        else
+          return 1
+        end
+      else
+        #如果第一音节阳性元音为第 7 元音，却在后面出现第 6 元音，则错误
+        if mongolian_str.syllable[0] =~ /ᠦ/ and mongolian_str[mongolian_str.syllable[0].size..-1] =~ /ᠥ/
+          return 20
+        else
+          return 2
+        end
+      end
+    end
+  end
+  
+  # 对单词划分音节，返回音节数组
+  def syllable
+    mongolian_str = self.dup.to_str
+    mlist = []
+    s = ""
+    mongolian_str.each_char do |c|
+      if c =~ /[ᠠᠡᠢᠣᠤᠥᠦ]/
+        if s[-1] =~ /[ᠠᠡᠢᠣᠤᠥᠦ]/
+          s += c
+        else
+          if s[0..-2].size > 0
+            mlist << s[0..-2]
+            s = s[-1]
+            s += c
+          else
+            s += c
+          end
+        end
+      else
+        s += c
+      end
+    end
+    mlist << s if s.size > 0
+    return mlist
+  end
+  
+  # 蒙古语里没有复辅音是指在书面语单音节中没有复辅音 P89
+  def self.double_constant_check(str)
+    mongolian_str = str
+    result = 1
+    syllable(mongolian_str).each do |s|
+      sv = s.scan(/[ᠠᠡᠢᠣᠤᠥᠦ]/).join
+      if s.index(sv) < 2 and s.size - s.index(sv) - sv.size < 2
+        next
+      else
+        result = 0
+        break
+      end
+    end
+    return result
+  end
+  
+  # 硬辅音 ᠪ(b) ᠭ(g) ᠷ(r) ᠰ(s) ᠳ(d) 后接以 ᠲ(t) ᠴ(q) 为首的附加成分
+  # 软辅音 ᠨ(n) ᠮ(m) ᠯ(l) ᠩ(ng) 后接以 ᠳ(d) ᠵ(j)为首的附加成分
+  
+  # _ 用于分写 ᠠ(a)/ᠡ(e) 跟前面的辅音(h/g + a; n, l, m, s, sh, j, y, r , W + a/e)
+  def ae?(str)
+    if str =~ /᠎[ᠠᠡ]/
+      return true
+    else
+      false
+    end
+  end
+  
+end

data/lib/mongolian/stemmify.rb

@@ -0,0 +1,14 @@
+module Mongolian
+  ## 对做好分词的蒙古文词进行词干提取。
+  def mongolian_stemmify
+    w = self.dup.to_str
+    return w if w.length < 4
+    
+    # 在蒙古文空格后，必定是后缀，删除
+    if w =~ /᠎/
+      w = $`
+    end
+    
+    return w
+  end
+end

data/lib/mongolian/tokenizer.rb

@@ -0,0 +1,25 @@
+module Mongolian
+  ##
+  # 判断一个词或者字符是否包含蒙古文字符，
+  # 如果包含蒙古文返回 true，否则返回 nil。
+  def is_mongolian?
+    return true if !!(self =~ /\p{Mongolian}/)
+  end
+  
+  ## 对于混合有多个文种的一段文字，删除其中的非蒙古文字符和蒙古文标点符号后，
+  #  仅对蒙古文分词。
+  #  函数返回仅包含蒙古文的分词数组。
+  def mongolian_tokenize
+    w = self.dup.to_str
+    # 去除标点符号，用空格替换标点符号
+    w = w.gsub(/[!᠄?·᠃᠂⁈⁉᠁—;《》]/, " ")
+    new_w = ""
+    # 将字符串中所有非蒙古文字符删除
+    w.each_char do |c|
+      c = "" unless c.is_mongolian?
+      new_w = new_w + c
+    end
+    return new_w.split
+  end
+
+end

data/lib/mongolian/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Mongolian
+  VERSION = "0.1.0"
+end

data/lib/mongolian.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+require_relative "mongolian/version"
+require_relative "mongolian/latin"
+require_relative "mongolian/tokenizer"
+require_relative "mongolian/spellchecker"
+require_relative "mongolian/stemmify"
+
+module Mongolian
+  class Error < StandardError; end
+end
+
+## 扩展 String
+class String
+  include Mongolian
+end

data/sig/mongolian.rbs

@@ -0,0 +1,4 @@
+module Mongolian
+  VERSION: String
+  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+end

metadata

@@ -0,0 +1,59 @@
+--- !ruby/object:Gem::Specification
+name: mongolian
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Huang Fei-Long
+autorequire:
+bindir: exe
+cert_chain: []
+date: 2025-06-11 00:00:00.000000000 Z
+dependencies: []
+description: Mongolian information process, tokenizer, stemming, and so on ...
+email:
+- huangfeilong@imu.edu.cn
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/mongolian.rb
+- lib/mongolian/latin.rb
+- lib/mongolian/spellchecker.rb
+- lib/mongolian/stemmify.rb
+- lib/mongolian/tokenizer.rb
+- lib/mongolian/version.rb
+- sig/mongolian.rbs
+homepage: https://huangfeilong.cn/
+licenses:
+- MIT
+metadata:
+  allowed_push_host: https://rubygems.org
+  homepage_uri: https://huangfeilong.cn/
+  source_code_uri: https://github.com/hfl/mongolian
+  changelog_uri: https://github.com/hfl/mongolian/CHANGELOG.md
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.18
+signing_key:
+specification_version: 4
+summary: Mongolian Information Process Technology in Ruby.
+test_files: []