kotoshu 0.10.0 → 0.11.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.kotoshu-baseline.json +1 -0
- data/CHANGELOG.md +36 -0
- data/README.adoc +30 -19
- data/Rakefile +35 -0
- data/TODO.impl/00-vision.md +17 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +1 -1
- data/lib/kotoshu/baseline/store.rb +23 -0
- data/lib/kotoshu/cache/language_cache.rb +26 -11
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache.rb +3 -0
- data/lib/kotoshu/cli/baseline_command.rb +49 -11
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +10 -5
- data/lib/kotoshu/cli.rb +24 -62
- data/lib/kotoshu/fluent_checker.rb +3 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +16 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +3 -3
- data/lib/kotoshu/keyboard/registry.rb +33 -7
- data/lib/kotoshu/keyboard.rb +10 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language.rb +5 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/sr/language.rb +2 -1
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages.rb +12 -0
- data/lib/kotoshu/multi_language_checker.rb +3 -0
- data/lib/kotoshu/resource_manager.rb +12 -1
- data/lib/kotoshu/spellchecker.rb +24 -12
- data/lib/kotoshu/tasks/check_task.rb +9 -3
- data/lib/kotoshu/version.rb +1 -1
- data/lib/kotoshu.rb +2 -7
- data/scripts/extract_eval_grids.rb +1 -1
- data/sig/kotoshu.rbs +56 -0
- metadata +27 -6
- data/lib/kotoshu/commands/cache_command.rb +0 -421
- data/lib/kotoshu/commands/check_command.rb +0 -301
- data/lib/kotoshu/commands/model_command.rb +0 -291
- data/lib/kotoshu/commands.rb +0 -12
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: b024cec75cd870973f2993b7eac7b4dba110867803896bff14588513b9a75cb4
|
|
4
|
+
data.tar.gz: aa4a76db758b89a7ae14ac24f59905216be4a3fdc5e523a97c4c3275013da706
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e882b3899dedf60569ca598741be502d6606964616186de31e57ab3c2150e0886f1b445be7147167fc69c880eb603b9c5d9a1b67e20d1f5c801884a0f6083f19
|
|
7
|
+
data.tar.gz: 0ebadb70fffa35c2847268871ce0b030b9626989ad980ff8de02ab02bc53f62ee9f395b98c2e37d9ec4caab13bc91d7e1c5b69120cdc901c861c04d9dd02c91a
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":1,"entries":[{"file":"CHANGELOG.md","line":167,"word":"'","count":1},{"file":"CHANGELOG.md","line":789,"word":"'KOTOSHU","count":1},{"file":"CHANGELOG.md","line":530,"word":"'Multi","count":1},{"file":"CHANGELOG.md","line":488,"word":"'protocols","count":1},{"file":"CHANGELOG.md","line":261,"word":"'s","count":2},{"file":"CHANGELOG.md","line":528,"word":"AZERTY","count":1},{"file":"CHANGELOG.md","line":182,"word":"ActiveModel","count":2},{"file":"CHANGELOG.md","line":172,"word":"AffReader","count":3},{"file":"CHANGELOG.md","line":373,"word":"ArgumentError","count":2},{"file":"CHANGELOG.md","line":238,"word":"AsciiDoc","count":3},{"file":"CHANGELOG.md","line":777,"word":"Asciidoctor","count":1},{"file":"CHANGELOG.md","line":324,"word":"AuditLog","count":4},{"file":"CHANGELOG.md","line":658,"word":"AutoSetup","count":2},{"file":"CHANGELOG.md","line":69,"word":"BACKEND","count":2},{"file":"CHANGELOG.md","line":82,"word":"BDS","count":1},{"file":"CHANGELOG.md","line":137,"word":"BaseCache","count":6},{"file":"CHANGELOG.md","line":540,"word":"Behaviour","count":1},{"file":"CHANGELOG.md","line":389,"word":"BloomFilter","count":1},{"file":"CHANGELOG.md","line":282,"word":"CHECKCOMPOUNDPATTERN","count":2},{"file":"CHANGELOG.md","line":513,"word":"CHECKSHARPS","count":1},{"file":"CHANGELOG.md","line":510,"word":"CIRCUMFIX","count":1},{"file":"CHANGELOG.md","line":731,"word":"CJK","count":3},{"file":"CHANGELOG.md","line":165,"word":"CLI","count":21},{"file":"CHANGELOG.md","line":174,"word":"COMPOUNDRULE","count":2},{"file":"CHANGELOG.md","line":666,"word":"CONFIG","count":2},{"file":"CHANGELOG.md","line":391,"word":"CSpell","count":3},{"file":"CHANGELOG.md","line":555,"word":"CacheCommand","count":1},{"file":"CHANGELOG.md","line":1,"word":"Changelog","count":2},{"file":"CHANGELOG.md","line":197,"word":"CheckTask","count":1},{"file":"CHANGELOG.md","line":350,"word":"Cli","count":7},{"file":"CHANGELOG.md","line":149,"word":"CommonWordsLoader","count":1},{"file":"CHANGELOG.md","line":511,"word":"ConvTable","count":1},{"file":"CHANGELOG.md","line":51,"word":"DIC","count":1},{"file":"CHANGELOG.md","line":524,"word":"DRY'd","count":1},{"file":"CHANGELOG.md","line":108,"word":"Damerau","count":2},{"file":"CHANGELOG.md","line":297,"word":"DictionaryNotFoundError","count":1},{"file":"CHANGELOG.md","line":242,"word":"DocumentResult","count":2},{"file":"CHANGELOG.md","line":465,"word":"DoubleNegativeMatcher","count":1},{"file":"CHANGELOG.md","line":16,"word":"Dubeolsik","count":1},{"file":"CHANGELOG.md","line":145,"word":"ENV","count":3},{"file":"CHANGELOG.md","line":183,"word":"EachValidator","count":1},{"file":"CHANGELOG.md","line":113,"word":"EditDistance","count":2},{"file":"CHANGELOG.md","line":101,"word":"EditDistanceStrategy","count":2},{"file":"CHANGELOG.md","line":428,"word":"EmbeddingPipeline","count":1},{"file":"CHANGELOG.md","line":292,"word":"Embeddings","count":11},{"file":"CHANGELOG.md","line":573,"word":"EmptyLine","count":1},{"file":"CHANGELOG.md","line":309,"word":"EvictionPolicy","count":1},{"file":"CHANGELOG.md","line":572,"word":"ExtraSpacing","count":1},{"file":"CHANGELOG.md","line":628,"word":"FastText","count":4},{"file":"CHANGELOG.md","line":443,"word":"FrequencyCache","count":7},{"file":"CHANGELOG.md","line":141,"word":"FrequencyProvider","count":1},{"file":"CHANGELOG.md","line":441,"word":"FrozenError","count":1},{"file":"CHANGELOG.md","line":511,"word":"ICONV","count":3},{"file":"CHANGELOG.md","line":516,"word":"IJ","count":1},{"file":"CHANGELOG.md","line":15,"word":"IME","count":1},{"file":"CHANGELOG.md","line":283,"word":"INITCAP","count":1},{"file":"CHANGELOG.md","line":82,"word":"ISIRI","count":1},{"file":"CHANGELOG.md","line":21,"word":"InScript","count":1},{"file":"CHANGELOG.md","line":295,"word":"InferenceSession","count":1},{"file":"CHANGELOG.md","line":338,"word":"IntegrityError","count":3},{"file":"CHANGELOG.md","line":38,"word":"JCUKEN","count":4},{"file":"CHANGELOG.md","line":213,"word":"JSON","count":7},{"file":"CHANGELOG.md","line":513,"word":"KEEPCASE","count":1},{"file":"CHANGELOG.md","line":51,"word":"KOTOSHU","count":21},{"file":"CHANGELOG.md","line":48,"word":"Kotoshu","count":46},{"file":"CHANGELOG.md","line":777,"word":"Kramdown","count":1},{"file":"CHANGELOG.md","line":289,"word":"LFS","count":1},{"file":"CHANGELOG.md","line":311,"word":"LRU","count":1},{"file":"CHANGELOG.md","line":52,"word":"LSP","count":2},{"file":"CHANGELOG.md","line":30,"word":"LanguageCache","count":7},{"file":"CHANGELOG.md","line":745,"word":"LanguageResolver","count":1},{"file":"CHANGELOG.md","line":80,"word":"LatinBase","count":1},{"file":"CHANGELOG.md","line":113,"word":"Levenshtein","count":3},{"file":"CHANGELOG.md","line":71,"word":"LidModel","count":1},{"file":"CHANGELOG.md","line":489,"word":"LoadError","count":1},{"file":"CHANGELOG.md","line":393,"word":"LookupBuilder","count":1},{"file":"CHANGELOG.md","line":366,"word":"LruCache","count":1},{"file":"CHANGELOG.md","line":189,"word":"Matchers","count":1},{"file":"CHANGELOG.md","line":691,"word":"MiB","count":1},{"file":"CHANGELOG.md","line":334,"word":"ModelCache","count":2},{"file":"CHANGELOG.md","line":568,"word":"MultiWord","count":1},{"file":"CHANGELOG.md","line":139,"word":"NetHTTP","count":1},{"file":"CHANGELOG.md","line":429,"word":"NoMethodError","count":1},{"file":"CHANGELOG.md","line":436,"word":"NotImplementedError","count":1},{"file":"CHANGELOG.md","line":43,"word":"Nynorsk","count":1},{"file":"CHANGELOG.md","line":511,"word":"OCONV","count":3},{"file":"CHANGELOG.md","line":62,"word":"ONNX","count":15},{"file":"CHANGELOG.md","line":789,"word":"ONNX'","count":1},{"file":"CHANGELOG.md","line":474,"word":"OOV","count":4},{"file":"CHANGELOG.md","line":294,"word":"OnnxModel","count":3},{"file":"CHANGELOG.md","line":426,"word":"OnnxRuntimeModel","count":2},{"file":"CHANGELOG.md","line":705,"word":"OnnxUnavailable","count":1},{"file":"CHANGELOG.md","line":348,"word":"PersonalDictionary","count":1},{"file":"CHANGELOG.md","line":769,"word":"Phonet","count":1},{"file":"CHANGELOG.md","line":459,"word":"PhonetSuggest","count":2},{"file":"CHANGELOG.md","line":458,"word":"PhonetTable","count":1},{"file":"CHANGELOG.md","line":122,"word":"PlainText","count":5},{"file":"CHANGELOG.md","line":689,"word":"ProgressReporter","count":2},{"file":"CHANGELOG.md","line":83,"word":"QWERTZ","count":2},{"file":"CHANGELOG.md","line":257,"word":"README","count":3},{"file":"CHANGELOG.md","line":662,"word":"REPOS","count":1},{"file":"CHANGELOG.md","line":188,"word":"RSpec","count":1},{"file":"CHANGELOG.md","line":78,"word":"RTL","count":7},{"file":"CHANGELOG.md","line":173,"word":"RegexpError","count":1},{"file":"CHANGELOG.md","line":743,"word":"ResourceBundle","count":1},{"file":"CHANGELOG.md","line":141,"word":"ResourceManager","count":6},{"file":"CHANGELOG.md","line":139,"word":"ResourceNotCachedError","count":1},{"file":"CHANGELOG.md","line":656,"word":"ResourceNotSetupError","count":3},{"file":"CHANGELOG.md","line":147,"word":"ResourceUnavailable","count":1},{"file":"CHANGELOG.md","line":752,"word":"Ribose","count":2},{"file":"CHANGELOG.md","line":23,"word":"Romanized","count":1},{"file":"CHANGELOG.md","line":319,"word":"RotationPolicy","count":1},{"file":"CHANGELOG.md","line":189,"word":"Rspec","count":1},{"file":"CHANGELOG.md","line":570,"word":"RuboCop","count":1},{"file":"CHANGELOG.md","line":213,"word":"SARIF","count":5},{"file":"CHANGELOG.md","line":335,"word":"SHA","count":2},{"file":"CHANGELOG.md","line":347,"word":"SUBCOMMAND","count":3},{"file":"CHANGELOG.md","line":474,"word":"SemanticAnalyzer","count":1},{"file":"CHANGELOG.md","line":385,"word":"SemanticStrategy","count":3},{"file":"CHANGELOG.md","line":744,"word":"SetupResult","count":1},{"file":"CHANGELOG.md","line":370,"word":"SimilarityEngine","count":1},{"file":"CHANGELOG.md","line":601,"word":"SimilaritySearch","count":2},{"file":"CHANGELOG.md","line":659,"word":"SourceRegistry","count":2},{"file":"CHANGELOG.md","line":182,"word":"SpellingValidator","count":1},{"file":"CHANGELOG.md","line":31,"word":"StagedLanguages","count":1},{"file":"CHANGELOG.md","line":745,"word":"StatusReport","count":1},{"file":"CHANGELOG.md","line":598,"word":"StringReader","count":1},{"file":"CHANGELOG.md","line":594,"word":"SuggestionSet","count":2},{"file":"CHANGELOG.md","line":239,"word":"Suppressions","count":1},{"file":"CHANGELOG.md","line":624,"word":"SymSpell","count":1},{"file":"CHANGELOG.md","line":522,"word":"TODO","count":21},{"file":"CHANGELOG.md","line":737,"word":"TODOs","count":1},{"file":"CHANGELOG.md","line":773,"word":"TTLs","count":1},{"file":"CHANGELOG.md","line":686,"word":"TTY","count":3},{"file":"CHANGELOG.md","line":103,"word":"Teh","count":1},{"file":"CHANGELOG.md","line":440,"word":"Trie","count":1},{"file":"CHANGELOG.md","line":359,"word":"TypeError","count":2},{"file":"CHANGELOG.md","line":123,"word":"UnixWords","count":3},{"file":"CHANGELOG.md","line":350,"word":"UsageError","count":1},{"file":"CHANGELOG.md","line":182,"word":"Validators","count":1},{"file":"CHANGELOG.md","line":592,"word":"WordResult","count":3},{"file":"CHANGELOG.md","line":664,"word":"XDG","count":5},{"file":"CHANGELOG.md","line":564,"word":"YAML","count":1},{"file":"CHANGELOG.md","line":223,"word":"ZM","count":1},{"file":"CHANGELOG.md","line":512,"word":"ZWJ","count":1},{"file":"CHANGELOG.md","line":97,"word":"ZWNJ","count":2},{"file":"CHANGELOG.md","line":205,"word":"adoc","count":1},{"file":"CHANGELOG.md","line":172,"word":"aff","count":4},{"file":"CHANGELOG.md","line":78,"word":"ar","count":3},{"file":"CHANGELOG.md","line":394,"word":"args","count":1},{"file":"CHANGELOG.md","line":205,"word":"asciidoc","count":1},{"file":"CHANGELOG.md","line":371,"word":"attr","count":3},{"file":"CHANGELOG.md","line":359,"word":"autoload","count":7},{"file":"CHANGELOG.md","line":495,"word":"autoloads","count":1},{"file":"CHANGELOG.md","line":551,"word":"backend's","count":1},{"file":"CHANGELOG.md","line":767,"word":"backends","count":1},{"file":"CHANGELOG.md","line":202,"word":"baselined","count":1},{"file":"CHANGELOG.md","line":451,"word":"basename","count":1},{"file":"CHANGELOG.md","line":551,"word":"behaviour","count":1},{"file":"CHANGELOG.md","line":78,"word":"bg","count":2},{"file":"CHANGELOG.md","line":517,"word":"breakdefault","count":1},{"file":"CHANGELOG.md","line":296,"word":"camelCase","count":1},{"file":"CHANGELOG.md","line":666,"word":"cfg","count":1},{"file":"CHANGELOG.md","line":804,"word":"checksummed","count":1},{"file":"CHANGELOG.md","line":729,"word":"circumfix","count":2},{"file":"CHANGELOG.md","line":733,"word":"cjk","count":2},{"file":"CHANGELOG.md","line":697,"word":"cli","count":2},{"file":"CHANGELOG.md","line":51,"word":"config","count":5},{"file":"CHANGELOG.md","line":785,"word":"csv","count":1},{"file":"CHANGELOG.md","line":220,"word":"da","count":1},{"file":"CHANGELOG.md","line":20,"word":"danda","count":1},{"file":"CHANGELOG.md","line":170,"word":"de","count":3},{"file":"CHANGELOG.md","line":111,"word":"deduplicated","count":1},{"file":"CHANGELOG.md","line":104,"word":"definately","count":1},{"file":"CHANGELOG.md","line":51,"word":"dic","count":3},{"file":"CHANGELOG.md","line":411,"word":"docstring","count":1},{"file":"CHANGELOG.md","line":222,"word":"dotless","count":3},{"file":"CHANGELOG.md","line":565,"word":"ecke","count":1},{"file":"CHANGELOG.md","line":166,"word":"el","count":2},{"file":"CHANGELOG.md","line":487,"word":"embeddings","count":4},{"file":"CHANGELOG.md","line":14,"word":"eojeol","count":1},{"file":"CHANGELOG.md","line":683,"word":"errorCount","count":1},{"file":"CHANGELOG.md","line":514,"word":"eszet","count":1},{"file":"CHANGELOG.md","line":78,"word":"et","count":2},{"file":"CHANGELOG.md","line":22,"word":"eval","count":7},{"file":"CHANGELOG.md","line":788,"word":"exe","count":1},{"file":"CHANGELOG.md","line":45,"word":"fallbacks","count":1},{"file":"CHANGELOG.md","line":63,"word":"fastText","count":1},{"file":"CHANGELOG.md","line":661,"word":"fasttext","count":3},{"file":"CHANGELOG.md","line":517,"word":"forbiddenword","count":1},{"file":"CHANGELOG.md","line":18,"word":"fortis","count":1},{"file":"CHANGELOG.md","line":70,"word":"gemspec","count":2},{"file":"CHANGELOG.md","line":98,"word":"geresh","count":1},{"file":"CHANGELOG.md","line":98,"word":"gershayim","count":1},{"file":"CHANGELOG.md","line":196,"word":"gitignore","count":2},{"file":"CHANGELOG.md","line":316,"word":"hardcoded","count":1},{"file":"CHANGELOG.md","line":190,"word":"helo","count":1},{"file":"CHANGELOG.md","line":519,"word":"honoured","count":3},{"file":"CHANGELOG.md","line":6,"word":"html","count":1},{"file":"CHANGELOG.md","line":5,"word":"https","count":2},{"file":"CHANGELOG.md","line":220,"word":"hu","count":1},{"file":"CHANGELOG.md","line":637,"word":"hunspell","count":3},{"file":"CHANGELOG.md","line":522,"word":"impl","count":21},{"file":"CHANGELOG.md","line":530,"word":"implemented'","count":1},{"file":"CHANGELOG.md","line":246,"word":"init","count":1},{"file":"CHANGELOG.md","line":566,"word":"instantiation","count":1},{"file":"CHANGELOG.md","line":284,"word":"integrational","count":1},{"file":"CHANGELOG.md","line":179,"word":"integrations","count":1},{"file":"CHANGELOG.md","line":392,"word":"ivars","count":1},{"file":"CHANGELOG.md","line":223,"word":"izmir","count":1},{"file":"CHANGELOG.md","line":797,"word":"ja","count":1},{"file":"CHANGELOG.md","line":14,"word":"jamo","count":2},{"file":"CHANGELOG.md","line":157,"word":"jekyll","count":1},{"file":"CHANGELOG.md","line":201,"word":"json","count":9},{"file":"CHANGELOG.md","line":5,"word":"keepachangelog","count":1},{"file":"CHANGELOG.md","line":661,"word":"kelly","count":2},{"file":"CHANGELOG.md","line":405,"word":"klass","count":5},{"file":"CHANGELOG.md","line":13,"word":"ko","count":2},{"file":"CHANGELOG.md","line":25,"word":"kotoshu","count":80},{"file":"CHANGELOG.md","line":457,"word":"kwarg","count":1},{"file":"CHANGELOG.md","line":361,"word":"kwargs","count":2},{"file":"CHANGELOG.md","line":154,"word":"lang","count":4},{"file":"CHANGELOG.md","line":18,"word":"lenis","count":1},{"file":"CHANGELOG.md","line":326,"word":"lockfile","count":1},{"file":"CHANGELOG.md","line":78,"word":"lt","count":2},{"file":"CHANGELOG.md","line":248,"word":"lutaml","count":5},{"file":"CHANGELOG.md","line":78,"word":"lv","count":2},{"file":"CHANGELOG.md","line":105,"word":"ly","count":1},{"file":"CHANGELOG.md","line":70,"word":"magnus","count":1},{"file":"CHANGELOG.md","line":469,"word":"matcher","count":1},{"file":"CHANGELOG.md","line":188,"word":"matchers","count":1},{"file":"CHANGELOG.md","line":19,"word":"matras","count":1},{"file":"CHANGELOG.md","line":159,"word":"md","count":23},{"file":"CHANGELOG.md","line":205,"word":"mdx","count":1},{"file":"CHANGELOG.md","line":462,"word":"metaphone","count":1},{"file":"CHANGELOG.md","line":488,"word":"model'","count":1},{"file":"CHANGELOG.md","line":56,"word":"mtime","count":1},{"file":"CHANGELOG.md","line":677,"word":"mtimes","count":1},{"file":"CHANGELOG.md","line":302,"word":"namespace","count":5},{"file":"CHANGELOG.md","line":538,"word":"namespace's","count":1},{"file":"CHANGELOG.md","line":352,"word":"namespaces","count":2},{"file":"CHANGELOG.md","line":18,"word":"ne","count":2},{"file":"CHANGELOG.md","line":235,"word":"nestable","count":1},{"file":"CHANGELOG.md","line":283,"word":"ngram","count":3},{"file":"CHANGELOG.md","line":220,"word":"nl","count":1},{"file":"CHANGELOG.md","line":43,"word":"nn","count":1},{"file":"CHANGELOG.md","line":221,"word":"normalizer","count":3},{"file":"CHANGELOG.md","line":508,"word":"num","count":1},{"file":"CHANGELOG.md","line":582,"word":"onnx","count":6},{"file":"CHANGELOG.md","line":70,"word":"onnxruntime","count":8},{"file":"CHANGELOG.md","line":381,"word":"op'ing","count":1},{"file":"CHANGELOG.md","line":517,"word":"opentaal","count":1},{"file":"CHANGELOG.md","line":501,"word":"phonet","count":2},{"file":"CHANGELOG.md","line":445,"word":"positionally","count":1},{"file":"CHANGELOG.md","line":132,"word":"pre","count":6},{"file":"CHANGELOG.md","line":615,"word":"queryable","count":1},{"file":"CHANGELOG.md","line":716,"word":"quickstart","count":1},{"file":"CHANGELOG.md","line":231,"word":"qwertz","count":1},{"file":"CHANGELOG.md","line":229,"word":"rb","count":12},{"file":"CHANGELOG.md","line":22,"word":"repo","count":9},{"file":"CHANGELOG.md","line":267,"word":"repo's","count":1},{"file":"CHANGELOG.md","line":659,"word":"repos'","count":1},{"file":"CHANGELOG.md","line":280,"word":"rerank","count":1},{"file":"CHANGELOG.md","line":758,"word":"reranking","count":1},{"file":"CHANGELOG.md","line":220,"word":"ro","count":1},{"file":"CHANGELOG.md","line":261,"word":"roadmap","count":1},{"file":"CHANGELOG.md","line":188,"word":"rspec","count":2},{"file":"CHANGELOG.md","line":205,"word":"rst","count":1},{"file":"CHANGELOG.md","line":733,"word":"rtl","count":2},{"file":"CHANGELOG.md","line":798,"word":"ru","count":1},{"file":"CHANGELOG.md","line":570,"word":"rubocop","count":1},{"file":"CHANGELOG.md","line":681,"word":"sarif","count":2},{"file":"CHANGELOG.md","line":6,"word":"semver","count":1},{"file":"CHANGELOG.md","line":65,"word":"sha","count":3},{"file":"CHANGELOG.md","line":78,"word":"sk","count":2},{"file":"CHANGELOG.md","line":78,"word":"sl","count":2},{"file":"CHANGELOG.md","line":55,"word":"spellcheckable","count":1},{"file":"CHANGELOG.md","line":78,"word":"sr","count":2},{"file":"CHANGELOG.md","line":514,"word":"ss","count":1},{"file":"CHANGELOG.md","line":223,"word":"stanbul","count":1},{"file":"CHANGELOG.md","line":216,"word":"stdin","count":1},{"file":"CHANGELOG.md","line":401,"word":"subclasses","count":1},{"file":"CHANGELOG.md","line":348,"word":"subcommand","count":1},{"file":"CHANGELOG.md","line":383,"word":"subcommand's","count":1},{"file":"CHANGELOG.md","line":330,"word":"subcommands","count":1},{"file":"CHANGELOG.md","line":155,"word":"sublayout","count":1},{"file":"CHANGELOG.md","line":181,"word":"suika","count":2},{"file":"CHANGELOG.md","line":120,"word":"suleman","count":4},{"file":"CHANGELOG.md","line":676,"word":"summarises","count":1},{"file":"CHANGELOG.md","line":245,"word":"suppressedCount","count":1},{"file":"CHANGELOG.md","line":245,"word":"suppressedErrors","count":1},{"file":"CHANGELOG.md","line":251,"word":"suppressions","count":1},{"file":"CHANGELOG.md","line":220,"word":"sv","count":1},{"file":"CHANGELOG.md","line":624,"word":"symspell","count":2},{"file":"CHANGELOG.md","line":448,"word":"tmp","count":3},{"file":"CHANGELOG.md","line":355,"word":"tmpdir","count":1},{"file":"CHANGELOG.md","line":526,"word":"tokenization","count":1},{"file":"CHANGELOG.md","line":13,"word":"tokenizer","count":10},{"file":"CHANGELOG.md","line":36,"word":"tokenizers","count":4},{"file":"CHANGELOG.md","line":722,"word":"toolchain","count":1},{"file":"CHANGELOG.md","line":440,"word":"trie","count":1},{"file":"CHANGELOG.md","line":205,"word":"txt","count":1},{"file":"CHANGELOG.md","line":167,"word":"uk","count":2},{"file":"CHANGELOG.md","line":169,"word":"uncheckable","count":1},{"file":"CHANGELOG.md","line":683,"word":"uniqueErrorCount","count":1},{"file":"CHANGELOG.md","line":138,"word":"url","count":1},{"file":"CHANGELOG.md","line":768,"word":"usr","count":1},{"file":"CHANGELOG.md","line":508,"word":"utf","count":1},{"file":"CHANGELOG.md","line":120,"word":"uzair","count":4},{"file":"CHANGELOG.md","line":30,"word":"vendored","count":1},{"file":"CHANGELOG.md","line":19,"word":"virama","count":1},{"file":"CHANGELOG.md","line":683,"word":"wordCount","count":1},{"file":"CHANGELOG.md","line":254,"word":"yaml","count":3},{"file":"CHANGELOG.md","line":564,"word":"yml","count":1},{"file":"CHANGELOG.md","line":329,"word":"zsh","count":2},{"file":"CODE_OF_CONDUCT.md","line":122,"word":"CoC","count":2},{"file":"CODE_OF_CONDUCT.md","line":125,"word":"faq","count":2},{"file":"CODE_OF_CONDUCT.md","line":130,"word":"github","count":1},{"file":"CODE_OF_CONDUCT.md","line":119,"word":"html","count":2},{"file":"CODE_OF_CONDUCT.md","line":119,"word":"https","count":8},{"file":"CODE_OF_CONDUCT.md","line":130,"word":"mozilla","count":1},{"file":"CODE_OF_CONDUCT.md","line":34,"word":"others'","count":1},{"file":"CODE_OF_CONDUCT.md","line":30,"word":"sexualized","count":1},{"file":"CODE_OF_CONDUCT.md","line":8,"word":"socio","count":1},{"file":"CODE_OF_CONDUCT.md","line":119,"word":"www","count":7},{"file":"SECURITY.md","line":79,"word":"'HOME'","count":1},{"file":"SECURITY.md","line":79,"word":"AuditLog","count":1},{"file":"SECURITY.md","line":87,"word":"CLI","count":1},{"file":"SECURITY.md","line":53,"word":"CLI's","count":1},{"file":"SECURITY.md","line":79,"word":"ENV","count":1},{"file":"SECURITY.md","line":22,"word":"FastText","count":1},{"file":"SECURITY.md","line":24,"word":"HTTPS","count":1},{"file":"SECURITY.md","line":37,"word":"IntegrityError","count":1},{"file":"SECURITY.md","line":41,"word":"JSON","count":2},{"file":"SECURITY.md","line":60,"word":"KOTOSHU","count":2},{"file":"SECURITY.md","line":15,"word":"Kotoshu","count":5},{"file":"SECURITY.md","line":22,"word":"ONNX","count":1},{"file":"SECURITY.md","line":86,"word":"ResourceNotCachedError","count":1},{"file":"SECURITY.md","line":31,"word":"SHA","count":2},{"file":"SECURITY.md","line":6,"word":"acknowledgement","count":1},{"file":"SECURITY.md","line":22,"word":"fasttext","count":1},{"file":"SECURITY.md","line":24,"word":"githubusercontent","count":1},{"file":"SECURITY.md","line":72,"word":"jq","count":1},{"file":"SECURITY.md","line":30,"word":"json","count":1},{"file":"SECURITY.md","line":21,"word":"kelly","count":1},{"file":"SECURITY.md","line":5,"word":"kotoshu","count":14},{"file":"SECURITY.md","line":53,"word":"lang","count":1},{"file":"SECURITY.md","line":68,"word":"logrotate","count":1},{"file":"SECURITY.md","line":22,"word":"onnx","count":1},{"file":"SECURITY.md","line":93,"word":"onnxruntime","count":1},{"file":"SECURITY.md","line":87,"word":"pre","count":1},{"file":"SECURITY.md","line":18,"word":"repo","count":2},{"file":"SECURITY.md","line":26,"word":"repos","count":2},{"file":"SECURITY.md","line":93,"word":"suika","count":1},{"file":"SECURITY.md","line":61,"word":"umask","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":55,"word":"ActiveModelUnavailable","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"AffixRuleError","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":46,"word":"Autoloaded","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":69,"word":"CLI","count":5},{"file":"docs/STABILITY-1.0-AUDIT.md","line":45,"word":"CONFIG","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":91,"word":"CacheCommand","count":4},{"file":"docs/STABILITY-1.0-AUDIT.md","line":90,"word":"CheckCommand","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":57,"word":"CheckTask","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":69,"word":"Cli","count":10},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"ConfigurationError","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":67,"word":"ConformanceExporter","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":66,"word":"ConformanceRunner","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":48,"word":"DebugLogger","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":89,"word":"DictCommand","count":6},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"DictionaryNotFoundError","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":120,"word":"DocumentResult","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":40,"word":"ENV","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":49,"word":"FluentChecker","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"IntegrityError","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"InvalidDictionaryFormatError","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":221,"word":"JS","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":78,"word":"JSON","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":58,"word":"JekyllUnavailable","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":40,"word":"KOTOSHU","count":5},{"file":"docs/STABILITY-1.0-AUDIT.md","line":38,"word":"Kotoshu","count":50},{"file":"docs/STABILITY-1.0-AUDIT.md","line":110,"word":"KotoshuLid","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":107,"word":"KotoshuModel","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":106,"word":"KotoshuWasm","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":47,"word":"LanguageCache","count":3},{"file":"docs/STABILITY-1.0-AUDIT.md","line":47,"word":"LanguageIdentifier","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":56,"word":"Matcher","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":56,"word":"Matchers","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":48,"word":"MetricsCollector","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":47,"word":"ModelCache","count":3},{"file":"docs/STABILITY-1.0-AUDIT.md","line":92,"word":"ModelCommand","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":49,"word":"MultiLanguageChecker","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":90,"word":"NameError","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":46,"word":"Namespaces","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":100,"word":"NativeBackend","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":141,"word":"PRs","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":223,"word":"PyPI","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":227,"word":"RBS","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":23,"word":"README","count":10},{"file":"docs/STABILITY-1.0-AUDIT.md","line":9,"word":"Reachability","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":43,"word":"ResourceBundle","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":42,"word":"ResourceManager","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"ResourceNotCachedError","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"ResourceNotSetupError","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"ResourceResolutionError","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":56,"word":"Rspec","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":78,"word":"SARIF","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":221,"word":"SDK","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":47,"word":"SemanticAnalyzer","count":3},{"file":"docs/STABILITY-1.0-AUDIT.md","line":41,"word":"SourceRegistry","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":90,"word":"SpellChecker","count":3},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"SpellcheckError","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":55,"word":"SpellingValidator","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"SuikaUnavailable","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":154,"word":"Unwired","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":55,"word":"Validators","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":120,"word":"WordError","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":45,"word":"XDG","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":29,"word":"accessor","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":40,"word":"accessors","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":248,"word":"activemodel","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":137,"word":"additionalProperties","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":23,"word":"adoc","count":5},{"file":"docs/STABILITY-1.0-AUDIT.md","line":76,"word":"aff","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":106,"word":"affSource","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":107,"word":"arg","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":41,"word":"attrs","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":55,"word":"autoload","count":9},{"file":"docs/STABILITY-1.0-AUDIT.md","line":44,"word":"autoloaded","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":174,"word":"autoloads","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":100,"word":"backend","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":107,"word":"bucketsBytes","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":186,"word":"changelog","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":73,"word":"cli","count":21},{"file":"docs/STABILITY-1.0-AUDIT.md","line":38,"word":"config","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":135,"word":"const","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":90,"word":"csv","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":39,"word":"deprecations","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":51,"word":"deps","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":110,"word":"detectLanguage","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":76,"word":"dic","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":106,"word":"dicSource","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":197,"word":"docstrings","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":249,"word":"envs","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":10,"word":"exe","count":3},{"file":"docs/STABILITY-1.0-AUDIT.md","line":98,"word":"extconf","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":6,"word":"fac","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":92,"word":"fasttext","count":3},{"file":"docs/STABILITY-1.0-AUDIT.md","line":65,"word":"ffi","count":3},{"file":"docs/STABILITY-1.0-AUDIT.md","line":91,"word":"gemspec","count":5},{"file":"docs/STABILITY-1.0-AUDIT.md","line":208,"word":"gitignored","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":90,"word":"hunspell","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":81,"word":"init","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":191,"word":"inlined","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":51,"word":"integrations","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":58,"word":"jekyll","count":3},{"file":"docs/STABILITY-1.0-AUDIT.md","line":221,"word":"js","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":74,"word":"json","count":8},{"file":"docs/STABILITY-1.0-AUDIT.md","line":64,"word":"jsonl","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":6,"word":"kotoshu","count":119},{"file":"docs/STABILITY-1.0-AUDIT.md","line":38,"word":"kwargs","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":136,"word":"lang","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":177,"word":"lexically","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":110,"word":"loadLid","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":107,"word":"loadModel","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":7,"word":"md","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":107,"word":"modelBytes","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":29,"word":"namespace","count":6},{"file":"docs/STABILITY-1.0-AUDIT.md","line":46,"word":"namespaces","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":112,"word":"npm","count":5},{"file":"docs/STABILITY-1.0-AUDIT.md","line":92,"word":"onnx","count":3},{"file":"docs/STABILITY-1.0-AUDIT.md","line":110,"word":"onnxBytes","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":248,"word":"onnxruntime","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":118,"word":"openapi","count":6},{"file":"docs/STABILITY-1.0-AUDIT.md","line":127,"word":"prewarm","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":92,"word":"py","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":6,"word":"rb","count":98},{"file":"docs/STABILITY-1.0-AUDIT.md","line":227,"word":"rbs","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":164,"word":"repo","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":41,"word":"repos","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":108,"word":"rerank","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":56,"word":"rspec","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":74,"word":"sarif","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":120,"word":"schemas","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":109,"word":"semanticSuggest","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":137,"word":"sha","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":176,"word":"sig","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":106,"word":"src","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":90,"word":"subcommand","count":5},{"file":"docs/STABILITY-1.0-AUDIT.md","line":79,"word":"subcommands","count":3},{"file":"docs/STABILITY-1.0-AUDIT.md","line":248,"word":"suika","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":38,"word":"tokenize","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":30,"word":"trie","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":92,"word":"untracked","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":41,"word":"url","count":5},{"file":"docs/STABILITY-1.0-AUDIT.md","line":55,"word":"validator","count":1},{"file":"docs/STABILITY-1.0-AUDIT.md","line":55,"word":"validators","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":107,"word":"vocabBytes","count":2},{"file":"docs/STABILITY-1.0-AUDIT.md","line":65,"word":"wasm","count":19},{"file":"docs/STABILITY-1.0-AUDIT.md","line":90,"word":"yaml","count":8},{"file":"docs/STABILITY-1.0-AUDIT.md","line":77,"word":"yml","count":4},{"file":"docs/STABILITY-1.0-AUDIT.md","line":82,"word":"zsh","count":1},{"file":"docs/STABILITY-1.0.md","line":102,"word":"CHANGELOG","count":1},{"file":"docs/STABILITY-1.0.md","line":22,"word":"CLI","count":3},{"file":"docs/STABILITY-1.0.md","line":22,"word":"ENV","count":3},{"file":"docs/STABILITY-1.0.md","line":46,"word":"JS","count":1},{"file":"docs/STABILITY-1.0.md","line":31,"word":"JSON","count":2},{"file":"docs/STABILITY-1.0.md","line":121,"word":"KOTOSHU","count":1},{"file":"docs/STABILITY-1.0.md","line":1,"word":"Kotoshu","count":1},{"file":"docs/STABILITY-1.0.md","line":49,"word":"KotoshuLid","count":1},{"file":"docs/STABILITY-1.0.md","line":49,"word":"KotoshuModel","count":1},{"file":"docs/STABILITY-1.0.md","line":47,"word":"KotoshuWasm","count":2},{"file":"docs/STABILITY-1.0.md","line":31,"word":"SARIF","count":1},{"file":"docs/STABILITY-1.0.md","line":9,"word":"Semver","count":1},{"file":"docs/STABILITY-1.0.md","line":117,"word":"autoload","count":1},{"file":"docs/STABILITY-1.0.md","line":116,"word":"autoloads","count":1},{"file":"docs/STABILITY-1.0.md","line":54,"word":"bucketsBytes","count":1},{"file":"docs/STABILITY-1.0.md","line":122,"word":"changelog","count":1},{"file":"docs/STABILITY-1.0.md","line":31,"word":"cli","count":2},{"file":"docs/STABILITY-1.0.md","line":25,"word":"config","count":2},{"file":"docs/STABILITY-1.0.md","line":48,"word":"detectLanguage","count":1},{"file":"docs/STABILITY-1.0.md","line":82,"word":"fasttext","count":1},{"file":"docs/STABILITY-1.0.md","line":44,"word":"ffi","count":2},{"file":"docs/STABILITY-1.0.md","line":37,"word":"gemspec","count":1},{"file":"docs/STABILITY-1.0.md","line":43,"word":"json","count":3},{"file":"docs/STABILITY-1.0.md","line":23,"word":"jsonl","count":2},{"file":"docs/STABILITY-1.0.md","line":11,"word":"kotoshu","count":26},{"file":"docs/STABILITY-1.0.md","line":48,"word":"loadLid","count":1},{"file":"docs/STABILITY-1.0.md","line":48,"word":"loadModel","count":2},{"file":"docs/STABILITY-1.0.md","line":5,"word":"md","count":2},{"file":"docs/STABILITY-1.0.md","line":82,"word":"onnx","count":1},{"file":"docs/STABILITY-1.0.md","line":62,"word":"openapi","count":2},{"file":"docs/STABILITY-1.0.md","line":24,"word":"rb","count":11},{"file":"docs/STABILITY-1.0.md","line":104,"word":"repo","count":1},{"file":"docs/STABILITY-1.0.md","line":48,"word":"rerank","count":1},{"file":"docs/STABILITY-1.0.md","line":77,"word":"sarif","count":1},{"file":"docs/STABILITY-1.0.md","line":85,"word":"schemas","count":1},{"file":"docs/STABILITY-1.0.md","line":48,"word":"semanticSuggest","count":1},{"file":"docs/STABILITY-1.0.md","line":44,"word":"src","count":2},{"file":"docs/STABILITY-1.0.md","line":119,"word":"stderr","count":1},{"file":"docs/STABILITY-1.0.md","line":27,"word":"subclasses","count":1},{"file":"docs/STABILITY-1.0.md","line":110,"word":"url","count":3},{"file":"docs/STABILITY-1.0.md","line":41,"word":"wasm","count":5},{"file":"docs/STABILITY-1.0.md","line":62,"word":"yaml","count":2},{"file":"docs/STABILITY-1.0.md","line":76,"word":"yml","count":3},{"file":"docs/cache-architecture.md","line":378,"word":"'","count":1},{"file":"docs/cache-architecture.md","line":428,"word":"'Download","count":1},{"file":"docs/cache-architecture.md","line":428,"word":"'download","count":1},{"file":"docs/cache-architecture.md","line":188,"word":"'en","count":10},{"file":"docs/cache-architecture.md","line":433,"word":"'mc'","count":2},{"file":"docs/cache-architecture.md","line":433,"word":"'mycache'","count":2},{"file":"docs/cache-architecture.md","line":389,"word":"'wb'","count":1},{"file":"docs/cache-architecture.md","line":10,"word":"BaseCache","count":8},{"file":"docs/cache-architecture.md","line":364,"word":"CDN","count":2},{"file":"docs/cache-architecture.md","line":46,"word":"CLI","count":4},{"file":"docs/cache-architecture.md","line":40,"word":"EditDistance","count":1},{"file":"docs/cache-architecture.md","line":54,"word":"Embeddings","count":1},{"file":"docs/cache-architecture.md","line":128,"word":"FastText","count":7},{"file":"docs/cache-architecture.md","line":27,"word":"FrequencyCache","count":5},{"file":"docs/cache-architecture.md","line":428,"word":"GitHub'","count":1},{"file":"docs/cache-architecture.md","line":393,"word":"GzipReader","count":1},{"file":"docs/cache-architecture.md","line":180,"word":"IntegrityError","count":1},{"file":"docs/cache-architecture.md","line":246,"word":"JSON","count":5},{"file":"docs/cache-architecture.md","line":170,"word":"KOTOSHU","count":1},{"file":"docs/cache-architecture.md","line":1,"word":"Kotoshu","count":7},{"file":"docs/cache-architecture.md","line":25,"word":"LanguageCache","count":4},{"file":"docs/cache-architecture.md","line":183,"word":"Lifecycle","count":1},{"file":"docs/cache-architecture.md","line":26,"word":"ModelCache","count":7},{"file":"docs/cache-architecture.md","line":314,"word":"MyCache","count":2},{"file":"docs/cache-architecture.md","line":131,"word":"ONNX","count":3},{"file":"docs/cache-architecture.md","line":17,"word":"OOP","count":1},{"file":"docs/cache-architecture.md","line":279,"word":"ParserError","count":1},{"file":"docs/cache-architecture.md","line":500,"word":"Pre","count":1},{"file":"docs/cache-architecture.md","line":428,"word":"RESOURCE'","count":1},{"file":"docs/cache-architecture.md","line":457,"word":"RSpec","count":1},{"file":"docs/cache-architecture.md","line":172,"word":"ResourceNotSetupError","count":1},{"file":"docs/cache-architecture.md","line":278,"word":"RuntimeError","count":1},{"file":"docs/cache-architecture.md","line":166,"word":"SHA","count":4},{"file":"docs/cache-architecture.md","line":40,"word":"SpellChecker","count":1},{"file":"docs/cache-architecture.md","line":5,"word":"TTL","count":6},{"file":"docs/cache-architecture.md","line":388,"word":"URI","count":1},{"file":"docs/cache-architecture.md","line":393,"word":"Zlib","count":1},{"file":"docs/cache-architecture.md","line":115,"word":"aff","count":3},{"file":"docs/cache-architecture.md","line":117,"word":"ar","count":1},{"file":"docs/cache-architecture.md","line":501,"word":"backends","count":1},{"file":"docs/cache-architecture.md","line":166,"word":"bytes'","count":2},{"file":"docs/cache-architecture.md","line":425,"word":"cli","count":1},{"file":"docs/cache-architecture.md","line":118,"word":"de","count":3},{"file":"docs/cache-architecture.md","line":428,"word":"desc","count":1},{"file":"docs/cache-architecture.md","line":12,"word":"deserialization","count":1},{"file":"docs/cache-architecture.md","line":76,"word":"dest","count":12},{"file":"docs/cache-architecture.md","line":114,"word":"dic","count":3},{"file":"docs/cache-architecture.md","line":79,"word":"dir","count":2},{"file":"docs/cache-architecture.md","line":365,"word":"dl","count":3},{"file":"docs/cache-architecture.md","line":56,"word":"fasttext","count":15},{"file":"docs/cache-architecture.md","line":365,"word":"fbaipublicfiles","count":3},{"file":"docs/cache-architecture.md","line":251,"word":"githubusercontent","count":1},{"file":"docs/cache-architecture.md","line":386,"word":"gz","count":9},{"file":"docs/cache-architecture.md","line":378,"word":"gz'","count":1},{"file":"docs/cache-architecture.md","line":377,"word":"gzip","count":2},{"file":"docs/cache-architecture.md","line":406,"word":"gzipped","count":1},{"file":"docs/cache-architecture.md","line":251,"word":"https","count":4},{"file":"docs/cache-architecture.md","line":113,"word":"json","count":14},{"file":"docs/cache-architecture.md","line":263,"word":"kelly","count":1},{"file":"docs/cache-architecture.md","line":46,"word":"kotoshu","count":14},{"file":"docs/cache-architecture.md","line":136,"word":"lang","count":3},{"file":"docs/cache-architecture.md","line":501,"word":"memcached","count":1},{"file":"docs/cache-architecture.md","line":434,"word":"mycache","count":2},{"file":"docs/cache-architecture.md","line":57,"word":"onnx","count":14},{"file":"docs/cache-architecture.md","line":425,"word":"rb","count":5},{"file":"docs/cache-architecture.md","line":117,"word":"ru","count":3},{"file":"docs/cache-architecture.md","line":188,"word":"spelling'","count":10},{"file":"docs/cache-architecture.md","line":75,"word":"subclasses","count":2},{"file":"docs/cache-architecture.md","line":101,"word":"ttl","count":3},{"file":"docs/cache-architecture.md","line":168,"word":"unparsable","count":1},{"file":"docs/cache-architecture.md","line":388,"word":"uri","count":2},{"file":"docs/cache-architecture.md","line":86,"word":"url","count":14},{"file":"docs/cache-architecture.md","line":179,"word":"urls","count":2},{"file":"docs/cache-architecture.md","line":406,"word":"vec","count":2},{"file":"docs/verification-evidence/correct3/pr_body.md","line":12,"word":"'","count":4},{"file":"docs/verification-evidence/correct3/pr_body.md","line":19,"word":"AffReader","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":26,"word":"CLI","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":19,"word":"COMPOUNDRULE","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":19,"word":"CompoundRule","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":37,"word":"RBS","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":17,"word":"RegexpError","count":2},{"file":"docs/verification-evidence/correct3/pr_body.md","line":11,"word":"Tokenizer","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":17,"word":"aff","count":2},{"file":"docs/verification-evidence/correct3/pr_body.md","line":14,"word":"config","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":15,"word":"de","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":7,"word":"el","count":3},{"file":"docs/verification-evidence/correct3/pr_body.md","line":7,"word":"kotoshu","count":3},{"file":"docs/verification-evidence/correct3/pr_body.md","line":28,"word":"lang","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":25,"word":"rubocop","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":19,"word":"sv","count":3},{"file":"docs/verification-evidence/correct3/pr_body.md","line":14,"word":"tokenizer","count":3},{"file":"docs/verification-evidence/correct3/pr_body.md","line":11,"word":"tokenizers","count":2},{"file":"docs/verification-evidence/correct3/pr_body.md","line":15,"word":"whitespace","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":31,"word":"xqzz","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":32,"word":"xxqqz","count":1},{"file":"docs/verification-evidence/correct3/pr_body.md","line":33,"word":"zzqqxvv","count":1}]}
|
data/CHANGELOG.md
CHANGED
|
@@ -8,6 +8,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
10
|
### Added
|
|
11
|
+
- **Korean and Nepali full-feature modules** (plan 108) - the last
|
|
12
|
+
big-population languages without modules move from the plan 107
|
|
13
|
+
basic tier to full feature. `ko` ships the Hangul tokenizer
|
|
14
|
+
(an eojeol is a run of Hangul syllables U+AC00-D7A3 plus bare jamo
|
|
15
|
+
U+1100-11FF/U+3130-318F that escaped IME composition) and the
|
|
16
|
+
Dubeolsik keyboard grid keyed on jamo per the KS X 5002 standard 2-set
|
|
17
|
+
layout, so typo proximity lands on the unit vowel mergers and
|
|
18
|
+
fortis/lenis confusions operate on. `ne` ships the Devanagari
|
|
19
|
+
tokenizer (matras U+093E-094C, the virama U+094D and vowel signs
|
|
20
|
+
stay attached to the base consonant; the danda and Devanagari
|
|
21
|
+
digits are separators) and the Devanagari InScript grid (the Indian
|
|
22
|
+
national standard, mirrored from the models repo eval harness and
|
|
23
|
+
drift-checked against it; the Nepali Traditional Romanized layout
|
|
24
|
+
is more common on the ground but is not a documented standard).
|
|
25
|
+
Both dictionaries were already staged, so `kotoshu setup ko ne`
|
|
26
|
+
serves them; full-feature languages grow 33 to 35.
|
|
27
|
+
- **Basic support for every staged dictionary language** (plan 107) -
|
|
28
|
+
`kotoshu setup` now serves all 95 languages staged in the
|
|
29
|
+
kotoshu/dictionaries manifest, not just the ones with gem modules.
|
|
30
|
+
`LanguageCache::AVAILABLE_LANGUAGES` derives from a vendored
|
|
31
|
+
snapshot of the manifest (Cache::StagedLanguages, refreshed by
|
|
32
|
+
`rake kotoshu:staged_languages:sync`), keeping the hot path
|
|
33
|
+
network-free. Languages without a module run on the basic tier:
|
|
34
|
+
script classification (Language::Script) picks the tokenizer whose
|
|
35
|
+
word regex drives check/suggest word extraction (shared Latin /
|
|
36
|
+
Cyrillic / Greek tokenizers plus a generic regex tokenizer for
|
|
37
|
+
Arabic, Hebrew, Armenian, Georgian, Devanagari and Hangul), and the
|
|
38
|
+
keyboard registry falls back to the script's standard grid (JCUKEN
|
|
39
|
+
for Cyrillic, Arabic 101, Hebrew SI-1452, QWERTY otherwise) instead
|
|
40
|
+
of QWERTY for everything. Two documented tiers: full feature =
|
|
41
|
+
module (registered keyboard + verified specimens,
|
|
42
|
+
`LanguageCache.full_feature_languages`), basic = manifest-present.
|
|
43
|
+
`nn` (Norwegian Nynorsk) is wired as a full-feature module while
|
|
44
|
+
here - dictionary and model are both staged. Modules keep winning
|
|
45
|
+
over the fallbacks; the 32 existing modules behave identically
|
|
46
|
+
(conformance vectors untouched).
|
|
11
47
|
- **Personal dictionary in the check path** (plan 105) -
|
|
12
48
|
`Spellchecker#check` (and therefore `Kotoshu.check`,
|
|
13
49
|
`Kotoshu.check_file`, and `kotoshu check`) no longer flags words
|
data/README.adoc
CHANGED
|
@@ -17,7 +17,7 @@ directory layout, SHA-256 integrity verification, SARIF output, an
|
|
|
17
17
|
* `Kotoshu.check(text, language: "en")` / `Kotoshu.suggest("helo")` — full document check and suggestions
|
|
18
18
|
* `Kotoshu.spellchecker_for(lang)` — per-language checker resolved from the cache (`strict:` is a `Kotoshu.setup` option)
|
|
19
19
|
* `kotoshu check FILE` CLI with these flags:
|
|
20
|
-
** `--language
|
|
20
|
+
** `--language LANG|auto` — any set-up language code, script variants included (`sr-Latn`) (default: `auto`)
|
|
21
21
|
** `--format text|json|sarif`
|
|
22
22
|
** `--offline` — use only cached resources, never download
|
|
23
23
|
** `--strict` — exit 3 if any optional resource (frequency, model) can't load
|
|
@@ -218,7 +218,13 @@ typo-heavy text.
|
|
|
218
218
|
[[multi-language]]
|
|
219
219
|
== Multi-Language Support
|
|
220
220
|
|
|
221
|
-
Kotoshu
|
|
221
|
+
Kotoshu serves every language staged in the dictionaries manifest —
|
|
222
|
+
36 full-feature codes with per-language modules (script-aware
|
|
223
|
+
tokenizer, national keyboard grid, engine-verified specimens) and the
|
|
224
|
+
script-fallback basic tier for the rest; see the
|
|
225
|
+
<<language-matrix, Language Support Matrix>> for the breakdown.
|
|
226
|
+
Semantic reranking adds an ONNX model where the models registry pins
|
|
227
|
+
one (50+ languages), for example:
|
|
222
228
|
|
|
223
229
|
* *de* - German (Deutsch)
|
|
224
230
|
* *en* - English
|
|
@@ -937,28 +943,29 @@ script:
|
|
|
937
943
|
kotoshu completions languages
|
|
938
944
|
----
|
|
939
945
|
|
|
946
|
+
[[language-matrix]]
|
|
940
947
|
=== Language Support Matrix
|
|
941
948
|
|
|
942
|
-
Kotoshu
|
|
949
|
+
Kotoshu serves languages in tiers. Every language staged in the
|
|
950
|
+
kotoshu/dictionaries manifest downloads through `kotoshu setup` and
|
|
951
|
+
checks through the Hunspell engine. Region variants fold onto their
|
|
952
|
+
base dictionary (`en-GB` resolves `en`, `es-MX` resolves `es`);
|
|
953
|
+
script variants keep a dictionary of their own (`sr-Latn` stages the
|
|
954
|
+
Latin Serbian dictionary alongside the Cyrillic `sr`).
|
|
943
955
|
|
|
944
|
-
.
|
|
945
|
-
[width="100%",cols="^
|
|
956
|
+
.Language Support Matrix
|
|
957
|
+
[width="100%",cols="^2,^3,^3",options="header"]
|
|
946
958
|
|=========================================
|
|
947
|
-
|
|
|
959
|
+
| Tier | Languages | What it carries |
|
|
948
960
|
|=========================================
|
|
949
|
-
|
|
|
950
|
-
|
|
|
951
|
-
|
|
|
952
|
-
|
|
953
|
-
|
|
954
|
-
|
|
955
|
-
|
|
956
|
-
|
|
957
|
-
| el (Greek) | ✗ | ✗ | ✓ | ✗ | ✗ | Kelly frequency only |
|
|
958
|
-
| it (Italian) | ✗ | ✗ | ✓ | ✗ | ✗ | Kelly frequency only |
|
|
959
|
-
| no (Norwegian) | ✗ | ✗ | ✓ | ✗ | ✗ | Kelly frequency only |
|
|
960
|
-
| sv (Swedish) | ✗ | ✗ | ✓ | ✗ | ✗ | Kelly frequency only |
|
|
961
|
-
|=========================================]
|
|
961
|
+
| Full feature (35 languages, 36 codes) | ar bg ca cs da de el en es et fa fr he hr hu id it ko lt lv nb ne nl nn pl pt ro ru sk sl sr sr-Latn sv tr uk vi | Per-language module: staged dictionary + Hunspell affix rules, script-aware tokenizer, national keyboard grid for typo-proximity suggestions, engine-verified specimens |
|
|
962
|
+
| Basic (26 languages) | br cy eo eu fo fur fy ga gd gl hy hyw ia ie is ka la lb ltg mk mn nds oc rw tk tlh | Staged dictionary + script-fallback word extraction and the generic keyboard |
|
|
963
|
+
| Kelly frequency (8 languages) | ar el en it no ru sv zh | Top-1000 frequency list wired into suggestion ranking, on top of either tier |
|
|
964
|
+
|=========================================+
|
|
965
|
+
|
|
966
|
+
ONNX semantic models (opt-in, `--model`) resolve per the registry the
|
|
967
|
+
link:https://github.com/kotoshu/models-fasttext-onnx[models repo]
|
|
968
|
+
pins — 50+ languages across the tiers, in full/fluency/mini sizes.
|
|
962
969
|
|
|
963
970
|
.Dictionary Sources
|
|
964
971
|
[width="100%",cols="^1,^1,^1,^1",options="header"]
|
|
@@ -973,6 +980,10 @@ Kotoshu provides multi-language support with varying feature availability.
|
|
|
973
980
|
| ru (Russian) | 146,269 | BSD-style | Alexander Lebedev |
|
|
974
981
|
|=========================================+
|
|
975
982
|
|
|
983
|
+
Staged dictionaries for the other tiers carry their provenance in the
|
|
984
|
+
dictionaries repo manifest (`LICENSE` and source per file; LibreOffice
|
|
985
|
+
/ wooorm-dictionaries derivations dominate).
|
|
986
|
+
|
|
976
987
|
.Kelly Frequency Lists
|
|
977
988
|
[width="100%",cols="^1,^1,^1",options="header"]
|
|
978
989
|
|=========================================
|
data/Rakefile
CHANGED
|
@@ -32,6 +32,41 @@ report_divergences = lambda do |divergences|
|
|
|
32
32
|
end
|
|
33
33
|
|
|
34
34
|
namespace :kotoshu do
|
|
35
|
+
namespace :staged_languages do
|
|
36
|
+
desc <<~DESC
|
|
37
|
+
Regenerate the vendored staged-language list (plan 107). Fetches \
|
|
38
|
+
the dictionaries manifest at the SourceRegistry pin, extracts the \
|
|
39
|
+
distinct language codes from its resources, and rewrites \
|
|
40
|
+
lib/kotoshu/cache/staged_languages.rb in place so \
|
|
41
|
+
LanguageCache::AVAILABLE_LANGUAGES tracks the manifest. Run after \
|
|
42
|
+
the dictionaries repo stages new languages.
|
|
43
|
+
DESC
|
|
44
|
+
task :sync do
|
|
45
|
+
require "kotoshu"
|
|
46
|
+
require "json"
|
|
47
|
+
|
|
48
|
+
url = Kotoshu::SourceRegistry.new.url_for(:dict_manifest)
|
|
49
|
+
puts "Fetching #{url}"
|
|
50
|
+
bytes = Kotoshu::Integrity::NetHTTP.get(url)
|
|
51
|
+
abort "manifest not found at #{url}" if bytes.nil?
|
|
52
|
+
|
|
53
|
+
manifest = JSON.parse(bytes)
|
|
54
|
+
codes = manifest.fetch("resources").values.map { |r| r.fetch("language") }.uniq.sort
|
|
55
|
+
abort "manifest carried no languages" if codes.empty?
|
|
56
|
+
|
|
57
|
+
target = File.expand_path("lib/kotoshu/cache/staged_languages.rb", __dir__)
|
|
58
|
+
source = File.read(target)
|
|
59
|
+
updated = source.sub(/CODES = %w\[\n(?: {8}\S+(?: \S+)*\n)+ {6}\]\.freeze/) do
|
|
60
|
+
wrapped = codes.each_slice(10).map { |slice| " #{slice.join(' ')}" }.join("\n")
|
|
61
|
+
"CODES = %w[\n#{wrapped}\n ].freeze"
|
|
62
|
+
end
|
|
63
|
+
abort "CODES block not found in #{target}" if updated == source
|
|
64
|
+
|
|
65
|
+
File.write(target, updated)
|
|
66
|
+
puts "Rewrote #{target}: #{codes.size} languages (was #{Kotoshu::Cache::StagedLanguages::CODES.size})"
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
|
|
35
70
|
namespace :conformance do
|
|
36
71
|
desc <<~DESC
|
|
37
72
|
Export conformance vectors for kotoshu-rs (plan 67 M3). Walks the Spylls \
|
data/TODO.impl/00-vision.md
CHANGED
|
@@ -262,3 +262,20 @@ yet passing bucket bytes to loadModel.
|
|
|
262
262
|
| 106 | [Native LID detection](106-native-lid-detection.md) | detect is a 7-language heuristic; the pure-Rust lid reader exists unused |
|
|
263
263
|
| 107 | models [09 bucket expansion](../../models-fasttext-onnx/TODO.impl/09-bucket-table-expansion.md) | bucket OOV siblings exist only for en and de |
|
|
264
264
|
| — | site 11 playground buckets | wasm 0.4.0 accepts bucketsBytes; the worker never fetches the sibling |
|
|
265
|
+
|
|
266
|
+
|
|
267
|
+
Plans 107-109 are **wave 8, reachability and the road to 1.0**
|
|
268
|
+
(2026-09-07): after wave 7 (personal dict in check, 176-language
|
|
269
|
+
detection, buckets rolling to all languages), the remaining user gaps
|
|
270
|
+
are the sixty-plus staged dictionaries unreachable without a module,
|
|
271
|
+
the last big-population languages (ko, ne), and the absence of a
|
|
272
|
+
stability commitment adopters can build on.
|
|
273
|
+
|
|
274
|
+
| # | Plan | Gap it closes |
|
|
275
|
+
|---|---|---|
|
|
276
|
+
| 107 | [Basic support for every staged language](107-basic-support-every-staged-language.md) | setup rejects 60+ staged languages; modules become upgrades, not gates |
|
|
277
|
+
| 108 | [ko and ne full-feature](108-ko-ne-full-feature.md) | Hangul and Devanagari - the last ~100M speakers without support |
|
|
278
|
+
| 109 | [The 1.0 readiness audit](109-ten-audit.md) | no frozen surface or stability policy for adopters |
|
|
279
|
+
|
|
280
|
+
Sequencing: 108 after 107 (same area); 109 is analysis-only and runs in
|
|
281
|
+
parallel. The 1.0 cut itself remains the owner's version call.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Plan 107 — Basic support for every staged dictionary language
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
`kotoshu setup` only serves the 32 languages with modules. Sixty-plus staged
|
|
5
|
+
dictionaries (from gd to ia) are unreachable: a Welsh or Icelandic user
|
|
6
|
+
cannot use the gem at all, even though the dictionary and model are
|
|
7
|
+
downloadable. Modules should be the UPGRADE (keyboard + verified specimens),
|
|
8
|
+
not the gate.
|
|
9
|
+
|
|
10
|
+
## Work (gem repo)
|
|
11
|
+
1. `LanguageCache::AVAILABLE_LANGUAGES` derives from the dictionaries
|
|
12
|
+
manifest (the source the registry/staging uses) instead of the hardcoded
|
|
13
|
+
module list; modules register themselves as full-feature upgrades.
|
|
14
|
+
2. Default behavior for module-less languages: script-appropriate tokenizer
|
|
15
|
+
(reuse the script-regex machinery from plan 91), generic keyboard
|
|
16
|
+
fallback (QWERTY for Latin, JCUKEN Cyrillic, Arabic 101 Arabic-script),
|
|
17
|
+
setup/check/suggest all functional.
|
|
18
|
+
3. Wire nn (model + dictionary exist) as a full-feature module while here;
|
|
19
|
+
note sr-Latn as follow-up if the Latin mapping needs product sign-off.
|
|
20
|
+
4. Specs: setup + specimen round trip on 3 module-less staged languages
|
|
21
|
+
(e.g. is, cy, gd); full-feature languages behave identically to today;
|
|
22
|
+
conformance untouched.
|
|
23
|
+
|
|
24
|
+
## Verification
|
|
25
|
+
Live `kotoshu setup is && check -l is` round trip; suite green; the 32
|
|
26
|
+
modules' outputs byte-identical.
|
|
27
|
+
|
|
28
|
+
## Status
|
|
29
|
+
Executed 2026-09-07 — AVAILABLE_LANGUAGES derives from a vendored
|
|
30
|
+
snapshot of the dictionaries manifest (Cache::StagedLanguages, 95
|
|
31
|
+
languages, refreshed by `rake kotoshu:staged_languages:sync`);
|
|
32
|
+
module-less languages ride Language::Script (script tokenizer for
|
|
33
|
+
word extraction: shared Latin/Cyrillic/Greek classes plus a generic
|
|
34
|
+
regex tokenizer for Arabic/Hebrew/Armenian/Georgian/Devanagari/
|
|
35
|
+
Hangul) and the script keyboard default (JCUKEN Cyrillic, Arabic 101,
|
|
36
|
+
Hebrew SI-1452, QWERTY Latin). Tiers documented: full feature =
|
|
37
|
+
module (LanguageCache.full_feature_languages, 33 with nn); basic =
|
|
38
|
+
manifest-present. nn wired as a full-feature module. Specimens
|
|
39
|
+
engine-verified for is cy gd (basic) + nn; live round trips
|
|
40
|
+
setup+check -l for is cy gd nn; conformance 2630 vectors 0 failures;
|
|
41
|
+
suite 3948 examples 0 failures. sr-Latn stays follow-up (normalize
|
|
42
|
+
maps it to sr; needs product sign-off).
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# Plan 108 — Korean and Nepali: the last big-population languages
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
ko (~82M) and ne (~16M) were deferred from batch 3 because Hangul jamo
|
|
5
|
+
input and Devanagari conjuncts need ja-style tokenizer treatment. They are
|
|
6
|
+
the largest populations without full-feature support; both have staged
|
|
7
|
+
dictionaries and models.
|
|
8
|
+
|
|
9
|
+
## Work (gem repo; sequenced AFTER plan 107 merges — same area)
|
|
10
|
+
1. ko: Hangul tokenizer (syllable-block aware, jamo-level word regex so
|
|
11
|
+
check extracts real eojeol), Korean keyboard grid (Dubeolsik 2-set),
|
|
12
|
+
module + specimens engine-verified.
|
|
13
|
+
2. ne: Devanagari tokenizer (matras and conjuncts stay attached to the
|
|
14
|
+
base consonant — the Unicode grapheme rules the script needs), Roman
|
|
15
|
+
keyboard fallback or a Nepali traditional grid if defensibly sourced,
|
|
16
|
+
module + specimens.
|
|
17
|
+
3. AVAILABLE_LANGUAGES + site-flip deliverables (specimen table, sizes).
|
|
18
|
+
|
|
19
|
+
## Verification
|
|
20
|
+
Same discipline as batch 3: every specimen engine-verified through
|
|
21
|
+
correct?/suggest; RTL/complex-script sentence round trips through
|
|
22
|
+
Kotoshu.check; suite green.
|
|
23
|
+
|
|
24
|
+
## Status
|
|
25
|
+
Executed 2026-09-07 — ko: HangulTokenizer (eojeol = run of syllables
|
|
26
|
+
U+AC00-D7A3 plus bare jamo U+1100-11FF/U+3130-318F), Dubeolsik grid
|
|
27
|
+
keyed on jamo per KS X 5002 (the eval harness models ko slips as
|
|
28
|
+
jamo confusion pairs, not a key grid, so no drift check applies),
|
|
29
|
+
module + 7 engine-verified specimens. ne: DevanagariTokenizer
|
|
30
|
+
(matras and virama conjuncts attached to the base consonant; danda
|
|
31
|
+
and Devanagari digits are separators), Devanagari InScript grid
|
|
32
|
+
mirrored from the eval harness _NE_INSCRIPT and drift-checked (the
|
|
33
|
+
defensible standard — Traditional Romanized is common but
|
|
34
|
+
undocumented), module + 3 specimens. Full-feature languages 33 -> 35.
|
|
35
|
+
Live round trips setup + check -l ko ne recorded; conformance 2630
|
|
36
|
+
vectors 0 failures; suite 3985 examples 0 failures. Specimen table
|
|
37
|
+
and dictionary sizes in the PR body for the site flip.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Plan 109 — The 1.0 readiness audit
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Adopters hesitate at 0.x. A 1.0 cut needs a frozen public surface and a
|
|
5
|
+
stated policy. The audit inventories what 1.0 would freeze and what it
|
|
6
|
+
would deprecate — it does NOT cut versions (owner decision).
|
|
7
|
+
|
|
8
|
+
## Work (gem repo docs/, analysis only)
|
|
9
|
+
1. Inventory the public surface: Ruby API (Kotoshu module functions,
|
|
10
|
+
Spellchecker, Configuration, Tasks, validators), CLI flags, wasm JS
|
|
11
|
+
surface (KotoshuWasm, loadModel/rerank/semanticSuggest/loadLid/
|
|
12
|
+
detectLanguage), HTTP contract (openapi.yaml), action inputs, registry
|
|
13
|
+
schema.
|
|
14
|
+
2. For each: stable / needs-deprecation-notice / internal-accidentally-
|
|
15
|
+
public. Flag anything accidentally public that 1.0 should privatize.
|
|
16
|
+
3. Draft the stability policy (semver commitments, deprecation window,
|
|
17
|
+
conformance vectors as the behavioral contract) as docs/STABILITY.md.
|
|
18
|
+
4. A 1.0 checklist: what must land first (nothing mechanical is known to
|
|
19
|
+
block), what the release train looks like (gem+wasm+server together).
|
|
20
|
+
|
|
21
|
+
## Verification
|
|
22
|
+
Every inventory line cites the file that defines it; the deprecation list
|
|
23
|
+
reviewed against gemspec/exported constants.
|
|
24
|
+
|
|
25
|
+
## Status
|
|
26
|
+
Pending
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Plan 110 — sr-Latn wiring and the long-tail polish list
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Plan 107 wired nn and deferred sr-Latn (the Latin script variant of Serbian)
|
|
5
|
+
as follow-up. With the basic tier live, this is the last named language gap
|
|
6
|
+
in the corpus; everything after it is small-polish.
|
|
7
|
+
|
|
8
|
+
## Work (gem repo)
|
|
9
|
+
1. sr-Latn: register as a full-feature module reusing the Serbian dictionary
|
|
10
|
+
with the Latin tokenizer + the Croatian/Slovenian QWERTZ grid (the
|
|
11
|
+
scripts share the Latin layout), or map sr-Latn -> the staged sr-Latn
|
|
12
|
+
dictionary if one exists in the manifest — follow the data, document the
|
|
13
|
+
choice. Specimens engine-verified.
|
|
14
|
+
2. Polish sweep: any spellcheck_word_regex script set lacking a staged
|
|
15
|
+
language (inventory + one-line table in the PR body); README language
|
|
16
|
+
table refresh to the 35+ tiers.
|
|
17
|
+
3. Frequency-list inventory (no sourcing): list which staged languages have
|
|
18
|
+
published frequency data wired vs not — record only, sourcing is a
|
|
19
|
+
separate owner decision.
|
|
20
|
+
|
|
21
|
+
## Verification
|
|
22
|
+
Live setup + check -l sr-Latn round trip; suite green; conformance
|
|
23
|
+
untouched.
|
|
24
|
+
|
|
25
|
+
## Status
|
|
26
|
+
Executed 2026-09-07 — the data chose the distinct staged dictionary:
|
|
27
|
+
sr-Latn is a full-feature module (Languages::SerbianLatin, Latin
|
|
28
|
+
tokenizer, Serbian-Latin QWERTZ grid sharing the hrsl key table), and
|
|
29
|
+
the sr-Latn code now survives ResourceManager normalization
|
|
30
|
+
(script subtags kept, region subtags still fold) so the staged Latin
|
|
31
|
+
files are reachable — previously sr-Latn collapsed onto the Cyrillic
|
|
32
|
+
sr dictionary. Script-set and frequency inventories recorded in the
|
|
33
|
+
PR body (record-only). Suite green, conformance untouched.
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Plan 111 — The embedding bake-off: is fastText still the right model?
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Our gates prove int8 near-lossless RELATIVE to fastText — but fastText is
|
|
5
|
+
2017-era, context-blind, and our own numbers cap intended-top-5 near 0.21
|
|
6
|
+
on hard English typos even with buckets. The eval harness (keyboard-aware
|
|
7
|
+
grids, typo corpora, frozen gates) is the asset; use it to test successors.
|
|
8
|
+
|
|
9
|
+
## Work (models repo; eval only — no registry release without a verdict)
|
|
10
|
+
1. Baseline: current mini/fluency+buckets through the existing gates.
|
|
11
|
+
2. Candidate B: fastText + small cross-encoder rerank over (context,
|
|
12
|
+
candidate) pairs — ONNX-exportable distill class, 15-30 MB int8.
|
|
13
|
+
3. Candidate C: purpose-trained typo bi-encoder over the vendored
|
|
14
|
+
github-typo-corpus (train/eval split strictly by repo).
|
|
15
|
+
4. Candidate D: ModernBERT-class small encoder, same size budget.
|
|
16
|
+
5. Report: gates table per candidate per language (en de ru es minimum),
|
|
17
|
+
size, latency-per-suggest, and a ship/reject verdict with the same
|
|
18
|
+
discipline as int4 (reject on data is a complete outcome).
|
|
19
|
+
|
|
20
|
+
## Verification
|
|
21
|
+
Every number from the existing harness, unweakened; frozen corpora; no
|
|
22
|
+
registry changes in this plan.
|
|
23
|
+
|
|
24
|
+
## Status
|
|
25
|
+
Executed 2026-09-09 — models PR #25 merged. VERDICT: fastText stays.
|
|
26
|
+
B (MiniLM cross-encoder rerank) REJECT — loses 4/4 languages (en 0.194 vs
|
|
27
|
+
fluency 0.252 top-5; slate recall caps en at 0.43; context ablation
|
|
28
|
+
negative). D (ModernBERT-base) REJECT — representation failure (top-5
|
|
29
|
+
0.03-0.24), 5x size cap. C (0.5 MB char-BiGRU typo bi-encoder, 5.2
|
|
30
|
+
ms/suggest) REJECT as drop-in but PROMISING — beats the full tier on the
|
|
31
|
+
strict repo-clean subset for en and de; open thread needing non-en data,
|
|
32
|
+
a bigger clean bench, and a top-1 story. Full ladders in
|
|
33
|
+
eval/reports/bakeoff-v1.md.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Plan 112 — Deployability and CI gating
|
|
2
|
+
|
|
3
|
+
## Work (kotoshu-rs; site rides a follow-up)
|
|
4
|
+
1. Prebuilt worker on npm: @kotoshu/worker exposing the playground's
|
|
5
|
+
engine worker (load/check/suggest-batch/semantic/detect protocol) so
|
|
6
|
+
JS users embed the engine without reimplementing the worker; versioned
|
|
7
|
+
with the wasm package.
|
|
8
|
+
2. CI latency gate: the multi-language sweep bench (rs PR #23 harness)
|
|
9
|
+
runs on CI with budgets (en < 120 ms, pt < 700 ms avg) failing red.
|
|
10
|
+
3. Wasm memory ceiling test: dict + mini tier + buckets resident under a
|
|
11
|
+
stated budget per language class.
|
|
12
|
+
4. Language pack (design note, owner decision): one fetch for
|
|
13
|
+
dict+tier+buckets — additive registry field sketch in the PR body.
|
|
14
|
+
|
|
15
|
+
## Verification
|
|
16
|
+
Node smoke for the worker package; budgets tuned to measured p50s, not
|
|
17
|
+
aspirations; all existing CI green.
|
|
18
|
+
|
|
19
|
+
## Status
|
|
20
|
+
Executed 2026-09-09 — kotoshu-rs PR #28 merged. @kotoshu/worker package
|
|
21
|
+
(createEngine full protocol, ./engine-worker entry, semantic-merge pure
|
|
22
|
+
half; publish pending owner npm registration); latency.yml gate with
|
|
23
|
+
measured budgets (en 56ms avg vs 120 gate, pt 346 vs 700 — 2x headroom);
|
|
24
|
+
wasm32 memory ceilings frozen (en 46.9/64 MB, pt 165/192 MB) with
|
|
25
|
+
behavior probes; language-pack kotoshu://packs/{lang} sketch in the PR
|
|
26
|
+
body awaiting owner decision.
|