kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
# 66 — kotoshu-rs: the Rust core (parsanol blueprint)
|
|
2
|
+
|
|
3
|
+
Implements Pillar 3 of [65-universal-kotoshu.md](65-universal-kotoshu.md).
|
|
4
|
+
**The approach is copied wholesale from `parsanol-rs` + `parsanol-ruby`**
|
|
5
|
+
(Ribose, in production) because it demonstrably works: one pure-Rust
|
|
6
|
+
core, all FFI feature-gated *inside* the core, per-language packages as
|
|
7
|
+
thin cdylib shims, pure-Ruby fallback retained, dual-backend test suite.
|
|
8
|
+
|
|
9
|
+
## Decision: `ort`, not `onnxruntime`
|
|
10
|
+
|
|
11
|
+
- **`ort`** (pykeio/ort, v2.x): actively maintained, tracks current
|
|
12
|
+
onnxruntime, execution providers, and — decisive for us — a
|
|
13
|
+
**`load-dynamic`** mode that dlopens a host-provided
|
|
14
|
+
libonnxruntime instead of linking one in.
|
|
15
|
+
- The `onnxruntime` crate (nbigaouette) is effectively unmaintained and
|
|
16
|
+
pinned to old runtime versions. Rejected.
|
|
17
|
+
- **Policy**: inference is host-injected via an `EmbeddingProvider`
|
|
18
|
+
trait (rerank math in the core works on vectors the host supplies).
|
|
19
|
+
The core's own provider is `feature = "onnx"` →
|
|
20
|
+
`ort = { version = "2", default-features = false, features =
|
|
21
|
+
["load-dynamic"] }`, so Ruby/Python/Node hosts share the lib they
|
|
22
|
+
already ship (the `onnxruntime` gem / pip / onnxruntime-node).
|
|
23
|
+
Standalone Rust builds (CLI, LSP binary) enable ort's bundled
|
|
24
|
+
binaries instead. Minimum onnxruntime 1.x documented per release.
|
|
25
|
+
|
|
26
|
+
### Addendum (2026-09-02): runtime validation + `onnx-ir` assessment
|
|
27
|
+
|
|
28
|
+
- **ort under real load** — Weirich, "Rust, ORT, ONNX: Real-Time YOLO on
|
|
29
|
+
a Live Webcam" (Nov 2025): YOLOv11 at 30 FPS webcam inference in Rust
|
|
30
|
+
via a four-thread pipeline (capture → resize/pack → inference →
|
|
31
|
+
render), ~22% CPU on a MacBook; same expected on a Jetson with
|
|
32
|
+
hardware acceleration. Our workload (single-token embedding lookups,
|
|
33
|
+
rerank of ≤25 candidates) is orders of magnitude lighter — ort
|
|
34
|
+
headroom is not a risk. The article's pipeline shape is also the
|
|
35
|
+
reference pattern for the LSP/server: inference isolated on its own
|
|
36
|
+
worker thread, results serialized back; the editor loop never blocks
|
|
37
|
+
on a model call. ORT's WebAssembly support is confirmed for the
|
|
38
|
+
browser path.
|
|
39
|
+
- **`onnx-ir`** (tracel-ai, Burn ecosystem) is a pure-Rust ONNX
|
|
40
|
+
**parser → IR**, not a runtime: executing the graph means burn-import
|
|
41
|
+
compiling it to a Burn backend — and therefore **zero native
|
|
42
|
+
dependencies**. That matters exactly where `load-dynamic` cannot
|
|
43
|
+
work: static musl CLI binaries and wasm32. FastText-style graphs
|
|
44
|
+
(embedding gather + matmul + normalization) sit comfortably inside
|
|
45
|
+
Burn's operator coverage.
|
|
46
|
+
- **Policy consequence — the trait already anticipated this.** `ort`
|
|
47
|
+
stays THE provider (decided above). At P3, if ort's wasm story proves
|
|
48
|
+
heavy, evaluate a `burn` provider (`onnx-ir`/burn-import) behind the
|
|
49
|
+
same `EmbeddingProvider` trait, gated to `wasm32`/static targets
|
|
50
|
+
only. `tract` (sonos — pure-Rust inference, no C deps) is the noted
|
|
51
|
+
third option if Burn's op coverage ever falls short. One trait,
|
|
52
|
+
three candidate providers, no core changes.
|
|
53
|
+
|
|
54
|
+
## Workspace layout (mirrors parsanol-rs)
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
kotoshu-rs/
|
|
58
|
+
├── Cargo.toml # workspace: lto="fat", codegen-units=1
|
|
59
|
+
├── deny.toml typos.toml release-plz.toml .pre-commit-config.yaml
|
|
60
|
+
├── .cargo/config.toml # [net] git-fetch-with-cli; wasm32 getrandom flags
|
|
61
|
+
├── kotoshu/ # core crate, crate-type = ["rlib"]
|
|
62
|
+
│ └── src/
|
|
63
|
+
│ ├── lib.rs # pub mod ffi; pub use ffi::ruby as ruby_ffi
|
|
64
|
+
│ ├── portable/ # portability layer (parsanol pattern)
|
|
65
|
+
│ ├── dict/ # aff/dic parsing, affix expansion, DAFSA/trie
|
|
66
|
+
│ ├── suggest/ # edit-distance(banded), phonetic, keyboard, ngram
|
|
67
|
+
│ ├── rerank.rs # pure vector math + EmbeddingProvider trait
|
|
68
|
+
│ ├── resource/ # manifest parse, sha256, tier metadata
|
|
69
|
+
│ ├── tokenize.rs
|
|
70
|
+
│ └── ffi/
|
|
71
|
+
│ ├── mod.rs # feature gates + re-exports
|
|
72
|
+
│ ├── shared.rs # batch serialization shared by ALL bindings
|
|
73
|
+
│ ├── c.rs # C ABI — always available (parsanol: c/ always)
|
|
74
|
+
│ ├── ruby/ # feature = "ruby" → magnus
|
|
75
|
+
│ └── wasm/ # feature = "wasm" → wasm-bindgen, js-sys,
|
|
76
|
+
│ # console_error_panic_hook
|
|
77
|
+
├── benches/ examples/ tests/ # criterion, proptest, conformance-vector runner
|
|
78
|
+
└── .github/workflows/
|
|
79
|
+
├── _rust-setup.yml _rust-build-test.yml ci.yml
|
|
80
|
+
├── release.yml # release-plz → crates.io
|
|
81
|
+
├── release-binary.yml # kotoshu CLI / kotoshu-lsp single binaries
|
|
82
|
+
├── ruby-ffi.yml # builds core with feature "ruby", runs magnus tests
|
|
83
|
+
├── wasm.yml # wasm-pack build → publishes @kotoshu/wasm
|
|
84
|
+
└── docs.yml
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
**Features**: `default = []`, `ruby = ["magnus"]`,
|
|
88
|
+
`wasm = ["wasm-bindgen", "js-sys", "console_error_panic_hook"]`,
|
|
89
|
+
`onnx = ["ort/load-dynamic"]`, `parallel = ["rayon"]`, `logging = ["log"]`.
|
|
90
|
+
Core profile `opt-level 3, lto true, codegen-units 1`; dep crates
|
|
91
|
+
`opt-level "s"` — parsanol's exact settings.
|
|
92
|
+
|
|
93
|
+
## Language packages (mirrors parsanol-ruby)
|
|
94
|
+
|
|
95
|
+
| Package | Shim | Notes |
|
|
96
|
+
|---|---|---|
|
|
97
|
+
| `kotoshu` gem | `ext/kotoshu_native` cdylib: `#[magnus::init] fn init(r) { kotoshu::ffi::ruby::init(r) }` | gemspec: `spec.extensions`, runtime dep `rb_sys "~> 0.9"`, files include `Cargo.toml`+`Cargo.lock`, reject `*.so|dylib|bundle|dll`; versions independently of the core (parsanol: gem 1.3.x over core 0.5.x) |
|
|
98
|
+
| `kotoshu-python` | PyO3 shim crate in-repo, maturin wheels | over the same core; HTTP client remains the pure-Python path |
|
|
99
|
+
| `kotoshu-js` | none — re-exports `@kotoshu/wasm` | wasm package built+published from kotoshu-rs (`wasm.yml`); napi-rs native later only if wasm latency hurts |
|
|
100
|
+
| Rust CLI/LSP | direct dep | `release-binary.yml` ships static binaries (linux amd64/arm64, macOS universal, windows x64/arm64) |
|
|
101
|
+
|
|
102
|
+
**extconf.rb (copied verbatim in spirit)**: `require "rb_sys/mkmf"`;
|
|
103
|
+
`create_rust_makefile("kotoshu/kotoshu_native")` with
|
|
104
|
+
`r.profile = ENV.fetch("RB_SYS_CARGO_PROFILE", :dev).to_sym`,
|
|
105
|
+
`r.use_stable_api_compiled_fallback = true`,
|
|
106
|
+
`r.force_install_rust_toolchain = false`. Source gem compiles on
|
|
107
|
+
install — **no cross-compile matrix to maintain**; prebuilt platform
|
|
108
|
+
gems (rake-compiler-dock) only if adoption demands (gate, post-1.0).
|
|
109
|
+
|
|
110
|
+
**Ruby loader/fallback**: `Kotoshu::Native.available?` guard; every
|
|
111
|
+
public API keeps its pure-Ruby path; the ext is a pure accelerator —
|
|
112
|
+
identical results, faster. Env `KOTOSHU_BACKEND=native|ruby|auto`
|
|
113
|
+
selects explicitly for debugging/benching.
|
|
114
|
+
|
|
115
|
+
**Dual-backend conformance**: parsanol's `compat:` namespace, applied to
|
|
116
|
+
the suite we already own —
|
|
117
|
+
`rake compat:ruby` / `compat:native` / `compat:compare` run the gem's
|
|
118
|
+
full rspec suite (Splylls/Hunspell fixtures included) against each
|
|
119
|
+
backend and diff the outputs. CI runs both. Golden vectors from the
|
|
120
|
+
same fixtures are exported into `kotoshu/conformance` and consumed by
|
|
121
|
+
kotoshu-rs CI so the C ABI, wasm build, and future ports are held to
|
|
122
|
+
identical outputs — **one source of truth, three enforcement points.**
|
|
123
|
+
|
|
124
|
+
## Porting order (each step lands with conformance green)
|
|
125
|
+
|
|
126
|
+
1. **P0 scaffold**: workspace, `ffi/shared.rs` batch format, C ABI
|
|
127
|
+
skeleton, conformance-vector runner, CI (incl. `ruby-ffi.yml`,
|
|
128
|
+
`wasm.yml`), release-plz wired.
|
|
129
|
+
2. **P1 dictionary**: aff parser → dic loader → affix expansion →
|
|
130
|
+
lookup (`correct?`). Vectors: the gem's affix/lookup specs.
|
|
131
|
+
3. **P2 suggest**: banded edit-distance with threshold (port the gem's
|
|
132
|
+
optimized DP), phonetic, keyboard proximity, n-gram, composite
|
|
133
|
+
ranking, frequency bonus. Vectors: suggestion specs (ranked lists).
|
|
134
|
+
4. **P3 rerank + resources**: `EmbeddingProvider` trait + ort provider
|
|
135
|
+
(load-dynamic); manifest/sha256/tier parse. Vectors: rerank specs.
|
|
136
|
+
5. **P4 shims**: gem ext (0.9), Python wheel, wasm publish, single-binary
|
|
137
|
+
LSP/CLI.
|
|
138
|
+
|
|
139
|
+
## Parsanol details adopted as policy
|
|
140
|
+
|
|
141
|
+
- `[patch.crates-io]` git-revs for magnus/magnus-macros/rb-sys when
|
|
142
|
+
crates.io lags Ruby (current parsanol revs noted; relax on release).
|
|
143
|
+
- `ffi/shared.rs` batch FFI format — one serialization, all bindings;
|
|
144
|
+
parsanol measured 3–5× over object-by-object FFI.
|
|
145
|
+
- Quality kit: cargo-deny (`deny.toml`), typos, pre-commit, criterion
|
|
146
|
+
benches, proptest, `rust-version` floor per crate.
|
|
147
|
+
- Core repo tests the Ruby feature itself (`ruby-ffi.yml`) so binding
|
|
148
|
+
breakage is caught in the core's CI, not downstream.
|
|
149
|
+
- Separate version lines per package; release-plz automates the core's.
|
|
150
|
+
|
|
151
|
+
## Acceptance
|
|
152
|
+
|
|
153
|
+
- `kotoshu-rs` passes the full conformance-vector pack on the C ABI,
|
|
154
|
+
with `--features ruby` under MRI 3.2/3.3/3.4/4.0-head, and as wasm32.
|
|
155
|
+
- `rake compat:compare` in the gem shows zero diffs between backends.
|
|
156
|
+
- `gem install kotoshu` works WITHOUT a Rust toolchain (pure Ruby) and
|
|
157
|
+
WITH one (native build), same results either way.
|
|
158
|
+
- `@kotoshu/wasm` loads in Node 18+ and browsers; playground runs on it.
|
|
159
|
+
|
|
160
|
+
## Risks
|
|
161
|
+
|
|
162
|
+
Two engines during port (mitigated: dual-backend suite gates every
|
|
163
|
+
merge); Rust toolchain requirement for the native path (mitigated:
|
|
164
|
+
fallback + prebuilt gate); magnus/rb-sys rev churn (patch policy);
|
|
165
|
+
ort dynamic-lib skew (documented minimum, C API stable across 1.x).
|
|
166
|
+
|
|
167
|
+
## Status
|
|
168
|
+
|
|
169
|
+
_Planning._ Promotes into P0–P4 execution plans when cut 0.7 opens.
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
# 67 — Publishing the Stack: Tiered Models, kotoshu-rs, Access Libraries
|
|
2
|
+
|
|
3
|
+
The delivery train for [65-universal-kotoshu.md](65-universal-kotoshu.md):
|
|
4
|
+
turns plans 66 (Rust core), the models-repo plans 06/07 (tiers,
|
|
5
|
+
registry, releases) into a sequenced rollout of **what gets published,
|
|
6
|
+
from which repo, in what order**. Execution-level: each milestone has
|
|
7
|
+
one owner repo, a concrete deliverable, and acceptance.
|
|
8
|
+
|
|
9
|
+
## The train
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
M0 spec freeze ─► M1 gem tier client ─► M2 models release ─┐
|
|
13
|
+
(registry v1) (owner │
|
|
14
|
+
tags) ├─► M5 shims
|
|
15
|
+
M3 kotoshu-rs P0 (bootstrap) ─► M4 P1–P2 port ────────────┘ (ruby/wasm/python)
|
|
16
|
+
│
|
|
17
|
+
▼
|
|
18
|
+
M6 polish: platform gems gate,
|
|
19
|
+
signed bundles, site/docs
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
M0–M2 (models + gem) and M3–M4 (Rust) are **parallel tracks**; they
|
|
23
|
+
join at M5.
|
|
24
|
+
|
|
25
|
+
### M0 — Resource Spec v1 freeze
|
|
26
|
+
|
|
27
|
+
- Deliverable: `schemas/registry.schema.json` merged in
|
|
28
|
+
`models-fasttext-onnx` (plan 07). The registry shape is a cross-repo
|
|
29
|
+
API — freeze it before both the gem client and kotoshu-rs parse it.
|
|
30
|
+
- Acceptance: schema merged; a sample registry validates; id scheme
|
|
31
|
+
`kotoshu://models/{lang}/{tier}` locked.
|
|
32
|
+
|
|
33
|
+
### M1 — Gem: tier-aware ModelCache (PR to this repo)
|
|
34
|
+
|
|
35
|
+
- `Kotoshu.setup(:en, want: :model, tier: :fluency|:mini|:full)`;
|
|
36
|
+
`ModelCache` resolves tier from the registry; fallback chain
|
|
37
|
+
full→fluency→mini only when explicitly requested.
|
|
38
|
+
- Config: `model_tier` option (SCHEMA entry → env `KOTOSHU_MODEL_TIER`
|
|
39
|
+
automatic), CLI `kotoshu setup LANG --model --tier mini`.
|
|
40
|
+
- Resolve stays cache-only; a missing tier raises
|
|
41
|
+
`ResourceNotSetupError` exactly like today. No implicit downloads.
|
|
42
|
+
- Acceptance: specs for tier resolution + fallback; existing specs
|
|
43
|
+
untouched (default tier = owner decision, see gates below).
|
|
44
|
+
|
|
45
|
+
### M2 — First models release (owner action required)
|
|
46
|
+
|
|
47
|
+
- Plan 07 workflow builds the release; **the tag name is the owner's
|
|
48
|
+
decision** — the workflow refuses to invent versions. Default pin in
|
|
49
|
+
the gem follows only after the owner picks.
|
|
50
|
+
|
|
51
|
+
### M3 — kotoshu-rs bootstrap (new repo `kotoshu/kotoshu-rs`, P0 of plan 66)
|
|
52
|
+
|
|
53
|
+
- Repo creation under the org (owner or delegated `gh repo create`).
|
|
54
|
+
- Scaffold per plan 66: workspace (lto fat / codegen-units 1),
|
|
55
|
+
`kotoshu` core crate with `ffi/{mod,shared,c}.rs`, features
|
|
56
|
+
`default=[] ruby wasm onnx parallel logging`, deny/typos/pre-commit,
|
|
57
|
+
CI trio (`ci.yml`, `ruby-ffi.yml`, `wasm.yml`), release-plz config.
|
|
58
|
+
- **Conformance export first**: `rake kotoshu:conformance:export` in
|
|
59
|
+
this repo writes golden JSONL (fixtures + suggestion outputs) to
|
|
60
|
+
`conformance/`; kotoshu-rs CI consumes it — one source of truth,
|
|
61
|
+
three enforcement points (C ABI, ruby feature, wasm).
|
|
62
|
+
- Acceptance: green CI incl. `--features ruby` under one MRI and a
|
|
63
|
+
wasm32 build; C ABI round-trips the batch format on exported vectors.
|
|
64
|
+
|
|
65
|
+
### M4 — Port P1–P2: dictionary + suggest
|
|
66
|
+
|
|
67
|
+
- aff/dic parsing → affix expansion → lookup (P1); banded
|
|
68
|
+
edit-distance, phonetic, keyboard, n-gram, composite ranking,
|
|
69
|
+
frequency bonus (P2). Each lands with conformance green against the
|
|
70
|
+
golden vectors — no "port now, verify later".
|
|
71
|
+
- Acceptance: full conformance pack passes on the C ABI; criterion
|
|
72
|
+
benchmarks published in CI artifacts.
|
|
73
|
+
|
|
74
|
+
### M5 — Access libraries (the publish wave)
|
|
75
|
+
|
|
76
|
+
Order = blast-radius ascending, value descending:
|
|
77
|
+
|
|
78
|
+
1. **Ruby** — `ext/kotoshu_native` in this repo (thin magnus shim per
|
|
79
|
+
plan 66), `KOTOSHU_BACKEND=native|ruby|auto`, `rake compat:ruby|
|
|
80
|
+
native|compare` in CI. The gem keeps installing pure-Ruby with no
|
|
81
|
+
toolchain. PR-gated like every change to this repo.
|
|
82
|
+
2. **JS/WASM** — `@kotoshu/wasm` built+published from kotoshu-rs
|
|
83
|
+
(`wasm.yml`, wasm-pack); `kotoshu-js` re-exports it alongside the
|
|
84
|
+
HTTP client. **npm publish needs the org's npm credentials —
|
|
85
|
+
currently blocked** (same blocker as kotoshu-js itself).
|
|
86
|
+
3. **Python** — maturin wheel over the same core in `kotoshu-python`,
|
|
87
|
+
optional extra alongside the HTTP client. **PyPI publish needs the
|
|
88
|
+
org's token — currently blocked (403)**.
|
|
89
|
+
4. **Go** — stays on the HTTP path (kotoshu-server); cgo over the C
|
|
90
|
+
ABI is a deliberate non-goal until demand appears.
|
|
91
|
+
|
|
92
|
+
- Acceptance per shim: conformance vectors pass through the binding;
|
|
93
|
+
perf report vs pure-Ruby/HTTP path; version lines independent
|
|
94
|
+
(parsanol policy).
|
|
95
|
+
|
|
96
|
+
### M6 — Polish
|
|
97
|
+
|
|
98
|
+
- Prebuilt platform gems via rake-compiler-dock **only if** adoption
|
|
99
|
+
demands (gate, post-adoption — plan 66 policy).
|
|
100
|
+
- Signed offline bundles (plan 65 pillar: minisign/cosign over the
|
|
101
|
+
registry + assets) — signature verification in gem and kotoshu-rs.
|
|
102
|
+
- Site: tier badges on language pages, `docs/caching` tier section,
|
|
103
|
+
playground option to run `@kotoshu/wasm` locally once published.
|
|
104
|
+
|
|
105
|
+
## Owner-decision gates (never taken by tooling)
|
|
106
|
+
|
|
107
|
+
| Decision | Milestone |
|
|
108
|
+
|---|---|
|
|
109
|
+
| Registry/tier **default tier** (bandwidth vs accuracy) | M1 — **DECIDED 2026-09-04: `fluency`** (owner instruction: light/medium default; legacy caches bridged) |
|
|
110
|
+
| Models **release tag & versions** | M2, every release |
|
|
111
|
+
| kotoshu-rs **crate versions, MSRV**, release-plz first release | M3+ |
|
|
112
|
+
| Ruby ext gem version; python/js package versions | M5 |
|
|
113
|
+
| License attribution wording (CC BY-SA inheritance) | M2 |
|
|
114
|
+
| gem required_ruby_version changes (if any) | never tooling's call |
|
|
115
|
+
|
|
116
|
+
## Publishing blockers that exist today
|
|
117
|
+
|
|
118
|
+
- PyPI token invalid/absent; npm org auth absent. The M5 publish wave
|
|
119
|
+
for python/js cannot complete until the owner provides them — flag
|
|
120
|
+
early, publish the code (repos, CI) regardless.
|
|
121
|
+
|
|
122
|
+
## Acceptance (whole train)
|
|
123
|
+
|
|
124
|
+
- `gem install kotoshu` → `kotoshu setup en` pulls the fluency tier by
|
|
125
|
+
default (or the owner's chosen default), verified by sha256 from the
|
|
126
|
+
registry, cache-only afterwards.
|
|
127
|
+
- `kotoshu-rs` passes the conformance pack on C ABI, `--features ruby`
|
|
128
|
+
under MRI 3.2–4.0-head, and wasm32.
|
|
129
|
+
- `rake compat:compare` shows zero diffs between Ruby and native
|
|
130
|
+
backends.
|
|
131
|
+
- `@kotoshu/wasm` loads in Node 18+ and browsers; python wheel
|
|
132
|
+
installs without a Rust toolchain (or is clearly marked source-build).
|
|
133
|
+
|
|
134
|
+
## Status
|
|
135
|
+
|
|
136
|
+
_Planning._ Milestones promote to individual execution plans
|
|
137
|
+
(66 P0–P4 for M3–M4; per-shim plans for M5) as each opens.
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# 68 — SOTA Techniques: Research Adoption Plan
|
|
2
|
+
|
|
3
|
+
Mandate: survey 2023–2026 ML/AI and spellcheck research (arxiv, ACL,
|
|
4
|
+
lab reports) and the current frontier model lines, and decide what
|
|
5
|
+
Kotoshu adopts, adapts, or rejects. Research conducted 2026-09-02; the
|
|
6
|
+
full sourced brief lives in this plan's "Findings" section. Companion
|
|
7
|
+
execution context: [65](65-universal-kotoshu.md) (pillars),
|
|
8
|
+
[66](66-kotoshu-core.md) (Rust core, `ort`), [67](67-kotoshu-rs-and-access-libraries.md)
|
|
9
|
+
(delivery train); models-repo plans 06/07 (tiers, registry) already
|
|
10
|
+
shipped eval-gated tiers.
|
|
11
|
+
|
|
12
|
+
## Findings (compressed; URLs are the evidence)
|
|
13
|
+
|
|
14
|
+
### Spelling / GEC state of the art
|
|
15
|
+
|
|
16
|
+
| Approach | Class | BEA-2019 F0.5 | Note |
|
|
17
|
+
|---|---|---|---|
|
|
18
|
+
| GECToR-style tagging + distill | 100–250M | ~74–75 | Still beats prompting on precision-weighted benches — small supervised > LLM prompting for correction ([NLP-progress](http://nlpprogress.com/english/grammatical_error_correction.html), [Grammarly GECToR](https://www.grammarly.com/blog/engineering/experimenting-with-gector/)) |
|
|
19
|
+
| T5/BART seq2seq | 250M–3B | ~73 | Linear-decoding tricks close the gap ([gec-papers](https://github.com/Chunngai/gec-papers)) |
|
|
20
|
+
| Prompted GPT-4-class | API-scale | high-50s–60s | Win on JFLEG *fluency*, lose on precision ([OpenReview](https://openreview.net/forum?id=yAMayChodt)) |
|
|
21
|
+
| Fine-tuned 1–8B LLM | edge-LLM | closing | Overcorrection risk ([arxiv 2509.20811](https://arxiv.org/html/2509.20811v1)) |
|
|
22
|
+
| Masked-LM real-word ranking | small | ~85% acc | The closest published twin to our pipeline: dictionary candidates + neural ranker, with **training-data construction called "the crucial component"** ([arxiv 2410.23514](https://arxiv.org/html/2410.23514v1)) |
|
|
23
|
+
|
|
24
|
+
LLM-as-GEC-evaluator is now standard practice
|
|
25
|
+
([BEA 2024](https://aclanthology.org/2024.bea-1.6/)) — relevant to our
|
|
26
|
+
eval harness, not our runtime.
|
|
27
|
+
|
|
28
|
+
### Embeddings / tiering (directly about our shipped tiers)
|
|
29
|
+
|
|
30
|
+
- **Matryoshka / 2D-MRL**: plain truncation of non-MRL embeddings is
|
|
31
|
+
safe until deep reduction; MRL training only pays beyond that
|
|
32
|
+
([arxiv 2205.13147](https://arxiv.org/abs/2205.13147),
|
|
33
|
+
[arxiv 2605.16608](https://arxiv.org/html/2605.16608v2)). Our tiers
|
|
34
|
+
already are 2D truncation (vocab axis prefix + kept dims) —
|
|
35
|
+
independently validated by 2D-MRL work
|
|
36
|
+
([arxiv 2410.13230](https://arxiv.org/html/2410.13230v1),
|
|
37
|
+
[HF guide](https://huggingface.co/blog/matryoshka)). **A single
|
|
38
|
+
frequency-ordered file with declared truncation points can replace
|
|
39
|
+
three artifacts with no retraining.**
|
|
40
|
+
- **Quantization**: int8 near-lossless (matches our 0.9999 rank_corr);
|
|
41
|
+
int4 with group-wise scales (≤128) still beats ANN-approximation
|
|
42
|
+
error ([arxiv 2501.10534](https://arxiv.org/html/2501.10534v1)).
|
|
43
|
+
Full 120 MB → int4 ≈ 15–20 MB near-lossless.
|
|
44
|
+
- **Static embeddings**: Model2Vec/Potion — distilled static
|
|
45
|
+
embeddings, ~30 MB, 500× faster, beats other statics on MTEB
|
|
46
|
+
([MinishLab](https://github.com/MinishLab/model2vec),
|
|
47
|
+
[fastText comparison](https://minish.ai/blog/2025-07-28-fasttext/)).
|
|
48
|
+
Word-keyed like ours — same OOV gap.
|
|
49
|
+
- **OOV (our biggest functional gap — models are word-keyed)**:
|
|
50
|
+
fastText hashed char 3–6-gram buckets remain the standard answer;
|
|
51
|
+
collision-aware hash embeddings improve them
|
|
52
|
+
([arxiv 1709.03933](https://arxiv.org/abs/1709.03933)).
|
|
53
|
+
|
|
54
|
+
### Named model lines — verified, techniques extracted
|
|
55
|
+
|
|
56
|
+
- **Claude Fable 5 / 5.1** — VERIFIED
|
|
57
|
+
([announce](https://www.anthropic.com/news/claude-fable-5-mythos-5),
|
|
58
|
+
[5.1](https://www.anthropic.com/claude-fable-and-mythos-5-1),
|
|
59
|
+
[Willison](https://simonwillison.net/2026/Jun/9/claude-fable-5/));
|
|
60
|
+
"Fable 5.0" is not a named release. Techniques: same-weights
|
|
61
|
+
multi-safeguard releases, 5-level reasoning-effort control, cheap
|
|
62
|
+
cache reads. **Relevance: low** — effort-tiering is already our
|
|
63
|
+
tier/backend model.
|
|
64
|
+
- **GLM 5.1/5.2/5.3/5.3-Flash** — VERIFIED
|
|
65
|
+
([tech report](https://arxiv.org/html/2602.15763v1), z.ai blogs).
|
|
66
|
+
Key adoptable idea from 5.3: **RL on synthesized verifiable
|
|
67
|
+
environments with binary rewards** — all gains from post-training on
|
|
68
|
+
the same base. Spelling correction is verifiable by construction
|
|
69
|
+
(output == reference), making it an ideal small-scale RL domain.
|
|
70
|
+
- **DeepSeek V4 Pro/Flash** — VERIFIED
|
|
71
|
+
([report](https://arxiv.org/html/2606.19348v1)). Adoptable:
|
|
72
|
+
specialist→student **on-policy distillation (reverse KL)** — the
|
|
73
|
+
exact recipe for distilling a big reranker into our tier sizes;
|
|
74
|
+
FP4/QAT embedding-table practice. MoE/CSA/1M-context: scale-mismatched.
|
|
75
|
+
- **Hy4** — IDENTIFIED as Tencent Hunyuan Hy4 preview (770B/49B MoE,
|
|
76
|
+
Apache-2.0, [HF](https://huggingface.co/tencent/Hy4-preview)).
|
|
77
|
+
**Relevance: low** — no training details published.
|
|
78
|
+
|
|
79
|
+
## Adoption matrix
|
|
80
|
+
|
|
81
|
+
### Adopt now (Ruby today, no new deps)
|
|
82
|
+
|
|
83
|
+
| # | What | Where it lands | Why now |
|
|
84
|
+
|---|---|---|---|
|
|
85
|
+
| A1 | **Multilingual keyboard-aware typo generator** (MULTYPO-style) for the eval harness — our current `run_eval.py` typos are ASCII-lowercase-only, nearly meaningless for de/ru/ja/zh | models repo, `eval/` | Biggest correctness hole in our own gates; pure code; keyboard layouts already exist in the gem ([MULTYPO](https://github.com/cisnlp/multypo)) |
|
|
86
|
+
| A2 | **GitHub Typo Corpus** as a real-world multilingual eval set (350k+ edits, 15+ languages) alongside the synthetic proxy | models repo, `eval/corpora/` (download-once, cache) | Grounds the proxy metric in real human typos ([corpus](https://github.com/mhagiwara/github-typo-corpus), [LREC paper](https://aclanthology.org/2020.lrec-1.835/)) |
|
|
87
|
+
| A3 | **Confidence cascade, formalized**: dictionary/affix verdict first; ONNX rerank only for uncertain candidates; document thresholds | gem, semantic path (plan 05) | Published hybrid twin does exactly this; we half-do it today ([arxiv 2410.23514](https://arxiv.org/html/2410.23514v1)) |
|
|
88
|
+
| A4 | **Single nested (2D-Matryoshka) artifact**: one frequency-ordered file + declared truncation points replacing mini/fluency downloads; separate files stay available | models repo, post-07 experiment | Truncation safety is literature-validated; one download serves all tiers ([arxiv 2605.16608](https://arxiv.org/html/2605.16608v2)) |
|
|
89
|
+
|
|
90
|
+
### Adopt with kotoshu-rs (ort / Rust core)
|
|
91
|
+
|
|
92
|
+
| # | What | Lands in | Why then |
|
|
93
|
+
|---|---|---|---|
|
|
94
|
+
| B1 | **int4 group-128 quantized full tier** (120 MB → ~15–20 MB near-lossless); possibly a `nano` tier | kotoshu-rs P3 + models repo tiers | Needs the core's own dequant math / ort support; near-lossless per [2501.10534](https://arxiv.org/html/2501.10534v1) |
|
|
95
|
+
| B2 | **Hashed char n-gram OOV fallback** (fastText buckets or collision-aware hashing) so unseen words still embed | kotoshu-rs P3; converter in models repo | Fixes the word-keyed OOV gap at last ([1709.03933](https://arxiv.org/abs/1709.03933)) |
|
|
96
|
+
| B3 | **Tiny cross-encoder reranker option** (~22M distilled MiniLM class, ONNX) behind the provider trait | kotoshu-rs P3+ | Better rerank than cosine bi-encoder for hard cases; our ≤25-candidate scale keeps it cheap ([sbert efficiency](https://sbert.net/docs/cross_encoder/usage/efficiency.html)) |
|
|
97
|
+
|
|
98
|
+
### Adopt later (deliberate, data-gated)
|
|
99
|
+
|
|
100
|
+
| # | What | Gate |
|
|
101
|
+
|---|---|---|
|
|
102
|
+
| C1 | **Model2Vec/Potion static embeddings** as reranker base (~30 MB, 500× faster) | per-model license verified for redistribution; win over fastText shown on our eval |
|
|
103
|
+
| C2 | **RL with binary verifiable rewards** to train reranker weights (GLM-5.3 recipe, scaled down ~10⁶) | after conformance vectors exist (67 M3) so reward is measurable; needs a training pipeline owner |
|
|
104
|
+
| C3 | **On-policy distillation** of a large teacher into tier-sized rerankers (DeepSeek V4 recipe) | same gate as C2 |
|
|
105
|
+
| C4 | **LLM-as-evaluator** for eval reports (offline, not CI) | optional; proxy metrics suffice until corpora land |
|
|
106
|
+
| C5 | **Optional external LLM mode** (1–8B fine-tuned GEC or API) as an explicit opt-in "heavy" backend | user demand; never default (latency/RAM violate our constraints) |
|
|
107
|
+
|
|
108
|
+
### Reject
|
|
109
|
+
|
|
110
|
+
- **Shipping any LLM as the default engine** — violates small/offline/CPU.
|
|
111
|
+
- **MoE, MLA/CSA attention, 1M-context techniques** — meaningful only at
|
|
112
|
+
10⁹+ parameter scale; a 300-dim embedding table has nothing to sparse-ify.
|
|
113
|
+
- **Distillation-prevention / watermarking / cache-economics patterns**
|
|
114
|
+
from the frontier labs — business-side, not engine-side.
|
|
115
|
+
|
|
116
|
+
## Near-term execution (promotes from this plan)
|
|
117
|
+
|
|
118
|
+
1. **A1 typo generator** — extend `eval/` with per-language keyboard
|
|
119
|
+
noise models driven by the gem's existing layout data; regenerate
|
|
120
|
+
all 18 reports; gates unchanged (they will get *harder*, honestly).
|
|
121
|
+
2. **A2 corpus benchmark** — fetch script + cache + report format;
|
|
122
|
+
license (MIT) recorded in the registry-style metadata.
|
|
123
|
+
3. **A4 nested-artifact experiment** — build `fasttext.{lang}.matryoshka.onnx`
|
|
124
|
+
(full matrix, frequency-ordered, truncation manifest), eval
|
|
125
|
+
truncated views against the shipped tiers; adopt only if equal or
|
|
126
|
+
better; registry ids unchanged.
|
|
127
|
+
4. Record B1/B2 as named milestones inside plan 66 P3 (no separate
|
|
128
|
+
plan files until the core exists).
|
|
129
|
+
|
|
130
|
+
## Owner gates
|
|
131
|
+
|
|
132
|
+
New tier names (`nano`), changing the default tier, any new model
|
|
133
|
+
licenses (Model2Vec per-model, GitHub Typo Corpus MIT), and anything
|
|
134
|
+
requiring network in eval CI — all owner decisions.
|
|
135
|
+
|
|
136
|
+
## Status
|
|
137
|
+
|
|
138
|
+
**Adopted 2026-09-02.** Items A1–A4 promote to models-repo execution;
|
|
139
|
+
B1–B3 fold into plan 66 P3; C/D items remain gated. Research brief
|
|
140
|
+
preserved above for provenance.
|
|
141
|
+
|
|
142
|
+
**B1 evaluated (2026-09-03, models PR #11): DO NOT SHIP.** int4-group128
|
|
143
|
+
hits 16.2 MB (7.4x under full) and clears rank_corr on all 9 languages,
|
|
144
|
+
but top1_agreement fails the 0.95 fluency gate on 8 of 9 (ru only
|
|
145
|
+
passer) - 4-bit noise flips 1-2 typo probes per language. Implementation
|
|
146
|
+
verified exact (ORT round-trip 0.0), so the format itself loses;
|
|
147
|
+
int8-per-row stays. The levers were then exhausted (models PR #12): half-away rounding is
|
|
148
|
+
a provable no-op, group-size shrink helps monotonically on ja but
|
|
149
|
+
asymptotes below the gate (en pinned at 0.90), and the mixed
|
|
150
|
+
int4-mini fallback fails on de. B1 closed - int8-per-row is the
|
|
151
|
+
final answer. The third eval-driven rejection of the campaign (after
|
|
152
|
+
SVD dim-reduction and Model2Vec).
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# 69 — Keyboard-Aware Typo Eval + Real-Typo Corpus Benchmark
|
|
2
|
+
|
|
3
|
+
Executes the "adopt now" items A1 + A2 of
|
|
4
|
+
[68-sota-adoption.md](68-sota-adoption.md). Lands in
|
|
5
|
+
`models-fasttext-onnx` (`eval/`), feeding the same gate machinery as
|
|
6
|
+
plan 06.
|
|
7
|
+
|
|
8
|
+
## Goal
|
|
9
|
+
|
|
10
|
+
1. Replace the eval harness's ASCII-lowercase-only typo generator with
|
|
11
|
+
a **keyboard-layout-aware multilingual noise model** (MULTYPO-style,
|
|
12
|
+
[repo](https://github.com/cisnlp/multypo),
|
|
13
|
+
[arxiv 2005.01158](https://arxiv.org/html/2005.01158v1)):
|
|
14
|
+
QWERTY/QWERTZ/AZERTY/JCUKEN neighbor-weighted slips for
|
|
15
|
+
de/en/es/fr/pt/ru, umlaut/eszett keys for de, Cyrillic neighbors for
|
|
16
|
+
ru; for ja/ko/zh a visual/phonetic character-confusion model
|
|
17
|
+
(homoglyph + stroke-similar substitutions) replaces key-adjacency.
|
|
18
|
+
All deterministic (seeded), in-vocab typos only (the metric requires
|
|
19
|
+
both models to embed the typo).
|
|
20
|
+
2. Add the **GitHub Typo Corpus**
|
|
21
|
+
([repo](https://github.com/mhagiwara/github-typo-corpus),
|
|
22
|
+
[LREC 2020](https://aclanthology.org/2020.lrec-1.835/); 350k+ real
|
|
23
|
+
human typo corrections, 15+ languages, MIT) as a real-world
|
|
24
|
+
embedding-side benchmark: for corpus (typo → correction) pairs in
|
|
25
|
+
our 9 languages where both words are in-vocab, measure how the
|
|
26
|
+
cosine-to-typo ranking places the correction, per tier. Report-only
|
|
27
|
+
(no gate) in v1 — it measures the full correction pipeline only
|
|
28
|
+
after kotoshu-rs conformance vectors exist (plan 67 M3).
|
|
29
|
+
|
|
30
|
+
## Tasks
|
|
31
|
+
|
|
32
|
+
1. `eval/noise.py` — per-language deterministic typo generator:
|
|
33
|
+
layout-driven for Latin/Cyrillic scripts (the gem's own
|
|
34
|
+
`keyboard/layouts` are the layout source of truth — mirror them),
|
|
35
|
+
character-confusion for CJK. Same RNG discipline as `run_eval.py`
|
|
36
|
+
(`default_rng([seed, crc32(lang)])`).
|
|
37
|
+
2. Wire into `run_eval.top1_agreement_metric` as the typo source;
|
|
38
|
+
**gates unchanged** — they get honestly harder. Ladders climb if a
|
|
39
|
+
language now misses (record attempts as before).
|
|
40
|
+
3. `scripts/fetch_corpus.py` — download-once the GitHub Typo Corpus
|
|
41
|
+
release into `eval/corpora/` (gitignored), verify checksum, extract
|
|
42
|
+
per-language pairs.
|
|
43
|
+
4. `eval/corpus_bench.py` — the benchmark runner →
|
|
44
|
+
`eval/reports/corpus.{lang}.json` (committed): pairs evaluated,
|
|
45
|
+
correction-in-top-1/5/20 rates per tier vs full.
|
|
46
|
+
5. Regenerate all 18 tier reports with the new typo model; commit the
|
|
47
|
+
new reports + report the gate deltas in the PR.
|
|
48
|
+
|
|
49
|
+
## Acceptance
|
|
50
|
+
|
|
51
|
+
- `run_eval.py` produces identical results for identical inputs
|
|
52
|
+
(determinism test); non-Latin languages get realistic typos
|
|
53
|
+
(spot-check list in the PR).
|
|
54
|
+
- All 9×2 tier reports regenerated; any ladder climbs recorded.
|
|
55
|
+
- `corpus_bench.py` runs offline after `fetch_corpus.py`; reports
|
|
56
|
+
committed; corpus itself never committed (size + license header
|
|
57
|
+
attribution in the report JSON).
|
|
58
|
+
|
|
59
|
+
## Dependencies
|
|
60
|
+
|
|
61
|
+
- Feeds plan 06's gate machinery; blocks nothing.
|
|
62
|
+
- Owner gate: none (corpus is MIT; no new model distribution).
|
|
63
|
+
|
|
64
|
+
## Status
|
|
65
|
+
|
|
66
|
+
**Implemented (2026-09-03, models PR #5 merged).** eval/noise.py ships
|
|
67
|
+
layout-neighbor slips (QWERTY/QWERTZ/AZERTY/JCUKEN mirrored from the
|
|
68
|
+
gem layouts, diacritics included), Hangul jamo decomposition and
|
|
69
|
+
curated CJK confusion sets, transposition fallback; deterministic. All
|
|
70
|
+
18 reports regenerated under the harder typos - gates pass, no ladder
|
|
71
|
+
climbs; de/fluency corrected to 50k (the 60k was small-n noise).
|
|
72
|
+
GitHub Typo Corpus bench live (official S3 dead; Wayback snapshot with
|
|
73
|
+
self-pinned sha256); top-5 full-tier baselines 0.13 (ja) - 0.60 (ko).
|
|
74
|
+
Part of release v1.0.1.
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# 70 — Confidence Cascade in the Semantic Path
|
|
2
|
+
|
|
3
|
+
Executes adopt-now item A3 of
|
|
4
|
+
[68-sota-adoption.md](68-sota-adoption.md) in THIS repo (the gem),
|
|
5
|
+
following the published hybrid dictionary+neural architecture
|
|
6
|
+
([arxiv 2410.23514](https://arxiv.org/html/2410.23514v1): dictionary
|
|
7
|
+
first, neural rerank only for uncertain candidates — and their finding
|
|
8
|
+
that the confidence boundary is where the quality lives).
|
|
9
|
+
|
|
10
|
+
## Goal
|
|
11
|
+
|
|
12
|
+
Formalize what the semantic path already half-does: the ONNX rerank
|
|
13
|
+
runs **only when the traditional composite strategy is not already
|
|
14
|
+
confident**, with an explicit, configurable threshold — cutting model
|
|
15
|
+
loads and latency on the happy path without changing results where the
|
|
16
|
+
dictionary is certain.
|
|
17
|
+
|
|
18
|
+
## Design
|
|
19
|
+
|
|
20
|
+
1. `Configuration::SCHEMA` gains `semantic_cascade_threshold`
|
|
21
|
+
(float, default `1.0` — preserving today's always-rerank behavior;
|
|
22
|
+
ENV `KOTOSHU_SEMANTIC_CASCADE_THRESHOLD` automatic via the schema).
|
|
23
|
+
2. In `Suggestions::Generator` (or `SemanticStrategy`), when the
|
|
24
|
+
composite strategy's top candidate confidence ≥ threshold, skip the
|
|
25
|
+
semantic rerank for that suggestion set; log the skip under debug
|
|
26
|
+
metrics.
|
|
27
|
+
3. `docs/` note + README section: what the threshold means, how to
|
|
28
|
+
tune it (0.0 = never rerank, 1.0 = always), and that measured
|
|
29
|
+
guidance comes from the eval harness once plan 69's corpus bench
|
|
30
|
+
lands.
|
|
31
|
+
4. Spec coverage: behavior with threshold 1.0 (identical to today),
|
|
32
|
+
0.0, and a mid value — real model instances per the global no-doubles
|
|
33
|
+
rule; onnx-dependent specs tagged `:onnx` per the existing opt-in
|
|
34
|
+
convention.
|
|
35
|
+
|
|
36
|
+
## Acceptance
|
|
37
|
+
|
|
38
|
+
- Default behavior is byte-identical to today (existing suite green).
|
|
39
|
+
- A mid-threshold spec proves the skip happens exactly when confidence
|
|
40
|
+
≥ threshold, and never otherwise.
|
|
41
|
+
- `kotoshu check --verbose` surfaces cascade-skip counts.
|
|
42
|
+
|
|
43
|
+
## Dependencies
|
|
44
|
+
|
|
45
|
+
- Independent of 69/71; tuning guidance arrives from 69's corpus bench.
|
|
46
|
+
- Performance numbers for the README: measure rerank skip rate on the
|
|
47
|
+
existing example corpus at thresholds {1.0, 0.9, 0.8}.
|
|
48
|
+
|
|
49
|
+
## Status
|
|
50
|
+
|
|
51
|
+
**Implemented (2026-09-03, gem PR #98 merged).** semantic_cascade_
|
|
52
|
+
threshold in the SCHEMA (env automatic), Suggestions::SemanticCascade
|
|
53
|
+
decision object with the 1.0 never-skip sentinel (composite confidence
|
|
54
|
+
legitimately reaches exactly 1.0 - documented in ALWAYS_RERANK),
|
|
55
|
+
semantic_cascade_skips counters in metrics + verbose. Drive-by fixes:
|
|
56
|
+
dead convert_schema_value Integer/Float branches (ENV never converted),
|
|
57
|
+
missing Metrics::Collector / Debug::Logger autoloads. 97 examples
|
|
58
|
+
green, rubocop clean. Tuning guidance pending plan 69 corpus numbers
|
|
59
|
+
now committed in the models repo.
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# 71 — Nested (2D-Matryoshka) Single-Artifact Experiment
|
|
2
|
+
|
|
3
|
+
Executes adopt-now item A4 of
|
|
4
|
+
[68-sota-adoption.md](68-sota-adoption.md) in `models-fasttext-onnx`.
|
|
5
|
+
Literature: plain prefix truncation of embeddings is safe at our
|
|
6
|
+
ratios ([arxiv 2605.16608](https://arxiv.org/html/2605.16608v2));
|
|
7
|
+
2D-MRL nesting along vocab+dims is validated practice
|
|
8
|
+
([arxiv 2410.13230](https://arxiv.org/html/2410.13230v1),
|
|
9
|
+
[HF Matryoshka guide](https://huggingface.co/blog/matryoshka)).
|
|
10
|
+
|
|
11
|
+
## Goal
|
|
12
|
+
|
|
13
|
+
One int8 file per language, frequency-ordered, whose **row-prefix
|
|
14
|
+
views ARE the tiers**: a `fasttext.{lang}.nested.onnx` (100k × 300d,
|
|
15
|
+
int8-per-row, ~30 MB) from which `fluency` = first-50k view (60k de)
|
|
16
|
+
and `mini` = first-10k view. If the views eval identically to the
|
|
17
|
+
shipped separate tiers, consumers download one artifact per language
|
|
18
|
+
instead of up to three, and tier "upgrades" are free (the bytes are
|
|
19
|
+
already on disk).
|
|
20
|
+
|
|
21
|
+
## Tasks
|
|
22
|
+
|
|
23
|
+
1. `scripts/build_nested.py` — build the nested artifact from the
|
|
24
|
+
full model (same int8-per-row math as `build_tiers.py`; import, do
|
|
25
|
+
not duplicate). Emit a truncation manifest (per-language prefix
|
|
26
|
+
lengths) into the artifact's metadata_props and a sibling
|
|
27
|
+
`fasttext.{lang}.nested.json`.
|
|
28
|
+
2. `eval/` view-eval: run the existing gate metrics for the fluency
|
|
29
|
+
and mini views of the nested file against the shipped separate
|
|
30
|
+
tier files — assert metric parity (same seeds, same probe
|
|
31
|
+
sequences ⇒ identical numbers, since the construction is
|
|
32
|
+
identical; any delta is a bug, not noise).
|
|
33
|
+
3. Size/bandwidth report: per language, {3 separate files} vs {full +
|
|
34
|
+
nested}; which consumers win under which access pattern.
|
|
35
|
+
4. **Recommendation, not adoption**: registry ids and the release
|
|
36
|
+
workflow stay unchanged in this plan. Adoption = a follow-up
|
|
37
|
+
registry v2 field (`view_of`) + gem/kotoshu-rs resolver support,
|
|
38
|
+
proposed only if the parity + size report supports it.
|
|
39
|
+
|
|
40
|
+
## Acceptance
|
|
41
|
+
|
|
42
|
+
- Nested views eval **identically** (bit-level metric parity) to the
|
|
43
|
+
shipped tiers for all 9 languages.
|
|
44
|
+
- Size report committed (`eval/reports/nested.{lang}.json` summary).
|
|
45
|
+
- No change to registry.json, manifest.json, or release.yml in this
|
|
46
|
+
plan.
|
|
47
|
+
|
|
48
|
+
## Dependencies
|
|
49
|
+
|
|
50
|
+
- After 69 if report regeneration is in flight (same eval files) —
|
|
51
|
+
sequence to avoid collisions; otherwise independent.
|
|
52
|
+
- Owner gate: adoption itself (new registry field + default artifact
|
|
53
|
+
strategy) is an owner decision.
|
|
54
|
+
|
|
55
|
+
## Status
|
|
56
|
+
|
|
57
|
+
**Experiment complete (2026-09-03, models PR #6 merged).** build_
|
|
58
|
+
nested.py + check_nested_parity.py; 18/18 EXACT array + metric parity
|
|
59
|
+
against the keyboard-aware reports. Recommendation recorded in
|
|
60
|
+
nested.summary.json: separate tiers stay the default (mini-only users
|
|
61
|
+
would pay ~10x); nested ships as an optional all-in-one if adopted.
|
|
62
|
+
Adoption (registry view_of field + resolver support) remains an owner
|
|
63
|
+
decision - deliberately not taken.
|