kotoshu 1.0.2-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# 05 — Semantic Path Productionization
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
The ONNX/FastText semantic path is a first-class, always-available
|
|
6
|
+
dimension of analysis for every language that has an embedding model —
|
|
7
|
+
not an opt-in afterthought.
|
|
8
|
+
|
|
9
|
+
## Why
|
|
10
|
+
|
|
11
|
+
Today the semantic path is gated behind `ENV['KOTOSHU_REQUIRE_ONNX']`
|
|
12
|
+
(see `lib/kotoshu/models/onnx_model.rb` line 22). Without that env var,
|
|
13
|
+
the onnxruntime gem isn't loaded and semantic features silently
|
|
14
|
+
disappear. The hybrid model (`README.adoc` advertises "Hunspell
|
|
15
|
+
candidates + FastText reranking") is the recommended mode but isn't
|
|
16
|
+
wired as a default.
|
|
17
|
+
|
|
18
|
+
## Tasks
|
|
19
|
+
|
|
20
|
+
1. **Drop the env gate.** Require `onnxruntime` lazily on first
|
|
21
|
+
`OnnxModel` instantiation, with a clear error if the gem isn't
|
|
22
|
+
installed (`RuntimeError` with install instructions). Remove the
|
|
23
|
+
`KOTOSHU_REQUIRE_ONNX` env var — it's an undocumented footgun.
|
|
24
|
+
2. **Make `onnxruntime` a hard runtime dependency** in
|
|
25
|
+
`kotoshu.gemspec` (it already is, but the lazy require contradicts
|
|
26
|
+
this). Verify `gem install kotoshu` pulls it in.
|
|
27
|
+
3. **Default to hybrid mode.** The README says hybrid is "recommended"
|
|
28
|
+
but the code defaults to `hunspell` only (see
|
|
29
|
+
`commands/check_command.rb` line 23). Make `:hybrid` the default;
|
|
30
|
+
document when to choose pure `:hunspell` (speed) or pure
|
|
31
|
+
`:fasttext` (semantic-only, no dictionary).
|
|
32
|
+
4. **Memory budget.** ONNX models are 200MB-4GB each. Only one model
|
|
33
|
+
per language should be resident at a time. Wire
|
|
34
|
+
`Embeddings::LruCache` so the `ResourceManager` owns the cache and
|
|
35
|
+
evicts when memory pressure exceeds a configurable ceiling
|
|
36
|
+
(`Configuration.max_embedding_memory_mb`, default 1024).
|
|
37
|
+
5. **Cold-start latency.** Loading a 4GB ONNX model takes seconds.
|
|
38
|
+
Show a one-time progress message in the CLI; expose a `preload`
|
|
39
|
+
method on `OnnxModel` so background warming is possible.
|
|
40
|
+
6. **Vocabulary coverage fallback.** If a word is OOV for the
|
|
41
|
+
embedding model, fall back to subword FastText if available;
|
|
42
|
+
otherwise skip semantic reranking for that word (don't crash).
|
|
43
|
+
7. **Hybrid scoring formula.** Document and tune the actual formula:
|
|
44
|
+
`final_score = α·hunspell_distance_score + β·semantic_similarity +
|
|
45
|
+
γ·frequency_bonus`. Make α/β/γ configurable per language; defaults
|
|
46
|
+
per language live in the language module.
|
|
47
|
+
8. **Benchmark suite.** Add `spec/performance/semantic_bench.rb`:
|
|
48
|
+
- Time to load model (baseline: < 1 s for the 114 MB models)
|
|
49
|
+
- Time per 1000 suggestions with vs. without semantic rerank
|
|
50
|
+
(baseline: ~1–2 ms/query inference, ~115 MB resident per language)
|
|
51
|
+
- Memory ceiling under load
|
|
52
|
+
Track these in `PERFORMANCE.md` and gate against regressions in CI.
|
|
53
|
+
9. **ONNX Runtime version compatibility.** README claims
|
|
54
|
+
`onnxruntime 1.23.2+`. Verify across 1.22, 1.23, 1.24. Lock the gem
|
|
55
|
+
version range in gemspec and document.
|
|
56
|
+
|
|
57
|
+
## Acceptance criteria
|
|
58
|
+
|
|
59
|
+
- `Kotoshu.suggest("helo", language: "en")` uses hybrid mode by
|
|
60
|
+
default, no env var needed
|
|
61
|
+
- Loading two languages sequentially does not double memory (LRU
|
|
62
|
+
evicts)
|
|
63
|
+
- `spec/performance/semantic_bench.rb` runs in CI and fails on >20%
|
|
64
|
+
regression vs. baseline
|
|
65
|
+
- README's "hybrid" example works as documented
|
|
66
|
+
- `KOTOSHU_REQUIRE_ONNX` is removed from the codebase and from any
|
|
67
|
+
docs that mention it
|
|
68
|
+
|
|
69
|
+
## Dependencies
|
|
70
|
+
|
|
71
|
+
- Blocks: `11-release`
|
|
72
|
+
- Blocked by: `03-dynamic-download` (model resolution), `09-integrity-security`
|
|
73
|
+
- Cross-repo: depends on `models-fasttext-onnx/TODO.impl/01-publish-all-models.md`
|
|
74
|
+
|
|
75
|
+
## Out of scope
|
|
76
|
+
|
|
77
|
+
- Training new FastText models (use upstream Facebook vectors)
|
|
78
|
+
- GPU acceleration (CPU onnxruntime is the v1 target)
|
|
79
|
+
- Quantization / INT8 (already 37x compressed vs. raw .vec — defer)
|
|
80
|
+
|
|
81
|
+
## Status
|
|
82
|
+
|
|
83
|
+
_Pending._
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# 06 — CJK Support (Japanese, Chinese)
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Japanese and Chinese work end-to-end. CJK doesn't have "spelling" in the
|
|
6
|
+
alphabetic sense — the paradigm is morphological tokenization +
|
|
7
|
+
confusion-pair rules. Kotoshu should expose this honestly rather than
|
|
8
|
+
pretend a dictionary lookup works.
|
|
9
|
+
|
|
10
|
+
## Why
|
|
11
|
+
|
|
12
|
+
ROADMAP plan `004-japanese.md` and `005-chinese.md` are 🔲 Not Started.
|
|
13
|
+
The gem ships a `JapaneseTokenizer` skeleton under
|
|
14
|
+
`lib/kotoshu/language/tokenizer/` but no morphological backend, and a
|
|
15
|
+
`ja/language.rb` module with no real wiring. The `dictionaries` repo
|
|
16
|
+
has `ja/` and `zh/` dirs but only with `grammar/` subdirs (no
|
|
17
|
+
`spelling/`) — correct, since CJK doesn't use spelling dicts.
|
|
18
|
+
|
|
19
|
+
## Paradigm
|
|
20
|
+
|
|
21
|
+
| Concern | Latin | CJK |
|
|
22
|
+
|---|---|---|
|
|
23
|
+
| Spellcheck | Dictionary membership + affixes | N/A |
|
|
24
|
+
| Tokenization | Whitespace/regex | Morphological analyzer |
|
|
25
|
+
| Errors | Out-of-dict words | Confusion-pair rules (e.g. 辨/辯/辮) |
|
|
26
|
+
| Suggestions | Edit distance + phonetic | Confusion probability tables |
|
|
27
|
+
|
|
28
|
+
## Tasks
|
|
29
|
+
|
|
30
|
+
### Japanese
|
|
31
|
+
|
|
32
|
+
1. **Pick a morphological backend.** Options:
|
|
33
|
+
- **Sen** (Java, requires JRuby — likely unacceptable for pure-Ruby
|
|
34
|
+
promise)
|
|
35
|
+
- **Suika** (already in the gemspec — pure Ruby MeCab-IPADic port,
|
|
36
|
+
the right choice)
|
|
37
|
+
- FFI to MeCab (breaks pure-Ruby promise)
|
|
38
|
+
Decision: **use suika**. Add `Kotoshu::Language::Tokenizer::SuikaTokenizer`
|
|
39
|
+
that wraps `Suika::Tagger`.
|
|
40
|
+
2. **Wire `JapaneseTokenizer`** to delegate to suika for segmentation +
|
|
41
|
+
POS. Keep the existing class name for backward compatibility.
|
|
42
|
+
3. **Confusion rules.** Source a Japanese confusion-pair list (homophone
|
|
43
|
+
kanji 食べる/搗ける etc.) and ship as
|
|
44
|
+
`dictionaries/ja/grammar/confusion.yaml`. The grammar engine
|
|
45
|
+
(plan `08`) reads it.
|
|
46
|
+
4. **Reading-check rules.** Kotoshi-specific (言修 = ことしゅ): verify
|
|
47
|
+
furigana/reading correctness against jumbo dictionaries. Defer if
|
|
48
|
+
out of scope for v1.
|
|
49
|
+
5. **No embedding model.** Japanese FastText exists upstream but the
|
|
50
|
+
`models-fasttext-onnx` repo doesn't have `ja/`. Either convert it
|
|
51
|
+
(cross-ref `models-fasttext-onnx/TODO.impl/01-publish-all-models.md`)
|
|
52
|
+
or document Japanese as traditional-rules-only.
|
|
53
|
+
|
|
54
|
+
### Chinese
|
|
55
|
+
|
|
56
|
+
6. **Pick a segmenter.** Pure-Ruby options are limited. Options:
|
|
57
|
+
- Hand-rolled maximum-matching against a word list (acceptable for
|
|
58
|
+
v1, fast, deterministic)
|
|
59
|
+
- FFI to HanLP (breaks pure-Ruby promise)
|
|
60
|
+
- Port a small CRF model to Ruby (large effort)
|
|
61
|
+
Decision: **ship max-matching with the existing dictionaries/zh word
|
|
62
|
+
list for v1; note the accuracy ceiling in docs**.
|
|
63
|
+
7. **Simplified/Traditional** detection and cross-checking.
|
|
64
|
+
8. **Confusion rules.** Common confusions (的/得/地, 在/再, 做/作) ship
|
|
65
|
+
as `dictionaries/zh/grammar/confusion.yaml`.
|
|
66
|
+
|
|
67
|
+
### Shared
|
|
68
|
+
|
|
69
|
+
9. **Update `Language::Base`** so `script: :cjk` languages declare
|
|
70
|
+
`spell_checker: :passthrough` (use existing
|
|
71
|
+
`Components::PassthroughSpellChecker`) and require a tokenizer.
|
|
72
|
+
10. **CLI flag `--model`** for CJK should reject `:hunspell` and
|
|
73
|
+
`:hybrid` with a clear error explaining why.
|
|
74
|
+
11. **CJK-aware SARIF output.** Character offsets must be in Unicode
|
|
75
|
+
codepoints, not bytes; verify the `Location` class in `documents/`
|
|
76
|
+
handles this.
|
|
77
|
+
|
|
78
|
+
## Acceptance criteria
|
|
79
|
+
|
|
80
|
+
- `Kotoshu.check("私は 学生 です。")` tokenizes correctly and reports
|
|
81
|
+
no errors (with default rules)
|
|
82
|
+
- `Kotoshu.check("私はがくせいです。")` flags the use of hiragana where
|
|
83
|
+
kanji is conventional, with a confusion-rule suggestion
|
|
84
|
+
- `Kotoshu.check("他是在做好了。")` flags 的/得 confusion
|
|
85
|
+
- README documents that CJK is "rules + tokenization, no spellcheck"
|
|
86
|
+
rather than silently producing nonsense
|
|
87
|
+
|
|
88
|
+
## Dependencies
|
|
89
|
+
|
|
90
|
+
- Blocks: nothing
|
|
91
|
+
- Blocked by: `03-dynamic-download` (need to fetch the suika dict or
|
|
92
|
+
bundle it), `08-grammar-engine` (confusion rules need the engine)
|
|
93
|
+
- Cross-repo: `dictionaries/ja/grammar/` and `dictionaries/zh/grammar/`
|
|
94
|
+
need content; coordinate with
|
|
95
|
+
`dictionaries/TODO.impl/02-coverage-matrix.md`
|
|
96
|
+
|
|
97
|
+
## Out of scope
|
|
98
|
+
|
|
99
|
+
- Korean (similar paradigm but no plan written — add later)
|
|
100
|
+
- Full furigana generation
|
|
101
|
+
- Translation / ROMAJI conversion
|
|
102
|
+
|
|
103
|
+
## Status
|
|
104
|
+
|
|
105
|
+
_Pending._
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# 07 — RTL Script Support (Arabic, Hebrew, Persian, Urdu)
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Right-to-left scripts work end-to-end with correct shaping, normalization,
|
|
6
|
+
and bidirectional handling.
|
|
7
|
+
|
|
8
|
+
## Why
|
|
9
|
+
|
|
10
|
+
ROADMAP plan `007-rtl-languages.md` is 🔲 Not Started. The
|
|
11
|
+
`dictionaries` repo has `ar`, `he`, `fa` (likely; verify) dirs.
|
|
12
|
+
Hunspell handles RTL fine in principle, but Ruby's default string
|
|
13
|
+
handling needs explicit care.
|
|
14
|
+
|
|
15
|
+
## Tasks
|
|
16
|
+
|
|
17
|
+
1. **Verify dictionary coverage** in the `dictionaries` repo for `ar`,
|
|
18
|
+
`he`, `fa`, `ur`. If any are missing, file under
|
|
19
|
+
`dictionaries/TODO.impl/02-coverage-matrix.md`.
|
|
20
|
+
2. **Unicode normalization.** Arabic especially has presentation forms
|
|
21
|
+
(U+FExx range) that must be normalized to canonical forms (U+06xx)
|
|
22
|
+
before lookup. Add a `RtlNormalizer` under
|
|
23
|
+
`lib/kotoshu/language/normalizer/`.
|
|
24
|
+
3. **Bidi handling in output.** When SARIF/text output contains mixed
|
|
25
|
+
RTL/LTR (e.g. error message in English pointing at Arabic text),
|
|
26
|
+
apply explicit Unicode bidi marks (LRM/RLM) so terminal and editor
|
|
27
|
+
render correctly. Verify the `DisplayFormatter` handles this.
|
|
28
|
+
4. **Tokenizer.** Arabic/Hebrew script tokenizers differ from Latin
|
|
29
|
+
(clitics, the definite article "ال"). Add
|
|
30
|
+
`ArabicTokenizer`/`HebrewTokenizer` under
|
|
31
|
+
`lib/kotoshu/language/tokenizer/`.
|
|
32
|
+
5. **Keyboard layouts.** Add Arabic (101/102), Hebrew standard. Map to
|
|
33
|
+
the existing `KeyboardProximityStrategy`.
|
|
34
|
+
6. **FastText models.** All four languages have upstream FastText
|
|
35
|
+
crawl vectors. Verify they're in `models-fasttext-onnx` (the local
|
|
36
|
+
repo has 158 models; spot-check ar/he/fa/ur are present).
|
|
37
|
+
7. **Grammar rules (light).** Common confusions: Arabic alif variants
|
|
38
|
+
(ا/أ/إ/آ), Hebrew similar-final letter pairs (כ/ך, מ/ם). Ship as
|
|
39
|
+
`dictionaries/{code}/grammar/confusion.yaml`.
|
|
40
|
+
8. **Kelly frequency.** Arabic has Kelly data; verify Hebrew/Persian/Urdu
|
|
41
|
+
coverage in `frequency-list-kelly` and file extension tasks if
|
|
42
|
+
missing.
|
|
43
|
+
9. **Regression tests.** Each RTL language gets a spec with a known
|
|
44
|
+
error and assertion that the suggestion is in the correct script
|
|
45
|
+
and the position offset is correct in codepoints.
|
|
46
|
+
|
|
47
|
+
## Acceptance criteria
|
|
48
|
+
|
|
49
|
+
- `Kotoshu.check("السلام عليكم")` (correct) → no errors
|
|
50
|
+
- `Kotoshu.check("السلم عليكم")` (missing alif) → flagged with
|
|
51
|
+
suggestion "السلام"
|
|
52
|
+
- SARIF output for an RTL error renders correctly on GitHub
|
|
53
|
+
- Terminal output doesn't corrupt cursor position from bidi reordering
|
|
54
|
+
|
|
55
|
+
## Dependencies
|
|
56
|
+
|
|
57
|
+
- Blocks: nothing
|
|
58
|
+
- Blocked by: `03-dynamic-download`, `04-language-modules`
|
|
59
|
+
- Cross-repo: `dictionaries/TODO.impl/02-coverage-matrix.md`,
|
|
60
|
+
`frequency-list-kelly/TODO.impl/01-extend-coverage.md`,
|
|
61
|
+
`models-fasttext-onnx/TODO.impl/01-publish-all-models.md`
|
|
62
|
+
|
|
63
|
+
## Out of scope
|
|
64
|
+
|
|
65
|
+
- Diacritics-only checking (tashkeel) for Arabic — future
|
|
66
|
+
- Ligature-aware suggestions
|
|
67
|
+
- Calligraphy-style variants
|
|
68
|
+
|
|
69
|
+
## Status
|
|
70
|
+
|
|
71
|
+
_Pending._
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# 08 — Grammar Engine
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Ship a LanguageTool-style rule engine that loads per-language rule packs
|
|
6
|
+
from `dictionaries/{code}/grammar/rules.yaml`, runs pattern matchers
|
|
7
|
+
during analysis, and produces the same `WordResult`-style findings as
|
|
8
|
+
spellcheck errors.
|
|
9
|
+
|
|
10
|
+
## Why
|
|
11
|
+
|
|
12
|
+
The `grammar/` module exists (`lib/kotoshu/grammar/` — `rule.rb`,
|
|
13
|
+
`rule_engine.rb`, `rule_loader.rb`, `pattern_matchers/`) and there are
|
|
14
|
+
matchers for double_negative, possessive_context, vowel_sound. But:
|
|
15
|
+
|
|
16
|
+
- No rules are shipped in `dictionaries/*/grammar/rules.yaml` (README
|
|
17
|
+
marks it as "future")
|
|
18
|
+
- The engine isn't wired into the `Spellchecker.check` pipeline
|
|
19
|
+
- The CLI doesn't expose `--grammar on|off`
|
|
20
|
+
- No language has a meaningful rule pack
|
|
21
|
+
|
|
22
|
+
## Tasks
|
|
23
|
+
|
|
24
|
+
1. **Rule schema.** Finalize the YAML schema
|
|
25
|
+
`dictionaries/{code}/grammar/rules.yaml`:
|
|
26
|
+
```yaml
|
|
27
|
+
version: 1
|
|
28
|
+
language: en
|
|
29
|
+
rules:
|
|
30
|
+
- id: EN_A_VS_AN
|
|
31
|
+
category: GRAMMAR
|
|
32
|
+
description: "Use 'an' before vowel sounds"
|
|
33
|
+
pattern:
|
|
34
|
+
type: regex
|
|
35
|
+
before: '\b([aA])\s+([aeiouAEIOU]\w*)'
|
|
36
|
+
suggestion: 'an \2'
|
|
37
|
+
examples:
|
|
38
|
+
- incorrect: "a apple"
|
|
39
|
+
correct: "an apple"
|
|
40
|
+
```
|
|
41
|
+
2. **Wire the engine into Spellchecker.** After dictionary lookup,
|
|
42
|
+
`Grammar::RuleEngine.new(rules).apply(tokens)` runs and appends
|
|
43
|
+
findings to the `DocumentResult`. Each finding has `category`
|
|
44
|
+
(`GRAMMAR`, `STYLE`, `TYPOS`, `CONFUSION_WORDS`) so output can
|
|
45
|
+
filter.
|
|
46
|
+
3. **Loader integration.** `ResourceManager` (plan `03`) loads
|
|
47
|
+
`grammar/rules.yaml` as part of the resource bundle for any language
|
|
48
|
+
that ships one.
|
|
49
|
+
4. **Pattern matcher coverage.** Beyond the three current matchers,
|
|
50
|
+
add:
|
|
51
|
+
- `regex_matcher` — generic regex-based pattern
|
|
52
|
+
- `confusion_matcher` — pair-based confusion (的/得, à/a)
|
|
53
|
+
- `repetition_matcher` — doubled words ("the the")
|
|
54
|
+
- `capitalization_matcher` — sentence-start capitalization
|
|
55
|
+
5. **Rule disable/enable.** Configuration:
|
|
56
|
+
- `Kotoshu.configure { |c| c.disabled_rules = %w[EN_A_VS_AN] }`
|
|
57
|
+
- CLI: `--disable-rule ID`, `--enable-only-category GRAMMAR`
|
|
58
|
+
6. **Per-language rule packs.** For each of the 6 fully-supported
|
|
59
|
+
languages, ship a starter pack of 10-20 high-confidence rules
|
|
60
|
+
sourced from LanguageTool's open-source rule sets (license: LGPL).
|
|
61
|
+
Coordinate cross-repo with `dictionaries/TODO.impl/02-coverage-matrix.md`.
|
|
62
|
+
7. **Rule test fixtures.** Every rule has positive and negative
|
|
63
|
+
examples in its YAML; the spec suite auto-discovers and tests them.
|
|
64
|
+
8. **Performance.** Rule matching must add < 20% to check latency for
|
|
65
|
+
typical documents. Benchmark in `spec/performance/grammar_bench.rb`.
|
|
66
|
+
9. **Auto-correction.** `Kotoshu.correct(text)` / `check --apply` applies
|
|
67
|
+
the top suggestion for each finding when confidence is above a
|
|
68
|
+
threshold. Strategies: `:top` (always top suggestion), `:interactive`
|
|
69
|
+
(prompt per finding — reuse `cli/interactive_reviewer`), `:smart`
|
|
70
|
+
(skip when multiple equally-ranked suggestions exist). Default:
|
|
71
|
+
`:smart`.
|
|
72
|
+
|
|
73
|
+
## Acceptance criteria
|
|
74
|
+
|
|
75
|
+
- `Kotoshu.check("I want to a apple.")` flags the a→an error
|
|
76
|
+
- `Kotoshu.check("the the cat")` flags the doubled word
|
|
77
|
+
- `kotoshu check file.txt --disable-rule EN_A_VS_AN` skips that rule
|
|
78
|
+
- Every shipped rule has passing examples in CI
|
|
79
|
+
- `dictionaries/en/grammar/rules.yaml` has ≥ 15 rules
|
|
80
|
+
|
|
81
|
+
## Dependencies
|
|
82
|
+
|
|
83
|
+
- Blocks: `06-cjk` (needs confusion rules), `07-rtl` (needs confusion
|
|
84
|
+
rules), `11-release`
|
|
85
|
+
- Blocked by: `03-dynamic-download` (rule loading)
|
|
86
|
+
- Cross-repo: `dictionaries/TODO.impl/02-coverage-matrix.md` (rule
|
|
87
|
+
packs must exist)
|
|
88
|
+
|
|
89
|
+
## Out of scope
|
|
90
|
+
|
|
91
|
+
- ML-based grammar models (n-gram error detection is in ROADMAP plan
|
|
92
|
+
008 but defer from v1)
|
|
93
|
+
- Style scoring / "tone" suggestions
|
|
94
|
+
- Premium/paid rule packs
|
|
95
|
+
|
|
96
|
+
## Status
|
|
97
|
+
|
|
98
|
+
_Pending._
|
|
99
|
+
|
|
100
|
+
## Source
|
|
101
|
+
|
|
102
|
+
`docs/KOTOSHU_LANGUAGETOOL_GAPS.md` (dated 2026-01-30) is the reference for
|
|
103
|
+
the 11-language target matrix (en, fr, de, es, ru, ar, pt, zh-Hans,
|
|
104
|
+
zh-Hant, ja, ko) and per-language tokenization/normalization rules.
|
|
105
|
+
Tokenization and language-specific requirements feed `04-language-modules`;
|
|
106
|
+
CLI surface and output formats feed `02-cli-unification`.
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# 09 — Integrity & Security
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Every resource Kotoshu downloads from the network is verified against a
|
|
6
|
+
cryptographic checksum before use. Releases are signed. Downloaded data
|
|
7
|
+
cannot silently change without detection.
|
|
8
|
+
|
|
9
|
+
## Why
|
|
10
|
+
|
|
11
|
+
`Configuration::SCHEMA` has `auto_download: true` by default. The three
|
|
12
|
+
caches pull arbitrary bytes over HTTPS with zero integrity verification.
|
|
13
|
+
For CI/CD use (SARIF pipelines), supply-chain attacks (a compromised
|
|
14
|
+
GitHub repo swapping in a malicious dictionary), and reproducibility,
|
|
15
|
+
this is unacceptable. SLSA / supply-chain-attestation norms now expect
|
|
16
|
+
this baseline.
|
|
17
|
+
|
|
18
|
+
## Tasks
|
|
19
|
+
|
|
20
|
+
1. **Manifest format.** Define `manifest.json` at the root of each
|
|
21
|
+
content repo:
|
|
22
|
+
```json
|
|
23
|
+
{
|
|
24
|
+
"version": 1,
|
|
25
|
+
"generated_at": "2026-06-25T10:00:00Z",
|
|
26
|
+
"resources": {
|
|
27
|
+
"en/spelling/index.dic": {
|
|
28
|
+
"size": 49568,
|
|
29
|
+
"sha256": "ab12...",
|
|
30
|
+
"language": "en",
|
|
31
|
+
"type": "spelling",
|
|
32
|
+
"license": "LGPL/MPL/GPL",
|
|
33
|
+
"source": "SCOWL"
|
|
34
|
+
},
|
|
35
|
+
}
|
|
36
|
+
}
|
|
37
|
+
}
|
|
38
|
+
```
|
|
39
|
+
2. **Manifest fetching.** The caches fetch `manifest.json` first, then
|
|
40
|
+
individual resources. Resources without a manifest entry are refused.
|
|
41
|
+
3. **SHA-256 verification.** After download, recompute SHA-256 and
|
|
42
|
+
compare. Mismatch raises `Kotoshu::IntegrityError` with expected vs.
|
|
43
|
+
actual in the message.
|
|
44
|
+
4. **Signing (stretch).** Sign `manifest.json` with a Kotoshu project
|
|
45
|
+
key (minisign or sigstore). The gem bundles the public key; CI
|
|
46
|
+
verifies the signature. Stretch goal — checksum-only is the v1
|
|
47
|
+
baseline.
|
|
48
|
+
5. **Audit log.** Every download is logged to
|
|
49
|
+
`~/.kotoshu/audit.log` with timestamp, URL, size, SHA-256, source
|
|
50
|
+
manifest hash. Lets users inspect what was fetched.
|
|
51
|
+
6. **Pin/lockfile support.** `Kotoshu.configure { |c| c.resource_pin =
|
|
52
|
+
"v1.2.0" }` resolves against a specific content repo tag instead of
|
|
53
|
+
`main`. Required for reproducible CI.
|
|
54
|
+
7. **Rate limiting.** Respect GitHub's rate limits (60/hr unauthenticated,
|
|
55
|
+
5000/hr with token). Add exponential backoff and a clear error when
|
|
56
|
+
rate-limited. Support `KOTOSHU_GITHUB_TOKEN` env var.
|
|
57
|
+
8. **TLS pinning (optional).** Document that GitHub's CDN cert pin is
|
|
58
|
+
out of scope for v1 but the gem always verifies TLS chain.
|
|
59
|
+
9. **Threat model doc.** Add `SECURITY.md` documenting: what we verify
|
|
60
|
+
(checksums), what we don't (signing is stretch), how to report a
|
|
61
|
+
compromised resource.
|
|
62
|
+
|
|
63
|
+
## Acceptance criteria
|
|
64
|
+
|
|
65
|
+
- Tampering with a single byte in a cached file causes the next
|
|
66
|
+
`Kotoshu.check` to refuse to load it
|
|
67
|
+
- `Kotoshu.check(text, resource_pin: "v1.0.0")` resolves against
|
|
68
|
+
`dictionaries` repo tag `v1.0.0`
|
|
69
|
+
- `~/.kotoshu/audit.log` records every download
|
|
70
|
+
- CI runs without hitting GitHub rate limits on a fresh runner
|
|
71
|
+
- `SECURITY.md` exists and is linked from the README
|
|
72
|
+
|
|
73
|
+
## Dependencies
|
|
74
|
+
|
|
75
|
+
- Blocks: `03-dynamic-download` (ResourceManager needs this to verify),
|
|
76
|
+
`11-release`
|
|
77
|
+
- Cross-repo: each content repo must ship `manifest.json`:
|
|
78
|
+
- `dictionaries/TODO.impl/01-manifest-checksums.md`
|
|
79
|
+
- `frequency-list-kelly/TODO.impl/02-data-validation.md`
|
|
80
|
+
- `models-fasttext-onnx/TODO.impl/03-manifest-checksums.md`
|
|
81
|
+
|
|
82
|
+
## Out of scope
|
|
83
|
+
|
|
84
|
+
- Full SLSA Level 3 attestation (overkill for v1)
|
|
85
|
+
- Mirroring/CDN strategy (GitHub raw content is fine for v1)
|
|
86
|
+
- Encrypting cached files at rest (user's filesystem is their boundary)
|
|
87
|
+
|
|
88
|
+
## Status
|
|
89
|
+
|
|
90
|
+
_Pending._
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
# 10 — Testing & CI
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
A test suite that meaningfully verifies behavior (not just that methods
|
|
6
|
+
are called) and a CI pipeline that prevents regressions across the
|
|
7
|
+
supported Ruby version and language matrix.
|
|
8
|
+
|
|
9
|
+
## Why
|
|
10
|
+
|
|
11
|
+
Current state:
|
|
12
|
+
- `:network` specs are gated behind `NETWORK_TESTS=1`, meaning default
|
|
13
|
+
`bundle exec rspec` skips large chunks of behavior
|
|
14
|
+
- No CI workflow visible in the repo for the gem itself
|
|
15
|
+
- SimpleCov is configured but `minimum_coverage` is commented out
|
|
16
|
+
(`# TODO: Re-enable when coverage increases`)
|
|
17
|
+
- No performance regression gates
|
|
18
|
+
- Many root-level `test_*.rb` files bypass RSpec entirely
|
|
19
|
+
|
|
20
|
+
## Tasks
|
|
21
|
+
|
|
22
|
+
1. **CI workflow.** Add `.github/workflows/ci.yml`:
|
|
23
|
+
- Matrix: Ruby 3.1, 3.2, 3.3
|
|
24
|
+
- Job 1: `bundle exec rake` (default = spec + rubocop)
|
|
25
|
+
- Job 2 (nightly, separate): `NETWORK_TESTS=1 bundle exec rspec` —
|
|
26
|
+
uses cached resources to avoid re-downloading
|
|
27
|
+
- Job 3: `bundle exec rspec spec/performance/` — runs benchmarks,
|
|
28
|
+
fails on >20% regression vs. baseline committed in
|
|
29
|
+
`spec/performance/baseline.json`
|
|
30
|
+
2. **Coverage gate.** Re-enable `minimum_coverage` once it's at a
|
|
31
|
+
meaningful number. Target: 80% line coverage of `lib/` excluding
|
|
32
|
+
`lib/kotoshu/cli/` (interactive UI is hard to test).
|
|
33
|
+
3. **Spec hygiene.**
|
|
34
|
+
- No `double()` anywhere (project rule). Audit and replace with real
|
|
35
|
+
instances or `Struct`.
|
|
36
|
+
- No VCR / mocks of HTTP (commit `90aa886` already removed these;
|
|
37
|
+
ensure they don't come back). Network specs use a Sinatra fake
|
|
38
|
+
server fixture.
|
|
39
|
+
- Every spec under `spec/kotoshu/` mirrors `lib/kotoshu/` 1:1.
|
|
40
|
+
4. **Property-based tests.** Expand `spec/properties/`:
|
|
41
|
+
- Tokenizer round-trip: every word a tokenizer extracts can be
|
|
42
|
+
re-inserted at its offset to reconstruct the original
|
|
43
|
+
- Suggestion idempotency: a correct word never appears in
|
|
44
|
+
suggestions for itself
|
|
45
|
+
- Hunspell fixture port: every Spylls test scenario is covered
|
|
46
|
+
5. **Integration test matrix.** A single
|
|
47
|
+
`spec/integration/matrix_spec.rb` iterates each supported language
|
|
48
|
+
and runs 5 sample documents through the full pipeline (tokenize →
|
|
49
|
+
resolve → analyze → suggest → format). Tags: `:network` (downloads),
|
|
50
|
+
`:slow` (>5s). Lets us see at a glance which languages are healthy.
|
|
51
|
+
6. **SARIF validation.** Spec that runs `check` and feeds the SARIF
|
|
52
|
+
output through the official SARIF validator gem; fail CI if invalid.
|
|
53
|
+
7. **Move root scratch files.** `test_affix*.rb`, `test_onnx_*.rb`,
|
|
54
|
+
`test_vocab_setup.rb`, `test_combined_cache.rb`, `verify_onnx_all.rb`,
|
|
55
|
+
`deploy_onnx_all_languages.rb`, `monitor_onnx_progress.sh` — move
|
|
56
|
+
useful ones into `scripts/`, delete pure scratch (ask first per
|
|
57
|
+
global rule).
|
|
58
|
+
8. **Conformance harness.** A script that runs Kotoshu against the
|
|
59
|
+
Hunspell reference test corpus and reports pass rate. Becomes the
|
|
60
|
+
source of truth for plan `01-hunspell-correctness` status.
|
|
61
|
+
9. **Flake detection.** CI runs the full suite 3 times; any example
|
|
62
|
+
that fails non-deterministically is flagged.
|
|
63
|
+
|
|
64
|
+
## Acceptance criteria
|
|
65
|
+
|
|
66
|
+
- PRs to `main` cannot merge if `bundle exec rake` fails
|
|
67
|
+
- Coverage report shows ≥ 80% on `lib/` (excluding CLI)
|
|
68
|
+
- `NETWORK_TESTS=1` job runs nightly, posts a summary of
|
|
69
|
+
language-matrix results to the repo's Actions tab
|
|
70
|
+
- No `double()` calls in the codebase (`grep -rn 'double(' spec/` is
|
|
71
|
+
empty)
|
|
72
|
+
- No `test_*.rb` files in the repo root
|
|
73
|
+
|
|
74
|
+
## Dependencies
|
|
75
|
+
|
|
76
|
+
- None — can run in parallel with everything else
|
|
77
|
+
- Unlocks: confidence for `11-release`
|
|
78
|
+
|
|
79
|
+
## Out of scope
|
|
80
|
+
|
|
81
|
+
- Fuzzing (out of scope for v1)
|
|
82
|
+
- Mutation testing (`mutant` gem) — stretch goal
|
|
83
|
+
- Stress/load testing of a future HTTP server
|
|
84
|
+
|
|
85
|
+
## Status
|
|
86
|
+
|
|
87
|
+
_Pending._
|
|
88
|
+
|
|
89
|
+
## Source
|
|
90
|
+
|
|
91
|
+
`docs/TDD_ITERATION_STRATEGY.md` (original dated 2025-01-29) is the
|
|
92
|
+
methodology reference: TDD + MDD cycle, walking skeleton, 10-week
|
|
93
|
+
build-out, Definition of Done per vertical slice. The build-out is
|
|
94
|
+
substantially executed (walking skeleton, core models, all dictionary
|
|
95
|
+
backends, suggestion strategies, Hunspell port all exist with specs).
|
|
96
|
+
Load-bearing rules from that doc — no doubles, test-behavior-not-
|
|
97
|
+
implementation, Strangler Fig for large refactors — live in
|
|
98
|
+
`~/.claude/CLAUDE.md` and `CLAUDE.md`, not duplicated here.
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Plan 100 — Full-feature language batch 3: the highest-audience remainder
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
20 languages are full-feature (gem module + keyboard layout + verified
|
|
5
|
+
specimens); the dictionaries manifest and the model registry carry ~35 more
|
|
6
|
+
with both resources. Every remaining language without a module leaves
|
|
7
|
+
speakers on the degraded path. The gaps are the big-audience ones: ar (~400M),
|
|
8
|
+
id (~200M), fa (~80M), he, bg, sr, hr, sk, sl, and more.
|
|
9
|
+
|
|
10
|
+
## Work (gem repo only — the site flip rides a follow-up)
|
|
11
|
+
1. Enumerate: languages present in BOTH the dictionaries manifest and the
|
|
12
|
+
models registry that lack a gem module. Rank by speaker count.
|
|
13
|
+
2. For the top ~12 (expected: ar id fa he bg sr hr sk sl lt lv et — confirm
|
|
14
|
+
against data): create the language module (pattern of the 19 from PR
|
|
15
|
+
#123), keyboard layout (Arabic 101 for ar; Persian for fa; Hebrew for he;
|
|
16
|
+
Cyrillic/YUSI for bg/sr; QWERTZ/Slovak for the Latinate), AVAILABLE_LANGUAGES
|
|
17
|
+
extension, and specimen pairs verified against the STAGED dictionary via
|
|
18
|
+
the real engine (correct()/suggest() round trips) — the PR #123 discipline.
|
|
19
|
+
3. RTL already wired (ar/id from batch 2); no tokenizer work needed.
|
|
20
|
+
4. Suite green; rubocop clean.
|
|
21
|
+
|
|
22
|
+
## Verification
|
|
23
|
+
One spec per module (specimen corrections engine-verified, layout smoke);
|
|
24
|
+
`kotoshu setup <lang>` + `check -l <lang>` exercised live for at least ar,
|
|
25
|
+
id, fa, he. Deliver the verified specimen table + sizes for the site flip.
|
|
26
|
+
|
|
27
|
+
## Status
|
|
28
|
+
Executed 2026-09-07 — gem PR #151 merged. Twelve languages (ar id fa he bg sr hr sk sl lt lv et) full-feature; AVAILABLE_LANGUAGES 20→32; 49 specimens; RTL tokenizer fix for ar/fa/he. Site flip PR #27.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Plan 101 — int4 tier experiment (models repo), eval-gated
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
The tier recipe settled at int8 (rank_corr 0.9999, near-lossless) after SVD
|
|
5
|
+
failed its gates. int4 halves the bytes again — mini ~1.5 MB, fluency ~8 MB —
|
|
6
|
+
which changes browser adoption math for the wasm playground. It was named in
|
|
7
|
+
the original kotoshu-rs TODO (B1) but never run through the gates.
|
|
8
|
+
|
|
9
|
+
## Work (models repo)
|
|
10
|
+
1. Quantization pass: per-row int4 (packed nibbles) + fp16 scale, over the
|
|
11
|
+
same vocab cuts as fluency/mini.
|
|
12
|
+
2. The existing eval gates unchanged and unweakened: rank correlation vs
|
|
13
|
+
full float32 on the keyboard-aware eval; accept only if rank_corr within
|
|
14
|
+
0.001 of int8's and top-1 agreement holds on the probe-limited languages.
|
|
15
|
+
3. If gates pass: registry v1.4.0 adding the int4 variants as NEW resources
|
|
16
|
+
(`{lang}/mini4`? or a field) — additive, never replacing; mirror with
|
|
17
|
+
ACAO:* like v1.2.1+. If gates fail: record the numbers as the verdict
|
|
18
|
+
(the SVD precedent — measured, not assumed).
|
|
19
|
+
4. ONNX reader side (kotoshu-rs) only if the gates pass.
|
|
20
|
+
|
|
21
|
+
## Verification
|
|
22
|
+
Gates table per language; byte sizes; the release registry byte-identical to
|
|
23
|
+
the committed one; no existing tier URLs change.
|
|
24
|
+
|
|
25
|
+
## Status
|
|
26
|
+
Executed 2026-09-07 — REJECTED on data (models PR #17, merged). int4-per-row
|
|
27
|
+
(one fp16 scale, packed nibbles) over 8 languages: rank_corr deltas -0.0146
|
|
28
|
+
to -0.0199 vs int8 (15-20x outside the 0.001 window), top-1 fails on 8/8
|
|
29
|
+
fluency4 and 2/8 mini4. Sizes halve (mini4 1.52 MB, fluency4 7.60 MB) but
|
|
30
|
+
accuracy does not hold. With the earlier group-128/64/32 sweep the scale
|
|
31
|
+
ladder is measured end to end — int8-per-row stands final. Registry
|
|
32
|
+
untouched; evidence in models eval/reports/.
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Plan 102 — Language detection end to end: registry mirror, wasm, playground
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
The gem detects document language (server /v1/detect) using its own detection
|
|
5
|
+
model — but the registry mirrors none of it, the wasm surface has no detect
|
|
6
|
+
API, and the playground makes a user pick a language dropdown blind. A
|
|
7
|
+
writer pasting text should be told what it is. The FastText LID model is
|
|
8
|
+
tiny (~1 MB) and the mirror pattern is proven (v1.2.1+).
|
|
9
|
+
|
|
10
|
+
## Work
|
|
11
|
+
1. Models repo: add the LID model as a registry resource (converted like the
|
|
12
|
+
tiers, int8-quantized if gates hold vs the float32 labels), mirrored with
|
|
13
|
+
ACAO:*; registry v1.4.x alongside plan 101 (coordinate the version).
|
|
14
|
+
2. kotoshu-rs: `detect_language(bytes, vocab_bytes) -> {code, score}` on the
|
|
15
|
+
model surface beside loadModel/rerank; wasm export `detectLanguage`;
|
|
16
|
+
tests against the gem's detection outputs on a fixed multilingual corpus.
|
|
17
|
+
3. Playground (rides the site follow-up with plans 99/100 flips): on load
|
|
18
|
+
and on demand ("detect" affordance beside the lang select), load the LID
|
|
19
|
+
model (~1 MB, cached like the semantic tier) and set/select the language,
|
|
20
|
+
showing the detected code + confidence. Never auto-switch a user's
|
|
21
|
+
explicit choice silently — propose it.
|
|
22
|
+
|
|
23
|
+
## Verification
|
|
24
|
+
Registry mirror ACAO + size checks; wasm parity vs gem detect on the corpus;
|
|
25
|
+
playground verified with multilingual paste (en/de/ru/ja/ar).
|
|
26
|
+
|
|
27
|
+
## Status
|
|
28
|
+
Executed 2026-09-07 (core) — models PR #18 (lid-176, registry v1.4.0) + kotoshu-rs PR #25 (loadLid/detectLanguage). Playground propose-UI still pending a wasm cut that ships the export.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Plan 103 — OOV bucket-table artifacts: model-generated candidates for the last mile
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
semanticSuggest embeds an OOV word via subword n-grams PRESENT IN THE VOCAB;
|
|
5
|
+
a typo whose n-grams never appear (Teh — only n-gram "teh") embeds nothing
|
|
6
|
+
and returns empty. fastText's real OOV path hashes n-grams into buckets the
|
|
7
|
+
training wrote to; exporting the bucket table closes that gap so the model,
|
|
8
|
+
not just the Damerau sweep, can reach the intended word. Recorded as the
|
|
9
|
+
honest gap in kotoshu-rs oov.rs docs and PR #18's report.
|
|
10
|
+
|
|
11
|
+
## Work (models repo, then kotoshu-rs)
|
|
12
|
+
1. Export the bucket-table rows the FastText training produced (hash bucket
|
|
13
|
+
→ vector) for each tier alongside the vocab; size audit first — the
|
|
14
|
+
bucket table may exceed the tier itself; if so, gate to the top-K buckets
|
|
15
|
+
by usage and measure the quality delta honestly.
|
|
16
|
+
2. kotoshu-rs: extend the int8 reader to consult bucket rows for n-grams
|
|
17
|
+
absent from vocab (feature-gated like `model`); semantic_neighbors then
|
|
18
|
+
covers bucket-backed OOV.
|
|
19
|
+
3. Gates: existing rerank conformance must not move (frozen); new specs for
|
|
20
|
+
bucket-backed neighbors (Teh→the expected); eval on the rerank probe set.
|
|
21
|
+
|
|
22
|
+
## Verification
|
|
23
|
+
Bucket-backed neighbor specs; size deltas recorded; rerank conformance 0
|
|
24
|
+
change; a 0.4.x wasm release only after both engines agree.
|
|
25
|
+
|
|
26
|
+
## Status
|
|
27
|
+
Executed 2026-09-07 — models PR #20 (en/de buckets) + kotoshu-rs PR #26 (bucket-backed OOV). Teh→the via buckets. Release assets + wasm 0.4.0 pin still owner/coordinated cut. (sequenced AFTER plan 101 — same repo, one worktree at a time)
|