kotoshu 1.0.2-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,281 @@
|
|
|
1
|
+
# Kotoshu Vision & Master Plan
|
|
2
|
+
|
|
3
|
+
> **Execution roadmap**: [65-universal-kotoshu.md](65-universal-kotoshu.md)
|
|
4
|
+
> is the master plan to "full success" — one Rust core with TS/Ruby/
|
|
5
|
+
> Python/Rust interfaces ([66-kotoshu-core.md](66-kotoshu-core.md),
|
|
6
|
+
> the parsanol blueprint: feature-gated FFI in the core, thin cdylib
|
|
7
|
+
> shims per language, `ort` for ONNX), tiered micro-models (mini ≈2 MB
|
|
8
|
+
> / fluency ≈14 MB / full ≈120 MB, benchmarked and labeled), and
|
|
9
|
+
> distribution on every channel (registries, editor marketplaces, OS
|
|
10
|
+
> packages, containers, signed offline bundles). Grounded in Microsoft
|
|
11
|
+
> Word's proof that a 14 MB signed ONNX fluency model is world-class.
|
|
12
|
+
> This file remains the philosophy; 65 is the road.
|
|
13
|
+
|
|
14
|
+
## What Kotoshu is becoming
|
|
15
|
+
|
|
16
|
+
**A pure-Ruby spellchecker that works for every language by dynamically
|
|
17
|
+
downloading the right dictionary + frequency data + embedding model on
|
|
18
|
+
demand.**
|
|
19
|
+
|
|
20
|
+
Three load-bearing promises:
|
|
21
|
+
|
|
22
|
+
1. **Pure Ruby workflow.** No shelling out to `hunspell`, `aspell`, or any
|
|
23
|
+
external binary. The only native dependency is `onnxruntime` (for embedding
|
|
24
|
+
inference). Every other piece — affix parsing, compounding, suggestion
|
|
25
|
+
algorithms, language detection, document parsing, interactive UI, output
|
|
26
|
+
formatting — is Ruby that ships in the gem.
|
|
27
|
+
2. **Dynamic resource resolution.** Given any text, Kotoshu detects the
|
|
28
|
+
language, resolves the dictionary + frequency list + embedding model +
|
|
29
|
+
grammar rules that language needs, downloads them on first use into
|
|
30
|
+
`~/.kotoshu/`, verifies their integrity, and caches with TTL. The user
|
|
31
|
+
never installs dictionaries manually.
|
|
32
|
+
3. **All languages.** Latin scripts (Hunspell morphological rules), CJK
|
|
33
|
+
(morphological tokenizers + confusion rules — no traditional spellcheck),
|
|
34
|
+
RTL (shaping-aware affix rules). A pluggable per-language module system
|
|
35
|
+
decides what each language uses.
|
|
36
|
+
|
|
37
|
+
## Three analysis dimensions, composed per language
|
|
38
|
+
|
|
39
|
+
| Dimension | Source | Role |
|
|
40
|
+
|---|---|---|
|
|
41
|
+
| Traditional | Hunspell `.aff`/`.dic` | Authoritative correctness, morphological generation |
|
|
42
|
+
| Semantic | FastText ONNX embeddings | Context-aware reranking, OOV handling |
|
|
43
|
+
| Grammar | `rules.yaml` per language | Style/usage rules LanguageTool-style |
|
|
44
|
+
|
|
45
|
+
Each language module declares which dimensions it supports; the pipeline
|
|
46
|
+
composes whatever is available.
|
|
47
|
+
|
|
48
|
+
## The five repos and what they're for
|
|
49
|
+
|
|
50
|
+
| Repo | Role |
|
|
51
|
+
|---|---|
|
|
52
|
+
| `kotoshu/kotoshu` | The gem (this one). Ruby API + CLI + analysis pipeline |
|
|
53
|
+
| `kotoshu/dictionaries` | Hunspell dictionaries, 98 language dirs |
|
|
54
|
+
| `kotoshu/frequency-list-kelly` | Kelly Project frequency tiers for ranking |
|
|
55
|
+
| `kotoshu/models-fasttext-onnx` | FastText embeddings converted to ONNX |
|
|
56
|
+
| `kotoshu/kotoshu.github.io` | Marketing/docs site |
|
|
57
|
+
|
|
58
|
+
The gem depends on the other four as **content repos** — never vendored, always
|
|
59
|
+
fetched on demand by the cache layer.
|
|
60
|
+
|
|
61
|
+
## Plan index for this repo
|
|
62
|
+
|
|
63
|
+
| # | Plan | Why |
|
|
64
|
+
|---|---|---|
|
|
65
|
+
| 01 | [Hunspell correctness](01-hunspell-correctness.md) | Core lookup is ~50% on Spylls fixtures — must be fixed before anything trusts it |
|
|
66
|
+
| 02 | [CLI unification](02-cli-unification.md) | Two competing `check` commands; only the basic one is wired |
|
|
67
|
+
| 03 | [Dynamic resource download](03-dynamic-download.md) | The promise — unified resolver for any language |
|
|
68
|
+
| 04 | [Language modules](04-language-modules.md) | Wire 93+ dictionaries into the language module system |
|
|
69
|
+
| 05 | [Semantic path productionization](05-semantic-path.md) | Make ONNX reranking always-available, memory-bounded |
|
|
70
|
+
| 06 | [CJK support](06-cjk-support.md) | Japanese + Chinese (different paradigm — tokenizers + confusion) |
|
|
71
|
+
| 07 | [RTL support](07-rtl-support.md) | Arabic, Hebrew, Persian, Urdu |
|
|
72
|
+
| 08 | [Grammar engine](08-grammar-engine.md) | LanguageTool-style rule loader and pattern matchers |
|
|
73
|
+
| 09 | [Integrity & security](09-integrity-security.md) | Checksums, signed releases, audit log for downloads |
|
|
74
|
+
| 10 | [Testing & CI](10-testing-ci.md) | Coverage targets, CI matrix, perf benchmarks |
|
|
75
|
+
| 11 | [Release v1.0](11-release-v1.md) | Tag, publish, CHANGELOG, supported-language matrix |
|
|
76
|
+
|
|
77
|
+
## Cross-repo dependency order
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
09-integrity ─┐
|
|
81
|
+
03-download ──┼─► 04-language-modules ─► 06-cjk ─┐
|
|
82
|
+
│ ├─► 08-grammar ─► 11-release
|
|
83
|
+
├─► 01-hunspell ──────────────────┤
|
|
84
|
+
└─► 05-semantic ───────────────────┘
|
|
85
|
+
|
|
86
|
+
02-cli, 07-rtl, 10-testing can run in parallel with the above
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
| Plan 03 (dynamic download) is the unlock — it depends on 09 (integrity) and
|
|
90
|
+
unblocks both 04 (language modules) and the user-facing "any language"
|
|
91
|
+
promise. Plans 01 (Hunspell) and 05 (semantic) can proceed independently.
|
|
92
|
+
|
|
93
|
+
Plans 60–64 are the **ecosystem / reach** track — they turn the working
|
|
94
|
+
gem into something users actually adopt. They live in their own repos
|
|
95
|
+
under `kotoshu/` per the five-repo workspace model:
|
|
96
|
+
|
|
97
|
+
| # | Plan | Why |
|
|
98
|
+
|---|---|---|
|
|
99
|
+
| 60 | [LSP server](60-lsp-server.md) | One server, every editor. Highest-leverage reach. |
|
|
100
|
+
| 61 | [Editor & CI ecosystem](61-editor-ecosystem.md) | VS Code, Vim, JetBrains, GitHub Action, pre-commit hooks |
|
|
101
|
+
| 62 | [Framework integrations](62-framework-integrations.md) | Rails validators, Jekyll rake tasks, RSpec matchers — inside the library gem |
|
|
102
|
+
| 63 | [Web playground (WASM)](63-web-playground-wasm.md) | Browser playground at `kotoshu.github.io/playground/` |
|
|
103
|
+
| 64 | [HTTP API & SDKs](64-http-api-and-sdks.md) | Self-hostable server + Python / JS / Go SDKs for non-Ruby stacks |
|
|
104
|
+
|
|
105
|
+
Plan 67 is the **delivery train** that publishes 65/66: tiered models
|
|
106
|
+
(mini/fluency/full) through the models repo's registry + releases, the
|
|
107
|
+
kotoshu-rs bootstrap and port, and the Ruby / WASM / Python access
|
|
108
|
+
libraries — [67-kotoshu-rs-and-access-libraries.md](67-kotoshu-rs-and-access-libraries.md).
|
|
109
|
+
|
|
110
|
+
Plan 68 is the **research adoption plan** (2026-09-02 SOTA survey):
|
|
111
|
+
what modern GEC/embedding/quantization research and frontier-model
|
|
112
|
+
training techniques Kotoshu adopts, adapts, or rejects —
|
|
113
|
+
[68-sota-adoption.md](68-sota-adoption.md). Its adopt-now items are
|
|
114
|
+
executed by plans
|
|
115
|
+
[69 (keyboard typo eval + corpus benchmark)](69-sota-typo-eval.md),
|
|
116
|
+
[70 (confidence cascade)](70-sota-confidence-cascade.md), and
|
|
117
|
+
[71 (nested single-artifact experiment)](71-sota-nested-model.md).
|
|
118
|
+
The remaining SOTA items are planned as
|
|
119
|
+
[72 (Model2Vec evaluation)](72-model2vec-evaluation.md),
|
|
120
|
+
[73 (RL-trained reranker, gated)](73-rl-reranker.md),
|
|
121
|
+
[74 (distilled reranker, gated)](74-distilled-reranker.md),
|
|
122
|
+
[75 (LLM-as-evaluator)](75-llm-evaluator.md), and
|
|
123
|
+
[76 (optional external LLM backend, gated on demand)](76-optional-llm-backend.md).
|
|
124
|
+
|
|
125
|
+
Plans 77–82 are the **adoption wave** (owner ask 2026-09-05: are the
|
|
126
|
+
models comprehensive, how do we cover more places, docs + news):
|
|
127
|
+
closing the verified user gaps after all channels went live.
|
|
128
|
+
|
|
129
|
+
| # | Plan | Gap it closes |
|
|
130
|
+
|---|---|---|
|
|
131
|
+
| 77 | [Model coverage expansion](77-model-coverage-expansion.md) | Semantic models cover 9 languages vs 98 dictionaries — expand to 25+ |
|
|
132
|
+
| 78 | [In-browser playground](78-in-browser-playground.md) | /playground needs a running server; @kotoshu/wasm enables zero-install |
|
|
133
|
+
| 79 | [News + releases](79-news-and-releases.md) | No news surface on the site; seven registries shipped invisibly |
|
|
134
|
+
| 80 | [Ecosystem docs](80-ecosystem-docs.md) | Docs are gem-only; pip/npm/cargo/go/HTTP/LSP/Action users have no path in |
|
|
135
|
+
| 81 | [Editor reach](81-editor-reach.md) | kotoshu-lsp shipped but no VS Code package; no pre-commit hook (supersedes 61 remainder) |
|
|
136
|
+
| 82 | [Document ignores + baselines](82-doc-formats-and-ignores.md) | No inline suppressions; no CI baseline freeze |
|
|
137
|
+
|
|
138
|
+
Plans 83–89 are **wave 2** (2026-09-05, owner re-ask after the wave-1
|
|
139
|
+
ship): driving models toward full intersection coverage, making the
|
|
140
|
+
22 model languages full-feature, semantic reranking in the browser,
|
|
141
|
+
and the site/docs truth that follows.
|
|
142
|
+
|
|
143
|
+
| # | Plan | Gap it closes |
|
|
144
|
+
|---|---|---|
|
|
145
|
+
| 83 | [Models batch 2](83-models-batch-2.md) | ar/fa/he have modules but no models; ~50 more intersection languages; fi/id dictionary sourcing |
|
|
146
|
+
| 84 | [Language modules + keyboards wave 2](84-language-modules-wave2.md) | 13 model languages lack gem modules; 11 keyboard layouts missing |
|
|
147
|
+
| 85 | [Wasm model API](85-wasm-model-api.md) | Browser semantic reranking — no model API on the wasm surface, CORS distribution gap |
|
|
148
|
+
| 86 | [Site truth wave 2](86-site-truth-wave2.md) | Ignores/baselines undocumented; no Windows page; wrong personal.dic path in CLI docs; playground breadth |
|
|
149
|
+
| 87 | [LSP personal dictionary](87-lsp-personal-dictionary.md) | Server never reads personal.dic — add-to-dictionary appears broken |
|
|
150
|
+
| 88 | [CLI directory mode](88-cli-directory-mode.md) | No tree checking — the actual CI workflow |
|
|
151
|
+
| 89 | [Framework integrations](89-framework-integrations-exec.md) | Executes plan 62: Rails validator, RSpec matcher, Rake/Jekyll tasks |
|
|
152
|
+
|
|
153
|
+
Plans 91–95 are **wave 3** (2026-09-05 late): the correctness core the
|
|
154
|
+
site flip exposed, the browser-models distribution fix, Python wheels,
|
|
155
|
+
the docs for wave-2 features, and editor/action truth.
|
|
156
|
+
|
|
157
|
+
| # | Plan | Gap it closes |
|
|
158
|
+
|---|---|---|
|
|
159
|
+
| 91 | [Unicode + sv aff](91-unicode-and-sv-aff.md) | Executes plan 90: ASCII-only word detection hides non-Latin scripts; AffReader crashes on Swedish |
|
|
160
|
+
| 92 | [Tier mirror CORS](92-tier-mirror-cors.md) | Media host already sends ACAO — mirror the tiers, finish the browser-models path |
|
|
161
|
+
| 93 | [Python wheels](93-python-wheels.md) | kotoshu-native is macOS-only on PyPI; build the platform matrix + keyless publish |
|
|
162
|
+
| 94 | [Site integrations docs](94-site-integrations-docs.md) | Rails/RSpec/Rake/Jekyll + directory mode shipped undocumented; action page stale |
|
|
163
|
+
| 95 | [Editor + action polish](95-editor-action-polish.md) | Extension delegates the now-existing server command; action gains baseline + 19 languages |
|
|
164
|
+
|
|
165
|
+
Plans 96–97 are **wave 4** (2026-09-06): the audit found the wave-1–3
|
|
166
|
+
surface complete except two honest items — the HTTP server never
|
|
167
|
+
loads models, and plan 59's YARD reference is still unpublished. (The
|
|
168
|
+
Rust engine was audited too: its AST-based aff parser is immune to
|
|
169
|
+
the Swedish COMPOUNDRULE bug by design.)
|
|
170
|
+
|
|
171
|
+
| # | Plan | Gap it closes |
|
|
172
|
+
|---|---|---|
|
|
173
|
+
| 96 | [Server semantic models](96-server-semantic-models.md) | /v1/check is dictionary-only — SDK users get no reranking through HTTP |
|
|
174
|
+
| 97 | [YARD execution](97-yard-execution.md) | Executes plan 59: the Ruby API reference still does not exist |
|
|
175
|
+
|
|
176
|
+
Plan 98 is **wave 5, the cleanup and closeout** (2026-09-06): after the
|
|
177
|
+
release-day cuts (gem 0.8.0 → 0.9.0, @kotoshu/wasm 0.2.0, registry
|
|
178
|
+
v1.3.0 with nb, lsp/server 0.1.1 via trusted publishing) the workspace
|
|
179
|
+
was swept — 214 GB of models worktrees reclaimed, the plan corpus
|
|
180
|
+
01–98 fully tracked, verification evidence preserved.
|
|
181
|
+
|
|
182
|
+
| # | Plan | What it records |
|
|
183
|
+
|---|---|---|
|
|
184
|
+
| 98 | [Campaign cleanup](98-campaign-cleanup.md) | The inventory, the executed sweep, the keep list, and the deliberately-unforced items |
|
|
185
|
+
|
|
186
|
+
Campaign state at close: every channel live and populated; the
|
|
187
|
+
convertible model pool exhausted at 55 languages; owner-side
|
|
188
|
+
remnants (PyPI registrations, marketplace PAT, nds/fi calls) recorded
|
|
189
|
+
in the plans, not pending work.
|
|
190
|
+
|
|
191
|
+
Plans 99-104 are **wave 6, the live-quality follow-through**
|
|
192
|
+
(2026-09-07), written after the sweep-performance arc (gem 0.9.1 ->
|
|
193
|
+
0.9.2, @kotoshu/wasm 0.3.0 -> 0.3.2) and a fresh state audit: 55 model
|
|
194
|
+
languages x 3 tiers confirmed (ja/ko/zh rerank-only by design), no LID
|
|
195
|
+
resource mirrored, lsp/server `~> 0.6` constraints already resolving
|
|
196
|
+
current engines, site docs carrying stale version references.
|
|
197
|
+
|
|
198
|
+
| # | Plan | Gap it closes |
|
|
199
|
+
|---|---|---|
|
|
200
|
+
| 99 | [Site truth + performance docs](99-site-truth-performance-docs.md) | Docs misstate current versions; the measured per-language latency is unpublished |
|
|
201
|
+
| 100 | [Full-feature batch 3](100-full-feature-batch3.md) | ~35 languages have dictionaries + models but no gem module - the big-audience remainder (ar id fa he ...) |
|
|
202
|
+
| 101 | [int4 tier experiment](101-int4-tier-experiment.md) | Halving tier bytes was named (B1) but never run through the gates |
|
|
203
|
+
| 102 | [LID chain](102-lid-chain.md) | No detection in the registry, wasm, or playground - users pick the language blind |
|
|
204
|
+
| 103 | [OOV bucket tables](103-oov-bucket-tables.md) | Model candidates miss n-grams absent from vocab (the Teh gap's model half) |
|
|
205
|
+
| 104 | [VS Code marketplace readiness](104-vscode-marketplace-readiness.md) | The extension is built but not installable; everything short of the owner PAT |
|
|
206
|
+
|
|
207
|
+
Execution notes: plan 103 is sequenced after 101 (same repo, one
|
|
208
|
+
worktree at a time); all site-facing flips (99, 100, 102's playground
|
|
209
|
+
half, 104's link line) land through one coordinated site pass to keep
|
|
210
|
+
the repo conflict-free; lsp/server 0.1.2 bumps are deliberately NOT
|
|
211
|
+
planned - the `~> 0.6` floor already resolves 0.9.2 for fresh installs.
|
|
212
|
+
|
|
213
|
+
Dependency order across the ecosystem track:
|
|
214
|
+
|
|
215
|
+
```
|
|
216
|
+
60-lsp ─┬─► 61-editor-ecosystem
|
|
217
|
+
├─► 62-framework-integrations (also depends on 02)
|
|
218
|
+
└─► 64-http-api (also depends on 02, 03, 05, 09)
|
|
219
|
+
63-web-playground is independent (depends only on 02, 04)
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
## Cross-repo plan files
|
|
223
|
+
|
|
224
|
+
Each content repo has its own `TODO.impl/` directory:
|
|
225
|
+
|
|
226
|
+
- `dictionaries/TODO.impl/` — manifest, coverage matrix, releases, metadata, licenses
|
|
227
|
+
- `frequency-list-kelly/TODO.impl/` — coverage extension, validation, releases
|
|
228
|
+
- `models-fasttext-onnx/TODO.impl/` — publishing, storage, manifest, conversion CI
|
|
229
|
+
- `kotoshu.github.io/TODO.impl/` — site skeleton, content, examples landing
|
|
230
|
+
|
|
231
|
+
## Source-of-truth note
|
|
232
|
+
|
|
233
|
+
Older planning docs now live in `docs/`
|
|
234
|
+
(`KOTOSHU_FULL_PLAN.md`, `KOTOSHU_SOLIDIFICATION_PLAN.md`,
|
|
235
|
+
`KOTOSHU_LANGUAGETOOL_GAPS.md`, `ARCHITECTURE_IMPROVEMENTS.md`,
|
|
236
|
+
`ARCHITECTURE_IMPROVEMENTS_PLAN.md`, `TDD_ITERATION_STRATEGY.md`,
|
|
237
|
+
`ONNX_FUNCTIONAL_VERIFICATION.md`). Actionable items have been
|
|
238
|
+
integrated into the relevant execution plans below — pointers in each
|
|
239
|
+
plan name the source.
|
|
240
|
+
|
|
241
|
+
| Source | Integrated into |
|
|
242
|
+
|---|---|
|
|
243
|
+
| `docs/KOTOSHU_FULL_PLAN.md` | Phases 0–8 spread across `01`–`11` per phase scope |
|
|
244
|
+
| `docs/KOTOSHU_SOLIDIFICATION_PLAN.md` (49 OOP items) | `01` (Parts 1–2 Hunspell/CSpell), `04`/`06`/`07` (Part 3 LanguageTool), `10` (Part 5 testing); Part 4 perf items live in the relevant feature plan |
|
|
245
|
+
| `docs/KOTOSHU_LANGUAGETOOL_GAPS.md` | `08` (rules, confusion), `02` (CLI surface, output formats), `04` (target languages) |
|
|
246
|
+
| `docs/ARCHITECTURE_IMPROVEMENTS.md` (15 patterns) | Most patterns are architectural guidance applied across plans; load-bearing rules captured in `CLAUDE.md` |
|
|
247
|
+
| `docs/ARCHITECTURE_IMPROVEMENTS_PLAN.md` (5 phases) | Phase 1 perf items in `01`/`05`; Phase 2 architecture in `00-vision` (this doc); Phase 3 UX mostly implemented (`PersonalDictionary`, `ProjectConfig`, `FluentChecker` exist); Phase 4 quality in `10`; Phase 5 polish spread |
|
|
248
|
+
| `docs/TDD_ITERATION_STRATEGY.md` | `10` (test methodology + CI); project-rule overrides (no doubles, behavior-not-implementation) in `CLAUDE.md` |
|
|
249
|
+
| `docs/ONNX_FUNCTIONAL_VERIFICATION.md` | `05` (semantic path productionization) |
|
|
250
|
+
|
|
251
|
+
|
|
252
|
+
Plans 105-107 plus models 09 and site 11 are **wave 7, detection and
|
|
253
|
+
coverage depth** (2026-09-07): after the wave-6 releases (gem 0.9.3,
|
|
254
|
+
wasm 0.4.0, registry v1.4.0, action v2), the remaining gaps are the
|
|
255
|
+
personal dictionary never reaching the check path, detection stuck on
|
|
256
|
+
a 7-language heuristic, buckets only for en/de, and the playground not
|
|
257
|
+
yet passing bucket bytes to loadModel.
|
|
258
|
+
|
|
259
|
+
| # | Plan | Gap it closes |
|
|
260
|
+
|---|---|---|
|
|
261
|
+
| 105 | [Personal dictionary in check](105-personal-dictionary-in-check.md) | kotoshu personal add then check still flags the word |
|
|
262
|
+
| 106 | [Native LID detection](106-native-lid-detection.md) | detect is a 7-language heuristic; the pure-Rust lid reader exists unused |
|
|
263
|
+
| 107 | models [09 bucket expansion](../../models-fasttext-onnx/TODO.impl/09-bucket-table-expansion.md) | bucket OOV siblings exist only for en and de |
|
|
264
|
+
| — | site 11 playground buckets | wasm 0.4.0 accepts bucketsBytes; the worker never fetches the sibling |
|
|
265
|
+
|
|
266
|
+
|
|
267
|
+
Plans 107-109 are **wave 8, reachability and the road to 1.0**
|
|
268
|
+
(2026-09-07): after wave 7 (personal dict in check, 176-language
|
|
269
|
+
detection, buckets rolling to all languages), the remaining user gaps
|
|
270
|
+
are the sixty-plus staged dictionaries unreachable without a module,
|
|
271
|
+
the last big-population languages (ko, ne), and the absence of a
|
|
272
|
+
stability commitment adopters can build on.
|
|
273
|
+
|
|
274
|
+
| # | Plan | Gap it closes |
|
|
275
|
+
|---|---|---|
|
|
276
|
+
| 107 | [Basic support for every staged language](107-basic-support-every-staged-language.md) | setup rejects 60+ staged languages; modules become upgrades, not gates |
|
|
277
|
+
| 108 | [ko and ne full-feature](108-ko-ne-full-feature.md) | Hangul and Devanagari - the last ~100M speakers without support |
|
|
278
|
+
| 109 | [The 1.0 readiness audit](109-ten-audit.md) | no frozen surface or stability policy for adopters |
|
|
279
|
+
|
|
280
|
+
Sequencing: 108 after 107 (same area); 109 is analysis-only and runs in
|
|
281
|
+
parallel. The 1.0 cut itself remains the owner's version call.
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# 01 — Hunspell Correctness
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Pass 100% of the Spylls-ported fixture suite in `spec/hunspell_tests/` and
|
|
6
|
+
the Hunspell reference fixtures under
|
|
7
|
+
`/Users/mulgogi/src/external/hunspell/tests/`. Baseline at the start of
|
|
8
|
+
this plan was **48/107 (45 %)** passing; current state per
|
|
9
|
+
`CHANGELOG.md` is **803/866 (92.7 %)** — close but not done.
|
|
10
|
+
|
|
11
|
+
## Why
|
|
12
|
+
|
|
13
|
+
The lookup/affix/compound machinery is the foundation of the
|
|
14
|
+
traditional path; nothing downstream (suggestions, grammar,
|
|
15
|
+
multi-language) can be trusted until this is correct.
|
|
16
|
+
|
|
17
|
+
## Current blockers
|
|
18
|
+
|
|
19
|
+
- `lib/kotoshu/algorithms/lookup.rb` is a procedural ~877-line Python
|
|
20
|
+
port; hash/array passing instead of domain models.
|
|
21
|
+
- Overlap between `Lookup`, `LookupBuilder`, `Lookuper` (not MECE).
|
|
22
|
+
- Affix data split between `AffReader`, `aff_data.rb`, `lookup.rb`.
|
|
23
|
+
- Missing: complex compounding, circumfix, ICONV/OCONV, German ß,
|
|
24
|
+
Turkish dotless-i, special-character handling.
|
|
25
|
+
|
|
26
|
+
## Tasks
|
|
27
|
+
|
|
28
|
+
1. **Audit current pass rate.** Run `bundle exec rspec spec/hunspell_tests/`
|
|
29
|
+
and `bundle exec rspec spec/integration/`. Record the actual number
|
|
30
|
+
and failing fixture names in "Status" below.
|
|
31
|
+
2. **Compound rules.** Implement `COMPOUNDRULE`, `COMPOUNDFLAG`,
|
|
32
|
+
`COMPOUNDBEGIN/MIDDLE/END`, `COMPOUNDPERMFLAG`, `COMPOUNDMIN`,
|
|
33
|
+
`CHECKCOMPOUNDREP`, `CHECKCOMPOUNDCASE`, `CHECKCOMPOUNDTRIPLE`,
|
|
34
|
+
`CHECKCOMPOUNDDUP`, `FORBIDDENWORD` inside compounds.
|
|
35
|
+
3. **Circumfix.** Bidirectional prefix+suffix via `CIRCUMFIX` flag.
|
|
36
|
+
4. **ICONV / OCONV.** Input/output character conversion tables applied
|
|
37
|
+
at the right point in the pipeline.
|
|
38
|
+
5. **Special characters.** German ß uppercasing, Turkish dotless-i,
|
|
39
|
+
Dutch IJ, Break patterns (`BREAK`), `IGNORE` chars, `CHECKSHARPS`,
|
|
40
|
+
Arabic character ignoring, UTF-8 BOM, legacy encodings
|
|
41
|
+
(Windows-1251).
|
|
42
|
+
6. **MECE refactor of the lookup pipeline.** Replace the procedural
|
|
43
|
+
`lookup.rb` with a four-layer separation (per the reimplementation
|
|
44
|
+
plan):
|
|
45
|
+
- **Dictionary layer** (data structures): `WordRepository`,
|
|
46
|
+
`AffixRepository`, `CompoundRuleRepository`,
|
|
47
|
+
`MorphologicalAnalyzer`.
|
|
48
|
+
- **Lookup layer** (algorithms): `WordLookup`, `AffixLookup`,
|
|
49
|
+
`CompoundLookup`, `MorphologicalLookup`.
|
|
50
|
+
- **Validation layer** (business rules): `ConditionValidator`,
|
|
51
|
+
`FlagValidator`, `CompoundValidator`,
|
|
52
|
+
`CapitalizationValidator`.
|
|
53
|
+
- **Transformation layer** (operations): `AffixApplier`,
|
|
54
|
+
`AffixStripper`, `WordFormer`, `CompoundFormer`.
|
|
55
|
+
Collapse `Lookup`/`LookupBuilder`/`Lookuper` into one responsibility
|
|
56
|
+
boundary or document why three classes exist.
|
|
57
|
+
7. **Domain-model completion.** Verify `Word`, `AffixRule`,
|
|
58
|
+
`CompoundWord`, `Condition` carry the behavior, not the services
|
|
59
|
+
(per the global "models first" rule). Query methods required:
|
|
60
|
+
`word.stem`, `word.has_flag?(:verb)`, `word.forms`,
|
|
61
|
+
`affix_rule.valid_for?(stem)`, `affix_rule.apply(stem)`,
|
|
62
|
+
`affix_rule.can_combine?(other)`, `compound.valid_at?(position)`,
|
|
63
|
+
`condition.matches?(stem)`.
|
|
64
|
+
8. **Strangler-Fig migration.** Build the new pipeline alongside the
|
|
65
|
+
old; expose a configuration switch (e.g.
|
|
66
|
+
`Configuration.use_modern_lookup` defaulting to `false`) so the
|
|
67
|
+
cutover is per-language and reversible. Remove the old
|
|
68
|
+
`algorithms/lookup.rb` only after the new path passes every fixture
|
|
69
|
+
the old one did.
|
|
70
|
+
9. **Port the failing Spylls integrational fixtures** as named spec
|
|
71
|
+
examples: `dotless_i`, `IJ`, `ignore`, `checksharps`, `break`,
|
|
72
|
+
`needaffix`, `onlyincompound`, `alias`, `complexprefixes`,
|
|
73
|
+
`germancompounding`, `checkcompoundpattern`, plus the Hunspell
|
|
74
|
+
reference corpus fixtures (`1463589` Turkish UTF-8, `1592880`,
|
|
75
|
+
`1695964`, `1706659`, `1748408`, `breakdefault`, `breakoff`,
|
|
76
|
+
`encoding`, `utf8`, `utf8_bom`, `utf8_bom2`, `ignoreutf`,
|
|
77
|
+
`ignoresug`, `checksharpsutf`).
|
|
78
|
+
10. **Performance baseline.** Profile lookup, affix application, and
|
|
79
|
+
compound check; add `spec/performance/lookup_bench.rb` with
|
|
80
|
+
assertions.
|
|
81
|
+
|
|
82
|
+
## Acceptance criteria
|
|
83
|
+
|
|
84
|
+
- `bundle exec rspec spec/hunspell_tests/` is green (100 % of Spylls
|
|
85
|
+
fixtures).
|
|
86
|
+
- New specs added for every behavior fixed (no fixture passes
|
|
87
|
+
silently).
|
|
88
|
+
- `lib/kotoshu/algorithms/lookup.rb` is under 400 lines, or its
|
|
89
|
+
responsibilities have been redistributed across the four layers
|
|
90
|
+
above with the boundary documented.
|
|
91
|
+
- **Performance:** < 100 ms average lookup time per 1000 words;
|
|
92
|
+
< 100 MB memory for the en_US dictionary.
|
|
93
|
+
- **Code quality:** ≤ 50 lines per method, ≤ 300 lines per file, no
|
|
94
|
+
primitive obsession (domain models, not raw hashes), SRP followed,
|
|
95
|
+
dependencies on abstractions.
|
|
96
|
+
- No `double()` in any new spec (project rule — see `CLAUDE.md`).
|
|
97
|
+
|
|
98
|
+
## Dependencies
|
|
99
|
+
|
|
100
|
+
- None inside this repo. Blocks `04-language-modules` (which assumes
|
|
101
|
+
a trustworthy Hunspell path) and `11-release`.
|
|
102
|
+
|
|
103
|
+
## Out of scope
|
|
104
|
+
|
|
105
|
+
- Suggestion quality tuning (covered in `05-semantic-path`).
|
|
106
|
+
- Grammar rules (covered in `08-grammar-engine`).
|
|
107
|
+
- New languages beyond what the existing fixture suite covers.
|
|
108
|
+
|
|
109
|
+
## Status
|
|
110
|
+
|
|
111
|
+
_Pending — fill in actual pass rate after task 1._
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# 02 — CLI Unification
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
One canonical CLI surface that exposes every feature the gem supports, with
|
|
6
|
+
flags that match the README.
|
|
7
|
+
|
|
8
|
+
## Why
|
|
9
|
+
|
|
10
|
+
There are two competing implementations:
|
|
11
|
+
|
|
12
|
+
- `lib/kotoshu/cli.rb` — `Kotoshu::Cli::Cli < Thor`. This is what
|
|
13
|
+
`exe/kotoshu` actually starts. Has only a basic `check TARGET` command
|
|
14
|
+
and `dict`/`cache` subcommands.
|
|
15
|
+
- `lib/kotoshu/commands/check_command.rb` — `Kotoshu::CheckCommand < Thor`
|
|
16
|
+
with `namespace :check`, exposes `--language auto`, `--interactive`,
|
|
17
|
+
`--output`, `--format text|json|yaml|csv|sarif`, `--model fasttext|hunspell`,
|
|
18
|
+
`--download`. **Not registered as a subcommand anywhere — dead code.**
|
|
19
|
+
|
|
20
|
+
The README advertises `--interactive`, `--format sarif`, `--model`,
|
|
21
|
+
`--language auto`. Users running `bundle exec exe/kotoshu check README.md
|
|
22
|
+
--interactive` get an "unknown option" error.
|
|
23
|
+
|
|
24
|
+
## Tasks
|
|
25
|
+
|
|
26
|
+
1. **Decide canonical location.** Recommendation: keep the rich
|
|
27
|
+
`Kotoshu::CheckCommand` (it has all the features users want) and delete
|
|
28
|
+
the basic `check` from `cli.rb`. Wire `cli.rb` to register
|
|
29
|
+
`CheckCommand` via `subcommand "check", Kotoshu::CheckCommand` (or
|
|
30
|
+
register at top level if `check` is the only command).
|
|
31
|
+
2. **Move `DictCommand` and `CacheCommand`** into `lib/kotoshu/commands/`
|
|
32
|
+
alongside `CheckCommand` so all CLI commands live in one directory.
|
|
33
|
+
3. **Add a `model` subcommand** so users can manage ONNX models
|
|
34
|
+
(`kotoshu model download en`, `kotoshu model list`,
|
|
35
|
+
`kotoshu model validate`, `kotoshu model convert`). The README
|
|
36
|
+
references these — they live in `lib/kotoshu/commands/model_command.rb`
|
|
37
|
+
already; verify it's wired.
|
|
38
|
+
4. **Add `--from-stdin` mode** so the CLI can be used in shell pipes:
|
|
39
|
+
`echo "wrold" | kotoshu check --from-stdin`.
|
|
40
|
+
5. **Reconcile flag names** between README and code. Specifically:
|
|
41
|
+
- README says `--dictionary=unix_words`; `cli.rb` has both `--dictionary`
|
|
42
|
+
and `--dictionary-path` — pick one.
|
|
43
|
+
- README says `--output=json`; `CheckCommand` uses `--format=json
|
|
44
|
+
--output=FILE` — clarify the two-flag split.
|
|
45
|
+
6. **Exit codes.** Define and document: 0 = clean, 1 = errors found, 2 =
|
|
46
|
+
usage error, 3 = dictionary/model fetch failure. SARIF consumers
|
|
47
|
+
(GitHub Code Scanning) require this contract.
|
|
48
|
+
7. **Help text audit.** Every Thor `desc` should match what's in the
|
|
49
|
+
README, and the README should match what's in code.
|
|
50
|
+
|
|
51
|
+
## Acceptance criteria
|
|
52
|
+
|
|
53
|
+
- `bundle exec exe/kotoshu check README.md --interactive` works as
|
|
54
|
+
documented
|
|
55
|
+
- `bundle exec exe/kotoshu check README.md --format sarif --output
|
|
56
|
+
results.sarif` produces valid SARIF 2.1.0 that GitHub accepts
|
|
57
|
+
- `bundle exec exe/kotoshu --help` lists every command the README mentions
|
|
58
|
+
- No duplicate command classes; one canonical path per command
|
|
59
|
+
- A spec under `spec/kotoshu/cli/` exercises each subcommand end-to-end
|
|
60
|
+
(using a temp dir, not real `~/.kotoshu`)
|
|
61
|
+
|
|
62
|
+
## Dependencies
|
|
63
|
+
|
|
64
|
+
- Light dependency on `03-dynamic-download` (CLI should let users
|
|
65
|
+
pre-download resources), but can proceed independently.
|
|
66
|
+
|
|
67
|
+
## Out of scope
|
|
68
|
+
|
|
69
|
+
- LSP server, HTTP server (future work, not v1)
|
|
70
|
+
- GUI / TUI redesign — keep the existing Thor UX
|
|
71
|
+
|
|
72
|
+
## Status
|
|
73
|
+
|
|
74
|
+
_Pending._
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
# 03 — Dynamic Resource Download
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
One call — `Kotoshu.check(text)` — works for any of the supported
|
|
6
|
+
languages without the user pre-installing anything. The library detects
|
|
7
|
+
language, resolves what's needed, downloads it, verifies integrity, and
|
|
8
|
+
caches.
|
|
9
|
+
|
|
10
|
+
This is the heart of the project's promise.
|
|
11
|
+
|
|
12
|
+
## Why
|
|
13
|
+
|
|
14
|
+
Today the three caches (`LanguageCache`, `FrequencyCache`, `ModelCache`)
|
|
15
|
+
exist independently. The user (or the spellchecker) has to know which to
|
|
16
|
+
call. Configuration is keyed on `dictionary_type` and `language`, but
|
|
17
|
+
there's no unified resolver that takes arbitrary text and yields the full
|
|
18
|
+
resource bundle.
|
|
19
|
+
|
|
20
|
+
## The missing piece: `ResourceManager`
|
|
21
|
+
|
|
22
|
+
A new top-level component (autoloaded from
|
|
23
|
+
`lib/kotoshu/resource_manager.rb`) that owns the contract:
|
|
24
|
+
|
|
25
|
+
```ruby
|
|
26
|
+
bundle = Kotoshu::ResourceManager.resolve(
|
|
27
|
+
text: "Guten Tag, wie geht's?",
|
|
28
|
+
language: :auto, # or "de", "en-US", nil
|
|
29
|
+
want: %i[spelling frequency embeddings grammar],
|
|
30
|
+
force_download: false
|
|
31
|
+
)
|
|
32
|
+
# => #<ResourceBundle
|
|
33
|
+
# language: "de",
|
|
34
|
+
# dictionary: #<Dictionary::Hunspell ...>,
|
|
35
|
+
# frequency: #<FrequencyData ...>,
|
|
36
|
+
# model: #<OnnxModel ...>, # nil if unavailable
|
|
37
|
+
# rules: #<Grammar::RuleSet ...>, # nil if not present
|
|
38
|
+
# cached: true,
|
|
39
|
+
# source_urls: [...]>
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
The `Spellchecker` is rewritten to take a `ResourceBundle` instead of a
|
|
43
|
+
single `Dictionary::Base`.
|
|
44
|
+
|
|
45
|
+
## Tasks
|
|
46
|
+
|
|
47
|
+
1. **Define `ResourceBundle`** as a value object (Struct or plain class —
|
|
48
|
+
no hand-rolled `to_h`; project rule). Fields: `language`, `dictionary`,
|
|
49
|
+
`frequency`, `model`, `rules`, `cached`, `source_urls`.
|
|
50
|
+
2. **Implement `ResourceManager.resolve`.** Pseudocode:
|
|
51
|
+
```
|
|
52
|
+
detect language if :auto (Language::Identifier)
|
|
53
|
+
parallel-fetch each wanted resource (cache → download → verify)
|
|
54
|
+
construct ResourceBundle
|
|
55
|
+
```
|
|
56
|
+
Use existing caches — don't reimplement.
|
|
57
|
+
3. **Manifest verification** (depends on `09-integrity-security` and on
|
|
58
|
+
each content repo shipping a `manifest.json` with SHA-256s). Every
|
|
59
|
+
download is rejected if the checksum doesn't match.
|
|
60
|
+
4. **Offline mode.** `Kotoshu.check(text, offline: true)` only uses cached
|
|
61
|
+
resources; raises `ResourceNotCachedError` if missing instead of
|
|
62
|
+
hitting the network. Required for CI/CD and air-gapped use.
|
|
63
|
+
5. **Strict mode.** `Kotoshu.check(text, strict: true)` raises if any
|
|
64
|
+
wanted resource can't be obtained (vs. default graceful degradation:
|
|
65
|
+
missing model → skip semantic rerank; missing frequency → no bonus).
|
|
66
|
+
6. **Cache pre-warm CLI**. `kotoshu fetch de en es fr pt ru` resolves and
|
|
67
|
+
downloads all resource types for those languages. Useful before
|
|
68
|
+
air travel / CI.
|
|
69
|
+
7. **Configuration knobs**. Add to `Configuration::SCHEMA`:
|
|
70
|
+
- `resource_sources` — hash mapping resource type → URL base
|
|
71
|
+
(defaults to current `dictionaries_url`)
|
|
72
|
+
- `offline` (boolean, default false)
|
|
73
|
+
- `strict_resources` (boolean, default false)
|
|
74
|
+
- `parallel_downloads` (integer, default 4)
|
|
75
|
+
8. **Progress reporting.** A `Progress` callback hooks into the cache
|
|
76
|
+
downloads so the CLI can show `[en] downloading dictionary (3.2MB)…`
|
|
77
|
+
9. **Spec coverage.** A new `spec/integration/dynamic_resolution_spec.rb`
|
|
78
|
+
with a fake HTTP server (Sinatra::Base, no VCR — project rule from
|
|
79
|
+
commit `90aa886`) that serves fixtures and verifies the full resolve →
|
|
80
|
+
cache → re-use cycle. Tag `:network` so it's skippable.
|
|
81
|
+
|
|
82
|
+
## Acceptance criteria
|
|
83
|
+
|
|
84
|
+
- `Kotoshu.check("Bonjour le monde")` works on a fresh machine after
|
|
85
|
+
installing only the gem (auto-downloads fr dictionary + Kelly + ONNX)
|
|
86
|
+
- `Kotoshu.check("こんにちは", offline: true)` raises a useful error
|
|
87
|
+
explaining what to pre-fetch
|
|
88
|
+
- All downloads verify checksums; mismatch raises
|
|
89
|
+
`IntegrityError` with the expected vs. actual hash
|
|
90
|
+
- CLI `kotoshu fetch de` warms every resource type for German
|
|
91
|
+
- No network call when the cache is hot
|
|
92
|
+
|
|
93
|
+
## Dependencies
|
|
94
|
+
|
|
95
|
+
- Blocks: `04-language-modules`, `05-semantic`, `06-cjk`, `07-rtl`
|
|
96
|
+
- Blocked by: `09-integrity-security`, and each content repo's
|
|
97
|
+
`01-manifest-checksums.md`
|
|
98
|
+
|
|
99
|
+
## Out of scope
|
|
100
|
+
|
|
101
|
+
- P2P distribution, torrent mirrors
|
|
102
|
+
- Streaming/incremental dictionary load (load whole file then index)
|
|
103
|
+
- WebAssembly / in-browser use
|
|
104
|
+
|
|
105
|
+
## Status
|
|
106
|
+
|
|
107
|
+
_Pending._
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
# 04 — Language Modules
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Every language in `kotoshu/dictionaries` (98 dirs) is reachable through
|
|
6
|
+
the same `Language::*` module system, with the right tokenizer,
|
|
7
|
+
normalizer, keyboard layout, and resource bindings.
|
|
8
|
+
|
|
9
|
+
## Why
|
|
10
|
+
|
|
11
|
+
The current code ships 7 language modules under `lib/kotoshu/languages/`
|
|
12
|
+
(de, en, es, fr, ja, pt, ru) and 8 tokenizers under
|
|
13
|
+
`lib/kotoshu/language/tokenizer/`. The `dictionaries` repo exposes 98
|
|
14
|
+
language codes. The gap is the user-visible "all languages" promise.
|
|
15
|
+
|
|
16
|
+
## Tasks
|
|
17
|
+
|
|
18
|
+
1. **Language module contract.** Codify what
|
|
19
|
+
`Kotoshu::Language::Base` (in `language/languages/base.rb`) requires:
|
|
20
|
+
- `code` (ISO 639-1, e.g. `"de"`)
|
|
21
|
+
- `locale` (default, e.g. `"de-DE"`)
|
|
22
|
+
- `script` (`:latin | :cyrillic | :cjk | :rtl`)
|
|
23
|
+
- `tokenizer_class`
|
|
24
|
+
- `normalizer_class`
|
|
25
|
+
- `keyboard_layout` (symbol)
|
|
26
|
+
- `dictionary_source` (URL/path within `dictionaries` repo)
|
|
27
|
+
- `frequency_source` (URL/path within `frequency-list-kelly` repo, or
|
|
28
|
+
nil)
|
|
29
|
+
- `model_source` (URL/path within `models-fasttext-onnx` repo, or
|
|
30
|
+
nil)
|
|
31
|
+
- `grammar_rules_path` (within `dictionaries/{code}/grammar/`, or
|
|
32
|
+
nil)
|
|
33
|
+
2. **Audit the dictionaries repo** (cross-references
|
|
34
|
+
`dictionaries/TODO.impl/02-coverage-matrix.md`) and register a module
|
|
35
|
+
for every code that has at least a `spelling/` dir. Generate the
|
|
36
|
+
module files from a template rather than hand-writing 90+ files.
|
|
37
|
+
3. **Tokenizer coverage.** Latin languages share `LatinTokenizer`; map
|
|
38
|
+
each Latin language to it with locale-specific overrides
|
|
39
|
+
(contractions for fr/it/es, ß handling for de, etc.). Cyrillic uses
|
|
40
|
+
the same family. CJK and RTL get dedicated tokenizers (covered in
|
|
41
|
+
plans `06` and `07`).
|
|
42
|
+
4. **Normalizer coverage.** Per-language normalization rules (case
|
|
43
|
+
folding, accent stripping where appropriate, Unicode NFC).
|
|
44
|
+
5. **Keyboard layouts.** Currently `keyboard/layouts/` has qwerty,
|
|
45
|
+
qwertz, azerty, jcuken, dvorak. Add layouts needed by the rest of
|
|
46
|
+
the matrix (e.g. Turkish F, Arabic, Hebrew).
|
|
47
|
+
6. **Language auto-detection contract.** `Language::Identifier` returns
|
|
48
|
+
ISO 639-1 codes; ensure every code it can return has a registered
|
|
49
|
+
module. If detection returns an unsupported code, fall back to
|
|
50
|
+
English with a warning.
|
|
51
|
+
7. **Language-specific resource URL templates.** Centralize the URL
|
|
52
|
+
patterns in `LanguageCache`/`FrequencyCache`/`ModelCache` so a new
|
|
53
|
+
language module doesn't need to repeat them.
|
|
54
|
+
8. **`Kotoshu.supported_languages` accuracy.** Currently returns
|
|
55
|
+
`Language.supported_codes`; ensure this is the source of truth for
|
|
56
|
+
the README's language matrix and the CLI's `--language` enum.
|
|
57
|
+
|
|
58
|
+
## Acceptance criteria
|
|
59
|
+
|
|
60
|
+
- Running `Kotoshu.supported_languages.size` returns ≥ 90 (matching
|
|
61
|
+
dictionaries repo coverage)
|
|
62
|
+
- For every language code in
|
|
63
|
+
`dictionaries/*/spelling/index.dic`, a corresponding
|
|
64
|
+
`lib/kotoshu/languages/{code}/language.rb` exists and is registered
|
|
65
|
+
- A new integration spec `spec/integration/all_languages_spec.rb`
|
|
66
|
+
iterates every supported language and asserts that
|
|
67
|
+
`ResourceManager.resolve(text: sample_text, language: code)` returns
|
|
68
|
+
a valid `ResourceBundle` with a non-nil `dictionary`. Tag `:network`.
|
|
69
|
+
- README's "Language Support Matrix" is generated from code, not
|
|
70
|
+
hand-maintained
|
|
71
|
+
|
|
72
|
+
## Dependencies
|
|
73
|
+
|
|
74
|
+
- Blocks: nothing directly, but enables v1 release
|
|
75
|
+
- Blocked by: `01-hunspell-correctness` (need correct morphological
|
|
76
|
+
path), `03-dynamic-download` (need resource resolution)
|
|
77
|
+
- Cross-repo: depends on `dictionaries/TODO.impl/02-coverage-matrix.md`
|
|
78
|
+
being current
|
|
79
|
+
|
|
80
|
+
## Out of scope
|
|
81
|
+
|
|
82
|
+
- CJK and RTL tokenizers (separate plans `06`, `07`)
|
|
83
|
+
- New grammar rules per language (plan `08`)
|
|
84
|
+
- Custom dictionaries users might supply (already supported via
|
|
85
|
+
`Dictionary::Custom`)
|
|
86
|
+
|
|
87
|
+
## Status
|
|
88
|
+
|
|
89
|
+
_Pending._
|