kotoshu 1.0.2-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Plan 104 — VS Code marketplace + Open VSX readiness
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
The extension is built, CI-green, with a .vsix artifact — but it is not
|
|
5
|
+
installable without the marketplace listing, which is the single largest
|
|
6
|
+
adoption blocker for the editor audience. Publishing needs the owner's
|
|
7
|
+
publisher PAT (a credential we never hold); everything else can be ready
|
|
8
|
+
so publishing is one secret + one click.
|
|
9
|
+
|
|
10
|
+
## Work (vscode repo)
|
|
11
|
+
1. Open VSX publish workflow (release-ovsx.yml, secret-gated) beside the
|
|
12
|
+
existing CI — Open VSX serves VSCodium/Code-OSS users without the
|
|
13
|
+
Microsoft marketplace.
|
|
14
|
+
2. Metadata polish: gallery banner/colors, keywords, categories, repository
|
|
15
|
+
links, the kotoshu-vscode README (quickstart with kotoshu-lsp install,
|
|
16
|
+
settings reference: log level, personal dictionary command).
|
|
17
|
+
3. PREPUBLISH.md: exact owner steps — create publisher, add PAT as secret,
|
|
18
|
+
run the workflow; nothing else.
|
|
19
|
+
4. Site follow-up line (rides plan 99's pass): /docs/clients/lsp gains the
|
|
20
|
+
marketplace link placeholder to fill on publish day.
|
|
21
|
+
|
|
22
|
+
## Verification
|
|
23
|
+
`vsce package` clean with no warnings; workflow lint-valid; PREPUBLISH
|
|
24
|
+
steps verified against the actual GitHub UI labels.
|
|
25
|
+
|
|
26
|
+
## Status
|
|
27
|
+
Executed 2026-09-07 — vscode PR #3, merged. release-marketplace.yml (VSCE_PAT)
|
|
28
|
+
+ release-ovsx.yml (OPEN_VSX_TOKEN), both version-gated; PREPUBLISH.md with
|
|
29
|
+
verified owner steps; package.json publisher/categories/keywords/banner;
|
|
30
|
+
README quickstart against gem 0.1.1; vsce package warning-clean. Publishing
|
|
31
|
+
itself is one PAT secret + one tag — owner action. The /docs/clients/lsp
|
|
32
|
+
marketplace link rides the coordinated site pass.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Plan 105 — The personal dictionary reaches the check path
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
`kotoshu personal add recieve` then `kotoshu check` still flags the word —
|
|
5
|
+
PersonalDictionary is wired only into the `personal` subcommand and the LSP
|
|
6
|
+
(resolved server-side), not into `Spellchecker#check`. Users who add words
|
|
7
|
+
see them flagged again on the very next run; the README story ("add words
|
|
8
|
+
you accept") is currently false on the CLI/API path.
|
|
9
|
+
|
|
10
|
+
## Work (gem repo)
|
|
11
|
+
1. `Spellchecker#check` consults the personal dictionary: a word present in
|
|
12
|
+
personal.dic is not an error (mirror the LSP's semantics: filter at
|
|
13
|
+
result assembly, keep suppression metadata out of the error rows).
|
|
14
|
+
2. Reload semantics: the LSP reloads on mtime change; the CLI/one-shot path
|
|
15
|
+
loads once per process. Do not add hot reload here.
|
|
16
|
+
3. Opt-out: `Kotoshu.configure { personal_dictionary false }` (or the
|
|
17
|
+
existing config seam — find it) and `--no-personal` on `kotoshu check`.
|
|
18
|
+
4. Specs: add-then-check round trip on a real dictionary; opt-out flags
|
|
19
|
+
again; personal words still spellcheckable via suggest.
|
|
20
|
+
5. Docs truth: /docs/ignores gains the personal dictionary section (the
|
|
21
|
+
action PR #2 README noted this gap) — deliver the copy in the PR body
|
|
22
|
+
for the site follow-up, do not touch the site repo.
|
|
23
|
+
|
|
24
|
+
## Verification
|
|
25
|
+
Suite green (3,900+ examples); rubocop clean; `kotoshu personal add x &&
|
|
26
|
+
kotoshu check` live round trip recorded in the PR body.
|
|
27
|
+
|
|
28
|
+
## Status
|
|
29
|
+
Executed 2026-09-07 — gem PR (branch plan-105-personal-dict-in-check):
|
|
30
|
+
check-path filter mirroring the LSP, one load per process,
|
|
31
|
+
Configuration#personal_dictionary + KOTOSHU_PERSONAL_DICTIONARY +
|
|
32
|
+
`kotoshu check --no-personal`; suite pins KOTOSHU_PERSONAL_DIC for
|
|
33
|
+
hermeticity. /docs/ignores copy delivered in the PR body for the site
|
|
34
|
+
follow-up.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Plan 106 — Document language detection through the pure-Rust LID reader
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
The gem's LanguageIdentifier needs the Python fastText bindings (and breaks
|
|
5
|
+
under numpy 2), so /v1/detect ships a 7-language heuristic. kotoshu-rs now
|
|
6
|
+
carries a pure-Rust lid.176 reader (kotoshu/src/lid, parity 55/55 vs the
|
|
7
|
+
gem) behind the same `model` feature the native ext already builds.
|
|
8
|
+
|
|
9
|
+
## Work (gem repo)
|
|
10
|
+
1. Expose `Kotoshu.detect_language(text)` through the native backend:
|
|
11
|
+
download/resolve lid.176.onnx + vocab via the models registry
|
|
12
|
+
(kotoshu://models/lid/lid-176, v1.4.0), score with the ext, return
|
|
13
|
+
{ code, score }. Heuristic Language::Detector stays the fallback when
|
|
14
|
+
the native ext is absent or the model is not set up.
|
|
15
|
+
2. FFI seam: extend the magnus shim (ext/kotoshu_native) with the call —
|
|
16
|
+
mirror how suggest/check already cross; keep it behind the existing
|
|
17
|
+
feature so pure-Ruby installs are unaffected.
|
|
18
|
+
3. Server follow-up (kotoshu-server repo, separate PR riding this):
|
|
19
|
+
/v1/detect prefers Kotoshu.detect_language when available. Note it in
|
|
20
|
+
the PR body; do not touch the server repo yourself.
|
|
21
|
+
4. Specs: native-vs-fixture parity (reuse the rs corpus JSON shape),
|
|
22
|
+
fallback behavior, registry miss error.
|
|
23
|
+
|
|
24
|
+
## Verification
|
|
25
|
+
Suite green; a live detect over en/de/ru/ja/ar samples recorded in the PR
|
|
26
|
+
body; pure-Ruby suite still green (no new hard dependency).
|
|
27
|
+
|
|
28
|
+
## Status
|
|
29
|
+
Executed 2026-09-07 — gem PR (branch plan-106-native-lid-detection):
|
|
30
|
+
Kotoshu.detect_language -> Language::Detection through the ext
|
|
31
|
+
(LidModel.load/#detect in the magnus shim, kotoshu-rs model feature,
|
|
32
|
+
Cargo.lock bumped to the reader revision); ModelCache gained the
|
|
33
|
+
lid pair (download_lid/load_cached_lid, top-level lid/ dir);
|
|
34
|
+
Kotoshu.setup_lid two-stage; heuristic fallback incl.
|
|
35
|
+
KOTOSHU_BACKEND=ruby; parity 55/55 vs the shared frozen corpus.
|
|
36
|
+
Server follow-up (/v1/detect) noted in the PR body.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Plan 107 — Basic support for every staged dictionary language
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
`kotoshu setup` only serves the 32 languages with modules. Sixty-plus staged
|
|
5
|
+
dictionaries (from gd to ia) are unreachable: a Welsh or Icelandic user
|
|
6
|
+
cannot use the gem at all, even though the dictionary and model are
|
|
7
|
+
downloadable. Modules should be the UPGRADE (keyboard + verified specimens),
|
|
8
|
+
not the gate.
|
|
9
|
+
|
|
10
|
+
## Work (gem repo)
|
|
11
|
+
1. `LanguageCache::AVAILABLE_LANGUAGES` derives from the dictionaries
|
|
12
|
+
manifest (the source the registry/staging uses) instead of the hardcoded
|
|
13
|
+
module list; modules register themselves as full-feature upgrades.
|
|
14
|
+
2. Default behavior for module-less languages: script-appropriate tokenizer
|
|
15
|
+
(reuse the script-regex machinery from plan 91), generic keyboard
|
|
16
|
+
fallback (QWERTY for Latin, JCUKEN Cyrillic, Arabic 101 Arabic-script),
|
|
17
|
+
setup/check/suggest all functional.
|
|
18
|
+
3. Wire nn (model + dictionary exist) as a full-feature module while here;
|
|
19
|
+
note sr-Latn as follow-up if the Latin mapping needs product sign-off.
|
|
20
|
+
4. Specs: setup + specimen round trip on 3 module-less staged languages
|
|
21
|
+
(e.g. is, cy, gd); full-feature languages behave identically to today;
|
|
22
|
+
conformance untouched.
|
|
23
|
+
|
|
24
|
+
## Verification
|
|
25
|
+
Live `kotoshu setup is && check -l is` round trip; suite green; the 32
|
|
26
|
+
modules' outputs byte-identical.
|
|
27
|
+
|
|
28
|
+
## Status
|
|
29
|
+
Executed 2026-09-07 — AVAILABLE_LANGUAGES derives from a vendored
|
|
30
|
+
snapshot of the dictionaries manifest (Cache::StagedLanguages, 95
|
|
31
|
+
languages, refreshed by `rake kotoshu:staged_languages:sync`);
|
|
32
|
+
module-less languages ride Language::Script (script tokenizer for
|
|
33
|
+
word extraction: shared Latin/Cyrillic/Greek classes plus a generic
|
|
34
|
+
regex tokenizer for Arabic/Hebrew/Armenian/Georgian/Devanagari/
|
|
35
|
+
Hangul) and the script keyboard default (JCUKEN Cyrillic, Arabic 101,
|
|
36
|
+
Hebrew SI-1452, QWERTY Latin). Tiers documented: full feature =
|
|
37
|
+
module (LanguageCache.full_feature_languages, 33 with nn); basic =
|
|
38
|
+
manifest-present. nn wired as a full-feature module. Specimens
|
|
39
|
+
engine-verified for is cy gd (basic) + nn; live round trips
|
|
40
|
+
setup+check -l for is cy gd nn; conformance 2630 vectors 0 failures;
|
|
41
|
+
suite 3948 examples 0 failures. sr-Latn stays follow-up (normalize
|
|
42
|
+
maps it to sr; needs product sign-off).
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# Plan 108 — Korean and Nepali: the last big-population languages
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
ko (~82M) and ne (~16M) were deferred from batch 3 because Hangul jamo
|
|
5
|
+
input and Devanagari conjuncts need ja-style tokenizer treatment. They are
|
|
6
|
+
the largest populations without full-feature support; both have staged
|
|
7
|
+
dictionaries and models.
|
|
8
|
+
|
|
9
|
+
## Work (gem repo; sequenced AFTER plan 107 merges — same area)
|
|
10
|
+
1. ko: Hangul tokenizer (syllable-block aware, jamo-level word regex so
|
|
11
|
+
check extracts real eojeol), Korean keyboard grid (Dubeolsik 2-set),
|
|
12
|
+
module + specimens engine-verified.
|
|
13
|
+
2. ne: Devanagari tokenizer (matras and conjuncts stay attached to the
|
|
14
|
+
base consonant — the Unicode grapheme rules the script needs), Roman
|
|
15
|
+
keyboard fallback or a Nepali traditional grid if defensibly sourced,
|
|
16
|
+
module + specimens.
|
|
17
|
+
3. AVAILABLE_LANGUAGES + site-flip deliverables (specimen table, sizes).
|
|
18
|
+
|
|
19
|
+
## Verification
|
|
20
|
+
Same discipline as batch 3: every specimen engine-verified through
|
|
21
|
+
correct?/suggest; RTL/complex-script sentence round trips through
|
|
22
|
+
Kotoshu.check; suite green.
|
|
23
|
+
|
|
24
|
+
## Status
|
|
25
|
+
Executed 2026-09-07 — ko: HangulTokenizer (eojeol = run of syllables
|
|
26
|
+
U+AC00-D7A3 plus bare jamo U+1100-11FF/U+3130-318F), Dubeolsik grid
|
|
27
|
+
keyed on jamo per KS X 5002 (the eval harness models ko slips as
|
|
28
|
+
jamo confusion pairs, not a key grid, so no drift check applies),
|
|
29
|
+
module + 7 engine-verified specimens. ne: DevanagariTokenizer
|
|
30
|
+
(matras and virama conjuncts attached to the base consonant; danda
|
|
31
|
+
and Devanagari digits are separators), Devanagari InScript grid
|
|
32
|
+
mirrored from the eval harness _NE_INSCRIPT and drift-checked (the
|
|
33
|
+
defensible standard — Traditional Romanized is common but
|
|
34
|
+
undocumented), module + 3 specimens. Full-feature languages 33 -> 35.
|
|
35
|
+
Live round trips setup + check -l ko ne recorded; conformance 2630
|
|
36
|
+
vectors 0 failures; suite 3985 examples 0 failures. Specimen table
|
|
37
|
+
and dictionary sizes in the PR body for the site flip.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Plan 109 — The 1.0 readiness audit
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Adopters hesitate at 0.x. A 1.0 cut needs a frozen public surface and a
|
|
5
|
+
stated policy. The audit inventories what 1.0 would freeze and what it
|
|
6
|
+
would deprecate — it does NOT cut versions (owner decision).
|
|
7
|
+
|
|
8
|
+
## Work (gem repo docs/, analysis only)
|
|
9
|
+
1. Inventory the public surface: Ruby API (Kotoshu module functions,
|
|
10
|
+
Spellchecker, Configuration, Tasks, validators), CLI flags, wasm JS
|
|
11
|
+
surface (KotoshuWasm, loadModel/rerank/semanticSuggest/loadLid/
|
|
12
|
+
detectLanguage), HTTP contract (openapi.yaml), action inputs, registry
|
|
13
|
+
schema.
|
|
14
|
+
2. For each: stable / needs-deprecation-notice / internal-accidentally-
|
|
15
|
+
public. Flag anything accidentally public that 1.0 should privatize.
|
|
16
|
+
3. Draft the stability policy (semver commitments, deprecation window,
|
|
17
|
+
conformance vectors as the behavioral contract) as docs/STABILITY.md.
|
|
18
|
+
4. A 1.0 checklist: what must land first (nothing mechanical is known to
|
|
19
|
+
block), what the release train looks like (gem+wasm+server together).
|
|
20
|
+
|
|
21
|
+
## Verification
|
|
22
|
+
Every inventory line cites the file that defines it; the deprecation list
|
|
23
|
+
reviewed against gemspec/exported constants.
|
|
24
|
+
|
|
25
|
+
## Status
|
|
26
|
+
Pending
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
# 11 — Release v1.0
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Cut a v1.0.0 release of the `kotoshu` gem with everything in plans 01-10
|
|
6
|
+
landed (or explicitly descoped), and a published v1.0.0 tag of each
|
|
7
|
+
content repo.
|
|
8
|
+
|
|
9
|
+
## Why
|
|
10
|
+
|
|
11
|
+
No gem release exists. `git tag` shows nothing on `main`. The gemspec
|
|
12
|
+
already references `CHANGELOG.md` which doesn't exist. v1 is the
|
|
13
|
+
commitment that the public API is stable and the supported-language
|
|
14
|
+
matrix is real.
|
|
15
|
+
|
|
16
|
+
## Tasks
|
|
17
|
+
|
|
18
|
+
1. **API freeze audit.** Walk every public method on the `Kotoshu`
|
|
19
|
+
module (`lib/kotoshu.rb`) and the `Configuration` class. Ensure
|
|
20
|
+
every signature is documented, every option is in `SCHEMA`, and the
|
|
21
|
+
YARD docs are complete. After v1, breaking changes require v2.
|
|
22
|
+
2. **CHANGELOG.md.** Write the v1.0.0 entry following Keep a Changelog
|
|
23
|
+
format. The gemspec metadata points at this file.
|
|
24
|
+
3. **README accuracy pass.** Fix:
|
|
25
|
+
- License (says MIT, actually BSD-2-Clause)
|
|
26
|
+
- ONNX model count (says 6, will be ~30+ at v1)
|
|
27
|
+
- Language matrix (must match `Kotoshu.supported_languages`)
|
|
28
|
+
- CLI examples (every example must work verbatim)
|
|
29
|
+
4. **Deprecation policy.** Add `SECURITY.md` (from plan 09) and
|
|
30
|
+
`POLICY.md` documenting semver, supported Ruby versions, EOL
|
|
31
|
+
cadence.
|
|
32
|
+
5. **Sign the gem** with a project rubygems.org account. Document
|
|
33
|
+
who has release privileges.
|
|
34
|
+
6. **Content repo tags.** Tag each content repo in lockstep:
|
|
35
|
+
- `dictionaries` → `v1.0.0`
|
|
36
|
+
- `frequency-list-kelly` → `v1.0.0`
|
|
37
|
+
- `models-fasttext-onnx` → `v1.0.0`
|
|
38
|
+
Pin the gem's default `resource_sources` URLs to these tags (via
|
|
39
|
+
the manifest system from plan 09).
|
|
40
|
+
7. **Release automation.** GitHub Action that, on tag push:
|
|
41
|
+
- Builds the gem
|
|
42
|
+
- Publishes to rubygems.org
|
|
43
|
+
- Creates a GitHub Release with auto-generated notes
|
|
44
|
+
8. **Post-release smoke test.** A script that, on a fresh Docker
|
|
45
|
+
container with only Ruby installed:
|
|
46
|
+
- `gem install kotoshu`
|
|
47
|
+
- `ruby -e 'require "kotoshu"; puts Kotoshu.correct?("hello")'`
|
|
48
|
+
- Exercises each documented CLI example from the README
|
|
49
|
+
9. **Announcement.** Blog post on kotoshu.github.io (once it exists,
|
|
50
|
+
plan depends on `kotoshu.github.io/TODO.impl/`), Ruby Weekly
|
|
51
|
+
submission, Reddit /r/ruby.
|
|
52
|
+
|
|
53
|
+
## v1 scope (locked)
|
|
54
|
+
|
|
55
|
+
In scope:
|
|
56
|
+
- Traditional Hunspell path (Latin scripts) — plans 01, 04
|
|
57
|
+
- Semantic ONNX reranking — plan 05
|
|
58
|
+
- Dynamic download — plan 03
|
|
59
|
+
- Integrity verification — plan 09
|
|
60
|
+
- Grammar rules (starter pack for the 6 fully-supported languages) —
|
|
61
|
+
plan 08
|
|
62
|
+
- CLI with all documented flags — plan 02
|
|
63
|
+
- ≥ 30 supported languages with end-to-end coverage
|
|
64
|
+
|
|
65
|
+
Out of scope (post-v1):
|
|
66
|
+
- CJK full support — plan 06 (may ship experimental in v1.1)
|
|
67
|
+
- RTL full support — plan 07 (same)
|
|
68
|
+
- LSP / HTTP server
|
|
69
|
+
- Editor plugins
|
|
70
|
+
|
|
71
|
+
## Acceptance criteria
|
|
72
|
+
|
|
73
|
+
- `gem install kotoshu` works on a clean Ruby 3.1+ machine
|
|
74
|
+
- `Kotoshu::VERSION == "1.0.0"`
|
|
75
|
+
- `git tag v1.0.0` exists on `kotoshu/kotoshu`, `kotoshu/dictionaries`,
|
|
76
|
+
`kotoshu/frequency-list-kelly`, `kotoshu/models-fasttext-onnx`
|
|
77
|
+
- The post-release smoke test passes
|
|
78
|
+
- rubygems.org page shows the gem with correct metadata
|
|
79
|
+
- README's first 100 lines are accurate (no broken examples)
|
|
80
|
+
|
|
81
|
+
## Dependencies
|
|
82
|
+
|
|
83
|
+
- Blocked by: all of plans 01-10
|
|
84
|
+
- Cross-repo: each content repo's `TODO.impl/03-*-releases.md` (or
|
|
85
|
+
equivalent)
|
|
86
|
+
|
|
87
|
+
## Out of scope
|
|
88
|
+
|
|
89
|
+
- v1.1, v2 planning (post-release)
|
|
90
|
+
- Marketing beyond the announcement
|
|
91
|
+
- Conference talks (delightful but not blocking)
|
|
92
|
+
|
|
93
|
+
## Status
|
|
94
|
+
|
|
95
|
+
_Pending._
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Plan 110 — sr-Latn wiring and the long-tail polish list
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Plan 107 wired nn and deferred sr-Latn (the Latin script variant of Serbian)
|
|
5
|
+
as follow-up. With the basic tier live, this is the last named language gap
|
|
6
|
+
in the corpus; everything after it is small-polish.
|
|
7
|
+
|
|
8
|
+
## Work (gem repo)
|
|
9
|
+
1. sr-Latn: register as a full-feature module reusing the Serbian dictionary
|
|
10
|
+
with the Latin tokenizer + the Croatian/Slovenian QWERTZ grid (the
|
|
11
|
+
scripts share the Latin layout), or map sr-Latn -> the staged sr-Latn
|
|
12
|
+
dictionary if one exists in the manifest — follow the data, document the
|
|
13
|
+
choice. Specimens engine-verified.
|
|
14
|
+
2. Polish sweep: any spellcheck_word_regex script set lacking a staged
|
|
15
|
+
language (inventory + one-line table in the PR body); README language
|
|
16
|
+
table refresh to the 35+ tiers.
|
|
17
|
+
3. Frequency-list inventory (no sourcing): list which staged languages have
|
|
18
|
+
published frequency data wired vs not — record only, sourcing is a
|
|
19
|
+
separate owner decision.
|
|
20
|
+
|
|
21
|
+
## Verification
|
|
22
|
+
Live setup + check -l sr-Latn round trip; suite green; conformance
|
|
23
|
+
untouched.
|
|
24
|
+
|
|
25
|
+
## Status
|
|
26
|
+
Executed 2026-09-07 — the data chose the distinct staged dictionary:
|
|
27
|
+
sr-Latn is a full-feature module (Languages::SerbianLatin, Latin
|
|
28
|
+
tokenizer, Serbian-Latin QWERTZ grid sharing the hrsl key table), and
|
|
29
|
+
the sr-Latn code now survives ResourceManager normalization
|
|
30
|
+
(script subtags kept, region subtags still fold) so the staged Latin
|
|
31
|
+
files are reachable — previously sr-Latn collapsed onto the Cyrillic
|
|
32
|
+
sr dictionary. Script-set and frequency inventories recorded in the
|
|
33
|
+
PR body (record-only). Suite green, conformance untouched.
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Plan 111 — The embedding bake-off: is fastText still the right model?
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Our gates prove int8 near-lossless RELATIVE to fastText — but fastText is
|
|
5
|
+
2017-era, context-blind, and our own numbers cap intended-top-5 near 0.21
|
|
6
|
+
on hard English typos even with buckets. The eval harness (keyboard-aware
|
|
7
|
+
grids, typo corpora, frozen gates) is the asset; use it to test successors.
|
|
8
|
+
|
|
9
|
+
## Work (models repo; eval only — no registry release without a verdict)
|
|
10
|
+
1. Baseline: current mini/fluency+buckets through the existing gates.
|
|
11
|
+
2. Candidate B: fastText + small cross-encoder rerank over (context,
|
|
12
|
+
candidate) pairs — ONNX-exportable distill class, 15-30 MB int8.
|
|
13
|
+
3. Candidate C: purpose-trained typo bi-encoder over the vendored
|
|
14
|
+
github-typo-corpus (train/eval split strictly by repo).
|
|
15
|
+
4. Candidate D: ModernBERT-class small encoder, same size budget.
|
|
16
|
+
5. Report: gates table per candidate per language (en de ru es minimum),
|
|
17
|
+
size, latency-per-suggest, and a ship/reject verdict with the same
|
|
18
|
+
discipline as int4 (reject on data is a complete outcome).
|
|
19
|
+
|
|
20
|
+
## Verification
|
|
21
|
+
Every number from the existing harness, unweakened; frozen corpora; no
|
|
22
|
+
registry changes in this plan.
|
|
23
|
+
|
|
24
|
+
## Status
|
|
25
|
+
Executed 2026-09-09 — models PR #25 merged. VERDICT: fastText stays.
|
|
26
|
+
B (MiniLM cross-encoder rerank) REJECT — loses 4/4 languages (en 0.194 vs
|
|
27
|
+
fluency 0.252 top-5; slate recall caps en at 0.43; context ablation
|
|
28
|
+
negative). D (ModernBERT-base) REJECT — representation failure (top-5
|
|
29
|
+
0.03-0.24), 5x size cap. C (0.5 MB char-BiGRU typo bi-encoder, 5.2
|
|
30
|
+
ms/suggest) REJECT as drop-in but PROMISING — beats the full tier on the
|
|
31
|
+
strict repo-clean subset for en and de; open thread needing non-en data,
|
|
32
|
+
a bigger clean bench, and a top-1 story. Full ladders in
|
|
33
|
+
eval/reports/bakeoff-v1.md.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Plan 112 — Deployability and CI gating
|
|
2
|
+
|
|
3
|
+
## Work (kotoshu-rs; site rides a follow-up)
|
|
4
|
+
1. Prebuilt worker on npm: @kotoshu/worker exposing the playground's
|
|
5
|
+
engine worker (load/check/suggest-batch/semantic/detect protocol) so
|
|
6
|
+
JS users embed the engine without reimplementing the worker; versioned
|
|
7
|
+
with the wasm package.
|
|
8
|
+
2. CI latency gate: the multi-language sweep bench (rs PR #23 harness)
|
|
9
|
+
runs on CI with budgets (en < 120 ms, pt < 700 ms avg) failing red.
|
|
10
|
+
3. Wasm memory ceiling test: dict + mini tier + buckets resident under a
|
|
11
|
+
stated budget per language class.
|
|
12
|
+
4. Language pack (design note, owner decision): one fetch for
|
|
13
|
+
dict+tier+buckets — additive registry field sketch in the PR body.
|
|
14
|
+
|
|
15
|
+
## Verification
|
|
16
|
+
Node smoke for the worker package; budgets tuned to measured p50s, not
|
|
17
|
+
aspirations; all existing CI green.
|
|
18
|
+
|
|
19
|
+
## Status
|
|
20
|
+
Executed 2026-09-09 — kotoshu-rs PR #28 merged. @kotoshu/worker package
|
|
21
|
+
(createEngine full protocol, ./engine-worker entry, semantic-merge pure
|
|
22
|
+
half; publish pending owner npm registration); latency.yml gate with
|
|
23
|
+
measured budgets (en 56ms avg vs 120 gate, pt 346 vs 700 — 2x headroom);
|
|
24
|
+
wasm32 memory ceilings frozen (en 46.9/64 MB, pt 165/192 MB) with
|
|
25
|
+
behavior probes; language-pack kotoshu://packs/{lang} sketch in the PR
|
|
26
|
+
body awaiting owner decision.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Plan 113 — Language packs: one fetch for a whole language
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Loading a language today is 3+ round trips (dict aff+dic, tier, buckets,
|
|
5
|
+
sometimes registry) - the rs PR #28 sketch showed a single length-prefixed
|
|
6
|
+
pack is simpler for SDK users and cuts playground cold loads.
|
|
7
|
+
|
|
8
|
+
## Work
|
|
9
|
+
1. models repo: pack builder (dict sources + tier + buckets concatenated,
|
|
10
|
+
length-prefixed sections; kotoshu://packs/{lang} additive registry
|
|
11
|
+
entries, versioned with their parts); v1.6.0 when validated.
|
|
12
|
+
2. kotoshu-rs: wasm loadPack(bytes) -> {dictionary, model} handle; worker
|
|
13
|
+
package gains pack mode; keep per-artifact loading as fallback.
|
|
14
|
+
3. Site: playground opt-in pack path with one progress bar.
|
|
15
|
+
|
|
16
|
+
## Verification
|
|
17
|
+
Pack bytes == concatenation of parts (checksums per section); parity
|
|
18
|
+
specs per language; playground CDP cold-load comparison (requests and ms).
|
|
19
|
+
|
|
20
|
+
## Status
|
|
21
|
+
Executed 2026-09-09 — models PR #26 + kotoshu-rs PR #32, both merged.
|
|
22
|
+
KPK1 format (length+tag framed sections, sha256 footers, golden-sha parity
|
|
23
|
+
across implementations); en/de/pt packs LFS-committed with live mirrors;
|
|
24
|
+
additive kotoshu://packs/{lang} entries; wasm loadPack + worker pack mode
|
|
25
|
+
with per-artifact fallback. Registry v1.6.0 cut awaiting owner; site
|
|
26
|
+
playground flip optional (worker package already serves pack mode).
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Plan 114 — The typo bi-encoder thread (bake-off follow-up)
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Bake-off candidate C (0.5 MB char-BiGRU, 5.2 ms/suggest) beat the FULL
|
|
5
|
+
tier on repo-clean en/de pairs - the only successor that did. It needs:
|
|
6
|
+
non-en training data, a bigger clean bench, and a top-1 story.
|
|
7
|
+
|
|
8
|
+
## Work (models repo)
|
|
9
|
+
1. Extend training data beyond en (de ru es typo corpora; synth via the
|
|
10
|
+
existing generator where corpora are thin).
|
|
11
|
+
2. Grow the repo-clean bench (target 2000+ pairs/lang); freeze as the
|
|
12
|
+
C-benchmark.
|
|
13
|
+
3. Train v2; measure against fastText tiers on the same bench; ship as
|
|
14
|
+
an OPT-IN registry resource only if it wins en+de clearly and degrades
|
|
15
|
+
nowhere - reject honestly otherwise.
|
|
16
|
+
|
|
17
|
+
## Status
|
|
18
|
+
Executed 2026-09-09 — models PR #27 merged. VERDICT: REJECT, honest on
|
|
19
|
+
data; fastText tiers stay, no registry change, no tag. Frozen C-benchmark:
|
|
20
|
+
en 2509 real repo-clean pairs (target 2000+ met with real data only),
|
|
21
|
+
de/ru/es real clean pairs are 14/27/20 — the corpus holds only 212/551/186
|
|
22
|
+
unique pairs for those languages — plus a labeled 2000-pair noise.py synth
|
|
23
|
+
component per language, disjoint from all training sources. C v2 (same
|
|
24
|
+
0.481 MB architecture, training data the one changed variable) beats the
|
|
25
|
+
full tier on 2509 real unseen en pairs (+7.5 pp top-5, CI +5.9 to +9.1)
|
|
26
|
+
and fixed the top-1 gap (0.128 vs 0.103), but fails the pre-declared rule:
|
|
27
|
+
de real clears no paired CI at n=14, and es real regresses 10 pp top-5 at
|
|
28
|
+
n=20 (three independent es slices agree). Synth dominance (+40 to +48 pp)
|
|
29
|
+
is generator-domain and does not transfer to real es. Surviving thread for
|
|
30
|
+
a future plan: the C-retrieve + fastText-rescore hybrid beats the full
|
|
31
|
+
tier top-1 and top-5 on all four real components, but needs the
|
|
32
|
+
per-language vocab matrix cost priced and a real es/de typo corpus.
|
|
33
|
+
Full ladder: models eval/reports/biencoder-v2.md.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# Plan 115 — The hybrid retrieval thread: price it before believing it
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
Plan 114's reject left one survivor on record: C-retrieve (0.5 MB bi-encoder)
|
|
5
|
+
+ fastText-rescore beat the FULL tier on top-1 AND top-5 across all four
|
|
6
|
+
real bench components. Before any ship decision, its costs and its
|
|
7
|
+
dependence on synthetic-heavy slices must be priced honestly.
|
|
8
|
+
|
|
9
|
+
## Work (models repo)
|
|
10
|
+
1. Price the retrieval half: per-language 100k x 256 vocab matrix (fp16 +
|
|
11
|
+
int8 variants) — build for en, measure size, build time, top-k
|
|
12
|
+
retrieval latency, and quality vs the brute-force sweep on the frozen
|
|
13
|
+
C-benchmark.
|
|
14
|
+
2. Quality stress: score the hybrid ONLY on real-pair components
|
|
15
|
+
(en 2509; de/ru/es real n are tiny — report with CIs, no synth pooling
|
|
16
|
+
for the verdict).
|
|
17
|
+
3. Verdict per the plan-114 rule: ship as opt-in registry resource only
|
|
18
|
+
on clear wins with no regression; honest reject otherwise. No registry
|
|
19
|
+
changes on reject.
|
|
20
|
+
|
|
21
|
+
## Status
|
|
22
|
+
Pending
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Plan 116 — Baseline-aware suggestion skip in `kotoshu check`
|
|
2
|
+
|
|
3
|
+
Status: executed (gem PR #175, 2026-09-10)
|
|
4
|
+
|
|
5
|
+
Measured on this repo's own dogfood gate (prose baseline, 640 entries /
|
|
6
|
+
1,905 covered occurrences): check wall time 13m+ -> 13.7s, identical
|
|
7
|
+
output. Full default suite 4003/0.
|
|
8
|
+
|
|
9
|
+
Depends on: plan 82 (baselines), plan 113 (packs — unrelated but same file surfaces)
|
|
10
|
+
Blocks: nothing; targets the 0.11.x line before 1.0
|
|
11
|
+
|
|
12
|
+
## Problem
|
|
13
|
+
|
|
14
|
+
`kotoshu check --baseline` in CI regenerates the suggestion sweep for
|
|
15
|
+
every error it is about to suppress. On the kotoshu repo itself the
|
|
16
|
+
prose baseline holds 1,905 occurrences across 7 files; the check phase
|
|
17
|
+
of the dogfood gate spends ~10 CPU-minutes producing suggestions that
|
|
18
|
+
the baseline then throws away. The site repo's spellcheck job runs
|
|
19
|
+
6m48s for the same reason. Suggestions only matter for errors that
|
|
20
|
+
SURVIVE the baseline — those are the rows a human reads.
|
|
21
|
+
|
|
22
|
+
`kotoshu baseline init` already fixed this for itself
|
|
23
|
+
(`Spellchecker#check(text, suggestions: false)`, plan-executed in
|
|
24
|
+
"fix(cli): make baseline init usable on real repositories"). `check`
|
|
25
|
+
still pays full price.
|
|
26
|
+
|
|
27
|
+
## Design
|
|
28
|
+
|
|
29
|
+
The budget is knowable before checking: the baseline entry list for the
|
|
30
|
+
file under check says exactly which words will be absorbed (and how
|
|
31
|
+
many times). So:
|
|
32
|
+
|
|
33
|
+
1. `Cli::DirectoryCheck` (and the single-file check path) computes
|
|
34
|
+
`covered = baseline.entries_for(path)` → a word→count budget hash,
|
|
35
|
+
before invoking the spellcheck callback.
|
|
36
|
+
2. `Spellchecker#check` gains an optional `suggestions_filter:` hook —
|
|
37
|
+
a callable receiving the word, returning whether to generate
|
|
38
|
+
suggestions. Default nil = generate for everything (current
|
|
39
|
+
behavior). The loop already has the `suggestions:` branch; this
|
|
40
|
+
generalizes it without touching result shape.
|
|
41
|
+
- Occurrence counting must be honored: the budget for `wrold: 3`
|
|
42
|
+
means the first 3 occurrences skip, the 4th (a NEW error)
|
|
43
|
+
generates. The filter therefore closes over a mutable budget
|
|
44
|
+
hash in the caller, decremented per consumed occurrence.
|
|
45
|
+
3. Errors suppressed by the baseline keep their (empty) suggestion set
|
|
46
|
+
— unchanged from today, since suppressed rows render without
|
|
47
|
+
suggestions anyway.
|
|
48
|
+
4. New errors — beyond budget or unknown words — generate suggestions
|
|
49
|
+
exactly as today. SARIF, text, and JSON output shapes unchanged.
|
|
50
|
+
5. Inline-suppressed words: keep generating suggestions today (they
|
|
51
|
+
are dropped at assembly regardless) — fold into the same skip by
|
|
52
|
+
treating `kotoshu:disable` regions as covered when the suppression
|
|
53
|
+
scan already knows the word/line. Out of scope if the scan order
|
|
54
|
+
makes this awkward; the baseline case is the measured cost.
|
|
55
|
+
|
|
56
|
+
## Surfaces touched
|
|
57
|
+
|
|
58
|
+
- `lib/kotoshu/spellchecker.rb` — `check(text, suggestions:,
|
|
59
|
+
suggestions_filter:)`
|
|
60
|
+
- `lib/kotoshu/cli/directory_check.rb` — budget construction from the
|
|
61
|
+
loaded baseline, filter closure
|
|
62
|
+
- `lib/kotoshu/cli.rb` — single-file `--baseline` path, same wiring
|
|
63
|
+
- `spec/kotoshu/cli/baseline_cli_spec.rb` — new cases: covered word
|
|
64
|
+
beyond budget generates suggestions; covered occurrence does not;
|
|
65
|
+
no baseline = unchanged
|
|
66
|
+
|
|
67
|
+
## Non-goals
|
|
68
|
+
|
|
69
|
+
- No output format changes; suppressed rows stay suggestion-less.
|
|
70
|
+
- No change to `Baseline::Store` semantics (apply stays the authority
|
|
71
|
+
on pass/fail); the filter is purely a compute optimization whose
|
|
72
|
+
visible effect is empty suggestions on rows that were going to be
|
|
73
|
+
suppressed.
|
|
74
|
+
|
|
75
|
+
## Acceptance
|
|
76
|
+
|
|
77
|
+
- Dogfood gate on the kotoshu repo: check phase wall time drops from
|
|
78
|
+
~10 min to under ~2 min (the sweep itself, minus 1,905 suggestion
|
|
79
|
+
generations).
|
|
80
|
+
- All existing baseline specs green; new filter specs green.
|
|
81
|
+
- Site repo spellcheck job measurably faster after its gem bump.
|