kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Plan 132 — The techniques page; professional prose across the site
|
|
2
|
+
|
|
3
|
+
Status: executed (2026-09-12, site push; this plan)
|
|
4
|
+
|
|
5
|
+
## What shipped
|
|
6
|
+
|
|
7
|
+
1. /docs/techniques — a complete catalog of the stack's techniques in
|
|
8
|
+
four groups (engines, models, delivery, reliability), each entry
|
|
9
|
+
naming the mechanism and its measured effect in full sentences.
|
|
10
|
+
Written as the site's first MDX page: @astrojs/mdx installed and
|
|
11
|
+
wired, DocsLayout applied through an imported component wrapper.
|
|
12
|
+
2. A professional rewrite of the published copy: 34 fragment-style
|
|
13
|
+
news titles became declarative sentences, and 69 fragment-stacked
|
|
14
|
+
summaries and senses across all 36 entries were rewritten as full
|
|
15
|
+
sentences with clear subjects. The worst patterns eliminated:
|
|
16
|
+
em-dash fragment leads ("What 1.0 freezes:"), colon-list openers
|
|
17
|
+
("Also in the patch:"), trailing fragment appends ("— no download,
|
|
18
|
+
no CI seeding"), and lowercase clause starts.
|
|
19
|
+
3. The docs pages received the same treatment where fragments
|
|
20
|
+
appeared: the performance error-budget section, the caching
|
|
21
|
+
language-packs section, and three headlines.
|
|
22
|
+
|
|
23
|
+
## Conventions established
|
|
24
|
+
|
|
25
|
+
- Text-heavy documentation pages are written in MDX, not .astro HTML.
|
|
26
|
+
- News titles are complete clauses; summaries and senses are full
|
|
27
|
+
sentences. Release titles use the "package X.Y.Z: description" form.
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# Plan 133 — Rust-first distribution: precompiled native gems and complete wheels
|
|
2
|
+
|
|
3
|
+
Status: proposed (execution-ready)
|
|
4
|
+
Depends on: the 1.0 conformance contract (identical outputs make the
|
|
5
|
+
backend default flip correctness-free); plan 93 (the python wheel matrix)
|
|
6
|
+
|
|
7
|
+
## Problem
|
|
8
|
+
|
|
9
|
+
The Rust engine is the faster implementation (~10-15x on the sweep) and
|
|
10
|
+
the strategic default, but today reaching it requires a build toolchain:
|
|
11
|
+
the gem compiles its native extension at install time, and the python
|
|
12
|
+
native wheel matrix is incomplete on PyPI. Users who cannot run a
|
|
13
|
+
toolchain get pure Ruby whether they wanted it or not — and users who
|
|
14
|
+
wanted Rust pay a compile for it. The architecture makes both engines
|
|
15
|
+
byte-identical, so distribution — not capability — is the only gap.
|
|
16
|
+
|
|
17
|
+
## Design
|
|
18
|
+
|
|
19
|
+
### Ruby: precompiled platform gems (the rb_sys flow)
|
|
20
|
+
|
|
21
|
+
1. New gem workflow `release-native.yml`: build matrix over
|
|
22
|
+
{x86_64-linux, aarch64-linux, x86_64-linux-musl, arm64-darwin,
|
|
23
|
+
x86_64-darwin, x64-mingw-ucrt} x {ruby 3.1..4.0} using
|
|
24
|
+
rb-sys-dock images for the Linux targets and native runners for
|
|
25
|
+
macOS/Windows; each job builds and pushes
|
|
26
|
+
`kotoshu-X.Y.Z-<platform>.gem` through trusted publishing. The
|
|
27
|
+
pure-Ruby platform gem continues to publish exactly as today, so
|
|
28
|
+
every unmatched platform resolves to it.
|
|
29
|
+
2. Backend default `ruby` -> `auto`: use the native engine when the
|
|
30
|
+
extension loads, fall back silently to pure Ruby when it does not.
|
|
31
|
+
Auto never downloads and never fails a check; specs pin both
|
|
32
|
+
branches; the conformance job already runs both engines.
|
|
33
|
+
3. The docker CI image drops its build-tool layer (it installs the
|
|
34
|
+
platform gem instead) — smaller image, faster builds, and its
|
|
35
|
+
version guard gains a native-backend assertion.
|
|
36
|
+
|
|
37
|
+
### Python: complete the kotoshu-native wheel matrix
|
|
38
|
+
|
|
39
|
+
1. Audit PyPI's published wheels for kotoshu-native against the
|
|
40
|
+
plan-93 matrix: manylinux x86_64/aarch64, macOS x86_64/arm64,
|
|
41
|
+
windows x86_64 (+ arm64 if tractable), sdist.
|
|
42
|
+
2. Fill the gaps in release-pypi.yml (maturin + the standard target
|
|
43
|
+
matrix), publish a patch release, and verify pip resolves a wheel
|
|
44
|
+
(no source build) on each platform.
|
|
45
|
+
3. Add the trove classifiers for every published target.
|
|
46
|
+
|
|
47
|
+
### Site
|
|
48
|
+
|
|
49
|
+
Install page's engines-and-bindings section gains the resolution
|
|
50
|
+
sentence: the native engine installs automatically wherever a prebuilt
|
|
51
|
+
artifact exists for your platform, and pure Ruby is always the
|
|
52
|
+
fallback; python's entry notes the wheel coverage.
|
|
53
|
+
|
|
54
|
+
## Gates
|
|
55
|
+
|
|
56
|
+
- A clean-machine `gem install kotoshu` on each published platform
|
|
57
|
+
loads the native backend with no toolchain present (CI-verifiable
|
|
58
|
+
on the matrix runners).
|
|
59
|
+
- The 2,630-vector conformance compare passes under `auto` on both a
|
|
60
|
+
native and a pure-Ruby install.
|
|
61
|
+
- pip on each target platform resolves a wheel, not the sdist.
|
|
62
|
+
- The pure-Ruby path is unchanged: same gem, same outputs, same specs.
|
|
63
|
+
|
|
64
|
+
## Non-goals
|
|
65
|
+
|
|
66
|
+
- No behavior change of any kind — outputs are frozen; only which
|
|
67
|
+
engine computes them changes.
|
|
68
|
+
- No deprecation of anything: pure Ruby is a feature, not a legacy.
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# 20 — Encapsulation cleanup (T3, deferred past 0.3)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Deferred past 0.3 release. Documented for 0.4.
|
|
5
|
+
|
|
6
|
+
## Problem
|
|
7
|
+
The codebase has three categories of encapsulation violations prohibited
|
|
8
|
+
by `~/.claude/CLAUDE.md`:
|
|
9
|
+
|
|
10
|
+
| Violation | Count | Source |
|
|
11
|
+
|---|---|---|
|
|
12
|
+
| `obj.send(:private_method, ...)` | 4 | breaks private boundary |
|
|
13
|
+
| `instance_variable_set` / `instance_variable_get` | 21 | breaks encapsulation of another object's state |
|
|
14
|
+
| `respond_to?(:method)` for type checks | 24 | hides type errors until runtime |
|
|
15
|
+
|
|
16
|
+
New code in 0.3 follows the rules; this file tracks migrating the
|
|
17
|
+
existing violations.
|
|
18
|
+
|
|
19
|
+
## Plan
|
|
20
|
+
|
|
21
|
+
### send to private (4 occurrences)
|
|
22
|
+
Each call site is a design smell — the API boundary is wrong. Fix per
|
|
23
|
+
location by either:
|
|
24
|
+
- Making the method public (if callers outside genuinely need it).
|
|
25
|
+
- Adding a public method that does what the caller wants (rename intent).
|
|
26
|
+
- Inlining the call at the (usually single) call site.
|
|
27
|
+
|
|
28
|
+
### instance_variable_set / instance_variable_get (21)
|
|
29
|
+
Replace each with a public accessor or a constructor argument. If the
|
|
30
|
+
target object genuinely needs to be mutated externally, expose a public
|
|
31
|
+
setter that maintains invariants.
|
|
32
|
+
|
|
33
|
+
### respond_to? for type checks (24)
|
|
34
|
+
Replace with `is_a?(SpecificType)` or — better — redesign so the type
|
|
35
|
+
hierarchy makes the check unnecessary (polymorphism).
|
|
36
|
+
|
|
37
|
+
## Acceptance
|
|
38
|
+
|
|
39
|
+
- [ ] `grep -rn "\.send(" lib/ | wc -l` is 0 (or all remaining are
|
|
40
|
+
legitimate metaprogramming, with an inline comment justifying each).
|
|
41
|
+
- [ ] `grep -rn "instance_variable_set\|instance_variable_get" lib/` is 0.
|
|
42
|
+
- [ ] `grep -rn "respond_to?" lib/` returns only duck-typing cases that
|
|
43
|
+
have been audited and have an inline comment.
|
|
44
|
+
|
|
45
|
+
## Why deferred
|
|
46
|
+
Touching 49 call sites risks regressions in working code. 0.3 priority
|
|
47
|
+
is user-facing usability. Sweep this in a dedicated 0.4 PR.
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
# 21 — Migrate require_relative to autoload (T3, deferred past 0.3)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Deferred past 0.3 release. Documented for 0.4.
|
|
5
|
+
|
|
6
|
+
## Problem
|
|
7
|
+
288 `require_relative` calls in `lib/`. Per `~/.claude/CLAUDE.md`:
|
|
8
|
+
|
|
9
|
+
> NEVER use `require_relative` for internal library code. Never use
|
|
10
|
+
> `require` with a path to code within your own library. Use Ruby
|
|
11
|
+
> `autoload` instead. Define autoload entries in the immediate parent
|
|
12
|
+
> namespace's file — create that file if it doesn't exist.
|
|
13
|
+
|
|
14
|
+
Current state:
|
|
15
|
+
- `lib/kotoshu.rb` eagerly `require_relative`s the traditional path
|
|
16
|
+
(core models, dictionaries, strategies, configuration, spellchecker)
|
|
17
|
+
and uses `autoload` for the rest.
|
|
18
|
+
- Many subdirs lack a parent-namespace file (e.g. `lib/kotoshu/models.rb`
|
|
19
|
+
does not exist).
|
|
20
|
+
|
|
21
|
+
## Plan
|
|
22
|
+
|
|
23
|
+
### Phase 1 — Create missing parent namespace files
|
|
24
|
+
Create each of:
|
|
25
|
+
```
|
|
26
|
+
lib/kotoshu/models.rb
|
|
27
|
+
lib/kotoshu/dictionaries.rb
|
|
28
|
+
lib/kotoshu/dictionary.rb
|
|
29
|
+
lib/kotoshu/suggestions.rb
|
|
30
|
+
lib/kotoshu/analyzers.rb
|
|
31
|
+
lib/kotoshu/documents.rb
|
|
32
|
+
lib/kotoshu/components.rb
|
|
33
|
+
lib/kotoshu/plugins.rb
|
|
34
|
+
lib/kotoshu/data_structures.rb
|
|
35
|
+
lib/kotoshu/results.rb
|
|
36
|
+
lib/kotoshu/commands.rb
|
|
37
|
+
```
|
|
38
|
+
Each declares `module Kotoshu::<Ns>` with one `autoload :ClassName,
|
|
39
|
+
"kotoshu/<ns>/<class_name_in_snake_case>"` per file in the subdir.
|
|
40
|
+
|
|
41
|
+
### Phase 2 — Replace require_relative inside lib/
|
|
42
|
+
Inside `lib/kotoshu/**/*.rb`:
|
|
43
|
+
- Replace `require_relative "foo"` with `autoload :Foo, "kotoshu/<ns>/foo"`.
|
|
44
|
+
- The autoload declaration lives in the immediate parent namespace file
|
|
45
|
+
(created in Phase 1), not in the file that needs the class.
|
|
46
|
+
- The top-level `lib/kotoshu.rb` only `require_relative`s the namespace
|
|
47
|
+
files (`kotoshu/cache`, `kotoshu/models`, etc.), not individual classes.
|
|
48
|
+
|
|
49
|
+
### Phase 3 — Verify eager paths still work
|
|
50
|
+
Some classes must still load eagerly (the public facade in `lib/kotoshu.rb`).
|
|
51
|
+
Keep those as `require_relative "kotoshu/<ns>"` (which loads the parent
|
|
52
|
+
namespace file, which can either autoload or eager-require its children).
|
|
53
|
+
|
|
54
|
+
## Acceptance
|
|
55
|
+
|
|
56
|
+
- [ ] `grep -rn "require_relative" lib/ | wc -l` is ≤ ~10 (only top-level
|
|
57
|
+
`lib/kotoshu.rb` loading namespace files).
|
|
58
|
+
- [ ] `bundle exec rspec` passes with no regressions.
|
|
59
|
+
- [ ] Boot time (`bundle exec ruby -e 'require "kotoshu"; puts Kotoshu'`)
|
|
60
|
+
is no slower than today (autoload should make it faster).
|
|
61
|
+
|
|
62
|
+
## Why deferred
|
|
63
|
+
Touches every file in `lib/`. High regression surface. New 0.3 code
|
|
64
|
+
follows the autoload rule; full migration is a dedicated 0.4 effort.
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
# 22 — Single source of truth for URL + pin config (T1, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
|
|
5
|
+
|
|
6
|
+
Evidence:
|
|
7
|
+
|
|
8
|
+
- `lib/kotoshu/source_registry.rb` — `SourceRegistry` with a frozen
|
|
9
|
+
`SOURCES` map (spelling/grammar/dict_manifest/frequency/freq_manifest/
|
|
10
|
+
model/model_vocab/model_manifest/model_registry), per-repo
|
|
11
|
+
`default_pin` (`dictionaries` → `v1`, the other repos → `main`),
|
|
12
|
+
`url_for` / `pin_for_source`. Zero `require_relative`; autoloaded from
|
|
13
|
+
`lib/kotoshu.rb:46`.
|
|
14
|
+
- `Configuration` exposes `repos_base_url` (`KOTOSHU_REPOS_BASE_URL`),
|
|
15
|
+
`dictionaries_pin` (`KOTOSHU_DICTIONARIES_PIN`, default `v1`),
|
|
16
|
+
`frequency_pin`, `models_pin` (`lib/kotoshu/configuration.rb:75-98`)
|
|
17
|
+
and builds the registry via `#source_registry` (ibid., line 478).
|
|
18
|
+
- All three caches build URLs through the registry — no inline URL
|
|
19
|
+
strings: `language_cache.rb:195,204,233,261,295`,
|
|
20
|
+
`frequency_cache.rb:232,239`, `model_cache.rb` (now registry-driven,
|
|
21
|
+
see below).
|
|
22
|
+
- Spec: `spec/kotoshu/source_registry_spec.rb` — run 2026-09-03,
|
|
23
|
+
19 examples, 0 failures.
|
|
24
|
+
- Live check (2026-09-03):
|
|
25
|
+
`Kotoshu::SourceRegistry.new.url_for(:spelling, lang: "en", ext: "aff")`
|
|
26
|
+
→ `https://raw.githubusercontent.com/kotoshu/dictionaries/v1/en/spelling/index.aff`
|
|
27
|
+
(exactly the acceptance URL).
|
|
28
|
+
|
|
29
|
+
Deviations from the plan text:
|
|
30
|
+
|
|
31
|
+
- `dictionaries_url` / `models_url` were not removed — they remain in
|
|
32
|
+
`SCHEMA` marked "Deprecated" and are consumed nowhere outside
|
|
33
|
+
`configuration.rb`.
|
|
34
|
+
- The model source later evolved past URL templates into a tiered
|
|
35
|
+
`registry.json` (`lib/kotoshu/cache/model_registry.rb`,
|
|
36
|
+
TODO.impl/67 M1: mini/fluency/full tiers).
|
|
37
|
+
|
|
38
|
+
## Problem
|
|
39
|
+
Today there are three overlapping configuration knobs for where to fetch
|
|
40
|
+
resources from:
|
|
41
|
+
|
|
42
|
+
```ruby
|
|
43
|
+
dictionaries_url: "https://raw.githubusercontent.com/kotoshu/dictionaries/main"
|
|
44
|
+
resource_pin: "main" # used by language_cache, frequency_cache
|
|
45
|
+
models_url: "https://github.com/kotoshu/models-fasttext-onnx/raw/main"
|
|
46
|
+
models_pin: "main" # used by model_cache
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Three problems:
|
|
50
|
+
|
|
51
|
+
1. **`/main` is baked into `dictionaries_url` even though `resource_pin`
|
|
52
|
+
is meant to be the branch selector.** The URL builder in
|
|
53
|
+
`lib/kotoshu/cache/language_cache.rb:181` is:
|
|
54
|
+
```ruby
|
|
55
|
+
"#{@url_base}/dictionaries/#{@resource_pin}/#{language}/spelling/index.aff"
|
|
56
|
+
```
|
|
57
|
+
So the URL is `<url_base>/dictionaries/<pin>/<lang>/...` — and the
|
|
58
|
+
`dictionaries/main` in the default URL is dead weight, never substituted.
|
|
59
|
+
|
|
60
|
+
2. **Each repo has its own default branch.** `kotoshu/dictionaries` is
|
|
61
|
+
on `v1`; `frequency-list-kelly` and `models-fasttext-onnx` are on
|
|
62
|
+
`main`. A single `resource_pin` cannot honor both.
|
|
63
|
+
|
|
64
|
+
3. **URL templates are scattered.** `language_cache.rb`, `frequency_cache.rb`,
|
|
65
|
+
`model_cache.rb` each build their own URL strings. No single source.
|
|
66
|
+
|
|
67
|
+
## Plan
|
|
68
|
+
|
|
69
|
+
### Introduce `Kotoshu::SourceRegistry`
|
|
70
|
+
|
|
71
|
+
A new model class — single source of truth for "where does each resource
|
|
72
|
+
type live, and at what pin."
|
|
73
|
+
|
|
74
|
+
```ruby
|
|
75
|
+
# lib/kotoshu/source_registry.rb
|
|
76
|
+
module Kotoshu
|
|
77
|
+
class SourceRegistry
|
|
78
|
+
Source = Struct.new(:repo, :default_pin, :url_template, keyword_init: true)
|
|
79
|
+
|
|
80
|
+
SOURCES = {
|
|
81
|
+
spelling: Source.new(repo: "dictionaries", default_pin: "v1",
|
|
82
|
+
url_template: "%{base}/dictionaries/%{repo}/%{pin}/%{lang}/spelling/index.%{ext}"),
|
|
83
|
+
frequency: Source.new(repo: "frequency-list-kelly", default_pin: "main",
|
|
84
|
+
url_template: "%{base}/frequency-list-kelly/%{pin}/data/%{lang}.json"),
|
|
85
|
+
model: Source.new(repo: "models-fasttext-onnx", default_pin: "main",
|
|
86
|
+
url_template: "%{base}/models-fasttext-onnx/%{pin}/models/%{lang}/fasttext.%{lang}.onnx"),
|
|
87
|
+
manifest: Source.new(repo: "dictionaries", default_pin: "v1",
|
|
88
|
+
url_template: "%{base}/dictionaries/%{pin}/manifest.json")
|
|
89
|
+
}.freeze
|
|
90
|
+
|
|
91
|
+
def self.url_for(resource_kind, lang: nil, ext: nil, base: nil, pin: nil)
|
|
92
|
+
# ...
|
|
93
|
+
end
|
|
94
|
+
end
|
|
95
|
+
end
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### Replace `dictionaries_url`, `resource_pin`, `models_url`, `models_pin`
|
|
99
|
+
|
|
100
|
+
In `Configuration`:
|
|
101
|
+
- Keep `repos_base_url` (one config knob — the GitHub raw root).
|
|
102
|
+
- Keep per-source pin overrides: `dictionaries_pin`, `frequency_pin`,
|
|
103
|
+
`models_pin` — but with correct defaults (`v1` for dictionaries).
|
|
104
|
+
- Remove `dictionaries_url`, `models_url`, `resource_pin` (drift-prone).
|
|
105
|
+
|
|
106
|
+
Caches call `SourceRegistry.url_for(:spelling, lang: "en", ext: "aff")`
|
|
107
|
+
instead of building strings.
|
|
108
|
+
|
|
109
|
+
## Acceptance
|
|
110
|
+
|
|
111
|
+
- [ ] `SourceRegistry` exists, is fully tested, has zero `require_relative`.
|
|
112
|
+
- [ ] `Configuration` exposes only `repos_base_url`, `dictionaries_pin`,
|
|
113
|
+
`frequency_pin`, `models_pin`.
|
|
114
|
+
- [ ] Each cache class no longer builds URL strings inline — calls
|
|
115
|
+
`SourceRegistry.url_for`.
|
|
116
|
+
- [ ] Default URL for `kotoshu setup en` resolves to a real file
|
|
117
|
+
(i.e. `https://raw.githubusercontent.com/kotoshu/dictionaries/v1/en/spelling/index.aff`).
|
|
118
|
+
|
|
119
|
+
## Dependency
|
|
120
|
+
- Blocks #23 (URL defaults fix), #29 (status command needs to show URLs).
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# 23 — Fix URL + pin defaults so first-use doesn't 404 (T1, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
|
|
5
|
+
|
|
6
|
+
Evidence (all run 2026-09-03):
|
|
7
|
+
|
|
8
|
+
- `NETWORK_TESTS=1 bundle exec rspec spec/integration/end_to_end_spec.rb`
|
|
9
|
+
→ 12 examples, 0 failures — real download of `en` spelling from
|
|
10
|
+
`kotoshu/dictionaries` into an isolated tmp XDG home; setup returns
|
|
11
|
+
`:downloaded`, `correct?("hello")` true, `suggest("teh")` includes
|
|
12
|
+
`"the"`. No 404s.
|
|
13
|
+
- Live German acceptance, isolated tmp XDG home:
|
|
14
|
+
`Kotoshu.setup(:de, want: %i[spelling]).spelling` → `:downloaded`.
|
|
15
|
+
- Root cause fixed structurally: dictionaries default pin is `v1` in
|
|
16
|
+
`SourceRegistry::SOURCES[:spelling]` (`lib/kotoshu/source_registry.rb:22`).
|
|
17
|
+
|
|
18
|
+
The `en`/`de` correct?/suggest acceptance items were also exercised
|
|
19
|
+
through the real dev cache via the README quickstart (see #28).
|
|
20
|
+
|
|
21
|
+
## Problem
|
|
22
|
+
Today `Kotoshu.setup(:en)` fails:
|
|
23
|
+
|
|
24
|
+
```
|
|
25
|
+
$ KOTOSHU_OFFLINE=0 bundle exec ruby -e 'Kotoshu.setup(:en)'
|
|
26
|
+
# => 404 from raw.githubusercontent.com/kotoshu/dictionaries/main/...
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Two causes:
|
|
30
|
+
1. `dictionaries_url` default ends in `/main` but the dictionaries repo
|
|
31
|
+
default branch is `v1`.
|
|
32
|
+
2. `resource_pin` default is `main` regardless of repo.
|
|
33
|
+
|
|
34
|
+
## Plan
|
|
35
|
+
|
|
36
|
+
### Step 1 — Implement TODO #22 (`SourceRegistry`)
|
|
37
|
+
Done first.
|
|
38
|
+
|
|
39
|
+
### Step 2 — Update default pin for dictionaries to `v1`
|
|
40
|
+
Via `SourceRegistry::SOURCES[:spelling].default_pin = "v1"`.
|
|
41
|
+
|
|
42
|
+
### Step 3 — Verify end-to-end
|
|
43
|
+
```ruby
|
|
44
|
+
Kotoshu.setup(:en, want: %i[spelling]).spelling # => :downloaded
|
|
45
|
+
Kotoshu.correct?("hello") # => true
|
|
46
|
+
Kotoshu.suggest("teh").include?("the") # => true
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
## Acceptance
|
|
50
|
+
|
|
51
|
+
- [ ] Live run of `Kotoshu.setup(:en)` returns `:downloaded` (not 404).
|
|
52
|
+
- [ ] Live run of `Kotoshu.setup(:de)` returns `:downloaded`.
|
|
53
|
+
- [ ] `Kotoshu.correct?("hello")` returns `true` after setup.
|
|
54
|
+
- [ ] `Kotoshu.suggest("teh")` returns a list that includes `"the"`.
|
|
55
|
+
|
|
56
|
+
## Dependencies
|
|
57
|
+
- #22 (SourceRegistry)
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# 24 — CLI auto-setup prompt (T1, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
|
|
5
|
+
|
|
6
|
+
Evidence:
|
|
7
|
+
|
|
8
|
+
- `lib/kotoshu/cli/auto_setup.rb` — `Kotoshu::Cli::AutoSetup` exactly as
|
|
9
|
+
planned: non-TTY / offline → returns nil (no prompt); TTY → prints
|
|
10
|
+
`Language 'en' is not set up (missing spelling). Download now (~5 MB
|
|
11
|
+
from github.com/kotoshu/dictionaries)? [Y/n]`; on yes runs
|
|
12
|
+
`Kotoshu.setup(language, want:)`.
|
|
13
|
+
- Wired in the CLI dispatcher: `lib/kotoshu/cli.rb:307-308` rescues
|
|
14
|
+
`Kotoshu::ResourceNotSetupError`, delegates to `AutoSetup`, and
|
|
15
|
+
re-raises as `Errors::ResourceUnavailable` (exit 3) when declined.
|
|
16
|
+
- Library API stays strict — only the CLI layer prompts.
|
|
17
|
+
- Spec: `spec/kotoshu/cli/auto_setup_spec.rb` — run 2026-09-03,
|
|
18
|
+
0 failures. Exit code 3 is documented in the `check` long_desc and
|
|
19
|
+
README.adoc ("language not set up").
|
|
20
|
+
|
|
21
|
+
Not re-verified interactively today (requires a TTY); the non-TTY and
|
|
22
|
+
prompt logic are spec-covered.
|
|
23
|
+
|
|
24
|
+
## Problem
|
|
25
|
+
Today:
|
|
26
|
+
|
|
27
|
+
```
|
|
28
|
+
$ kotoshu check README.md
|
|
29
|
+
# (stack trace from ResourceNotSetupError)
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The library raises `ResourceNotSetupError` with a clear message — but
|
|
33
|
+
the CLI surfaces it as a stack trace. Users who install the gem and run
|
|
34
|
+
`kotoshu check` on day one don't know to run `kotoshu setup` first.
|
|
35
|
+
|
|
36
|
+
The two-stage setup/resolve split is correct for the **library** API
|
|
37
|
+
(see `lib/kotoshu/resource_manager.rb` — explicit setup, cache-only hot
|
|
38
|
+
path, no surprise downloads on metered networks). But the **CLI** should
|
|
39
|
+
be the friendly face.
|
|
40
|
+
|
|
41
|
+
## Plan
|
|
42
|
+
|
|
43
|
+
### Add `Kotoshu::Cli::AutoSetup` middleware
|
|
44
|
+
|
|
45
|
+
Catch `ResourceNotSetupError` in `Cli.start` (the existing dispatch
|
|
46
|
+
override, per project memory `kotoshu-cli-semantic-errors`). When caught:
|
|
47
|
+
|
|
48
|
+
1. Detect TTY. If non-TTY (piped output, CI), re-raise with the existing
|
|
49
|
+
message — no surprise interactive prompts in scripts.
|
|
50
|
+
2. If TTY, prompt:
|
|
51
|
+
```
|
|
52
|
+
Language 'en' is not set up (missing spelling).
|
|
53
|
+
Download now (~5 MB from github.com/kotoshu/dictionaries)? [Y/n]
|
|
54
|
+
```
|
|
55
|
+
3. On yes: call `Kotoshu.setup(lang, want: %i[spelling])`, then retry
|
|
56
|
+
the original command.
|
|
57
|
+
4. On no: exit 3 with the original message.
|
|
58
|
+
5. Honor `--offline` flag and `KOTOSHU_OFFLINE=1` env var — never prompt
|
|
59
|
+
in offline mode, just exit with the error.
|
|
60
|
+
|
|
61
|
+
### Why this respects the two-stage model
|
|
62
|
+
The library API (`Kotoshu.correct?`, `Kotoshu.suggest`) still raises
|
|
63
|
+
strictly. Only the human-facing CLI adds the prompt layer. Programmatic
|
|
64
|
+
users get the strict behavior they need.
|
|
65
|
+
|
|
66
|
+
## Acceptance
|
|
67
|
+
|
|
68
|
+
- [ ] `kotoshu check file.txt` on a fresh cache prompts interactively.
|
|
69
|
+
- [ ] `echo "text" | kotoshu check -` (non-TTY) does NOT prompt — exits 3
|
|
70
|
+
with the original error message.
|
|
71
|
+
- [ ] `KOTOSHU_OFFLINE=1 kotoshu check file.txt` does NOT prompt — exits 3.
|
|
72
|
+
- [ ] After "y" at the prompt, the check completes successfully.
|
|
73
|
+
- [ ] Existing CLI tests still pass; new spec covers the prompt path.
|
|
74
|
+
|
|
75
|
+
## Dependencies
|
|
76
|
+
- #23 (setup must actually work first)
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# 25 — Soft ONNX runtime require (T1, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
|
|
5
|
+
|
|
6
|
+
Evidence:
|
|
7
|
+
|
|
8
|
+
- `kotoshu.gemspec` runtime deps are only `thor`, `rubyzip`,
|
|
9
|
+
`lutaml-model` — no `onnxruntime` (nor `suika`; both documented as
|
|
10
|
+
soft deps in comments).
|
|
11
|
+
- `lib/kotoshu/models/onnx_model.rb:25-34` — load-time
|
|
12
|
+
`begin require "onnxruntime" ... rescue LoadError` sets
|
|
13
|
+
`ONNX_LOADED = false`; `KOTOSHU_NO_ONNX=1` forces it off (opt-out,
|
|
14
|
+
replacing the old opt-in).
|
|
15
|
+
- Actionable error: `OnnxModel::OnnxUnavailable` (ibid., lines 38-46)
|
|
16
|
+
— "Install with: gem install onnxruntime".
|
|
17
|
+
- `KOTOSHU_REQUIRE_ONNX` is gone from `lib/` and the docs; the only
|
|
18
|
+
remaining mentions are historical (CHANGELOG 0.1.0 entry) and in the
|
|
19
|
+
plan files themselves.
|
|
20
|
+
- Live: `bundle exec exe/kotoshu status` reports
|
|
21
|
+
`onnxruntime loaded` on this machine.
|
|
22
|
+
|
|
23
|
+
## Problem
|
|
24
|
+
`onnxruntime` is a runtime dependency in `kotoshu.gemspec`:
|
|
25
|
+
```ruby
|
|
26
|
+
spec.add_dependency "onnxruntime", "~> 0.10"
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
But `Models::OnnxModel` only does `require "onnxruntime"` when
|
|
30
|
+
`ENV['KOTOSHU_REQUIRE_ONNX']` is set. Worst of both worlds:
|
|
31
|
+
|
|
32
|
+
1. `bundle install` can fail on platforms where `onnxruntime` doesn't
|
|
33
|
+
build — the whole gem becomes unusable even for non-semantic use.
|
|
34
|
+
2. Even when installed, semantic analysis silently stays off until the
|
|
35
|
+
user sets an env var they have no way to discover.
|
|
36
|
+
|
|
37
|
+
## Plan
|
|
38
|
+
|
|
39
|
+
### Step 1 — Move onnxruntime to soft dependency
|
|
40
|
+
Remove from `spec.add_dependency`. Document in README that semantic
|
|
41
|
+
features require `gem install onnxruntime` (or `bundle add onnxruntime`).
|
|
42
|
+
|
|
43
|
+
### Step 2 — Replace env-var gate with load-time rescue
|
|
44
|
+
In `lib/kotoshu/models/onnx_model.rb`:
|
|
45
|
+
```ruby
|
|
46
|
+
begin
|
|
47
|
+
require "onnxruntime"
|
|
48
|
+
ONNX_LOADED = true
|
|
49
|
+
rescue LoadError => e
|
|
50
|
+
ONNX_LOADED = false
|
|
51
|
+
warn "kotoshu: onnxruntime not available (#{e.message}); semantic analysis disabled"
|
|
52
|
+
end
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### Step 3 — Fail with actionable error when semantic is requested but unavailable
|
|
56
|
+
`OnnxModel.from_file` raises `SemanticError` (already exists) with:
|
|
57
|
+
```
|
|
58
|
+
onnxruntime gem not installed. Install with: gem install onnxruntime
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### Step 4 — Drop `KOTOSHU_REQUIRE_ONNX` env var
|
|
62
|
+
If users want to disable ONNX explicitly (e.g. to avoid the load warning),
|
|
63
|
+
support `KOTOSHU_NO_ONNX=1` as opt-OUT (inverted from current opt-IN).
|
|
64
|
+
|
|
65
|
+
## Acceptance
|
|
66
|
+
|
|
67
|
+
- [ ] `gem install kotoshu` succeeds without `onnxruntime` installed.
|
|
68
|
+
- [ ] `Kotoshu.correct?("hello")` works without `onnxruntime` (traditional path).
|
|
69
|
+
- [ ] Calling semantic methods without onnxruntime raises a clear `SemanticError`.
|
|
70
|
+
- [ ] After `gem install onnxruntime`, semantic methods work without any env var.
|
|
71
|
+
- [ ] `KOTOSHU_NO_ONNX=1` disables semantic even when onnxruntime is installed.
|
|
72
|
+
- [ ] No mention of `KOTOSHU_REQUIRE_ONNX` in code or docs.
|
|
73
|
+
|
|
74
|
+
## Dependencies
|
|
75
|
+
- None (independent).
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# 26 — Generate .vocab.json for 9 active ONNX models (T1, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented — and superseded in scope (re-verified 2026-09-03 against 0.6.0).
|
|
5
|
+
|
|
6
|
+
Evidence:
|
|
7
|
+
|
|
8
|
+
- `kotoshu/models-fasttext-onnx` now ships 27 `.vocab.json` files on
|
|
9
|
+
disk (9 languages × 3 tiers: full/fluency/mini), e.g.
|
|
10
|
+
`models/en/fasttext.en.vocab.json`, `models/en/fasttext.en.mini.vocab.json`;
|
|
11
|
+
`manifest.json` carries them with `"type": "vocab"` entries.
|
|
12
|
+
- Gem side consumes them: `lib/kotoshu/embeddings/vocabulary.rb`
|
|
13
|
+
(`Embeddings::Vocabulary.from_file` line 137, `.from_cache` line 269)
|
|
14
|
+
and `Models::OnnxModel.from_file` requires the `.vocab.json` sibling
|
|
15
|
+
(`lib/kotoshu/models/onnx_model.rb:86-93`).
|
|
16
|
+
- Cache side: `SourceRegistry` has a `model_vocab` source and
|
|
17
|
+
`ModelCache` fetches vocab siblings with SHA-256 verification against
|
|
18
|
+
the registry (`lib/kotoshu/cache/model_cache.rb`).
|
|
19
|
+
|
|
20
|
+
Superseded: the "9 flat models" world was replaced by the tiered
|
|
21
|
+
registry (registry.json, mini/fluency/full — TODO.impl/67 M1), so the
|
|
22
|
+
plan's manifest-with-18-entries shape no longer applies.
|
|
23
|
+
|
|
24
|
+
Not re-verified live today: `Kotoshu.setup(:en, want: %i[model])`
|
|
25
|
+
(114 MB download; `kotoshu status` shows model ✗ in the dev cache).
|
|
26
|
+
|
|
27
|
+
## Problem
|
|
28
|
+
The 9 active ONNX models in `kotoshu/models-fasttext-onnx` have no
|
|
29
|
+
`.vocab.json` siblings:
|
|
30
|
+
```
|
|
31
|
+
models/de/fasttext.de.onnx # 114 MB
|
|
32
|
+
models/de/fasttext.de.vocab.json # MISSING
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
`OnnxModel.from_file` in the gem requires both files — the `.onnx` holds
|
|
36
|
+
only the embedding matrix; word → row index lives in `.vocab.json`.
|
|
37
|
+
Without it, every semantic lookup crashes.
|
|
38
|
+
|
|
39
|
+
The conversion script changes I made earlier (always emit vocab.json)
|
|
40
|
+
were post-hoc — they apply to FUTURE conversions, not the existing 158
|
|
41
|
+
files on disk.
|
|
42
|
+
|
|
43
|
+
## Plan
|
|
44
|
+
|
|
45
|
+
### Step 1 — Add a vocab-only extraction script
|
|
46
|
+
New script `scripts/extract_vocab_from_vec.py`:
|
|
47
|
+
- Input: FastText `.vec` file path + vocab size (default 100K)
|
|
48
|
+
- Output: `<stem>.vocab.json` matching the format emitted by
|
|
49
|
+
`fasttext_to_onnx.py`'s `save_vocabulary()`.
|
|
50
|
+
|
|
51
|
+
This is faster than re-running the full ONNX conversion (which would
|
|
52
|
+
also re-emit the 114 MB .onnx unnecessarily).
|
|
53
|
+
|
|
54
|
+
### Step 2 — Download the 9 source .vec files from FastText CDN
|
|
55
|
+
~7 GB total (compressed .vec.gz, ~700 MB per language). Cache under
|
|
56
|
+
`/tmp/fasttext-vec/` so re-runs don't re-download.
|
|
57
|
+
|
|
58
|
+
For each active language (de, en, es, fr, pt, ru, zh, ja, ko):
|
|
59
|
+
```
|
|
60
|
+
curl -O https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.<lang>.300.vec.gz
|
|
61
|
+
gunzip cc.<lang>.300.vec.gz
|
|
62
|
+
python3 scripts/extract_vocab_from_vec.py cc.<lang>.300.vec \
|
|
63
|
+
--vocab-size 100000 \
|
|
64
|
+
--output models/<lang>/fasttext.<lang>.vocab.json
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
### Step 3 — Verify vocab.json matches the existing .onnx
|
|
68
|
+
Each `.onnx` has `vocab_size` in its metadata. The `.vocab.json` must
|
|
69
|
+
have the same `vocab_size` AND the word order must match what was used
|
|
70
|
+
to build the embedding matrix. This is the load-bearing correctness
|
|
71
|
+
check — a mismatch makes embeddings return wrong vectors.
|
|
72
|
+
|
|
73
|
+
Verify by spot-checking a known word (e.g. "the" in en) and confirming
|
|
74
|
+
the embedding matches a reference value.
|
|
75
|
+
|
|
76
|
+
### Step 4 — Regenerate manifest + commit + push
|
|
77
|
+
The 9 new `.vocab.json` files get added to `manifest.json` via the
|
|
78
|
+
existing `generate_manifest.rb` script.
|
|
79
|
+
|
|
80
|
+
## Acceptance
|
|
81
|
+
|
|
82
|
+
- [ ] 9 `.vocab.json` files committed under `models/<lang>/`.
|
|
83
|
+
- [ ] `manifest.json` includes 18 entries (9 .onnx + 9 .vocab.json).
|
|
84
|
+
- [ ] Gem's `AVAILABLE_MODELS[:onnx]` unchanged (no code change).
|
|
85
|
+
- [ ] Live: `Kotoshu.setup(:en, want: %i[model])` succeeds.
|
|
86
|
+
- [ ] Live: `Kotoshu::Models::OnnxModel.from_file(...)` returns a working
|
|
87
|
+
model that can produce an embedding for "the".
|
|
88
|
+
|
|
89
|
+
## Dependencies
|
|
90
|
+
- None (independent of gem code).
|
|
91
|
+
|
|
92
|
+
## Risks
|
|
93
|
+
- 7 GB download may saturate a metered connection. Run on a connection
|
|
94
|
+
without caps.
|
|
95
|
+
- Vocab ordering mismatch with original conversion would silently
|
|
96
|
+
corrupt semantic results. Must verify by embedding spot-check.
|