kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,813 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
- **Korean and Nepali full-feature modules** (plan 108) - the last
|
|
12
|
+
big-population languages without modules move from the plan 107
|
|
13
|
+
basic tier to full feature. `ko` ships the Hangul tokenizer
|
|
14
|
+
(an eojeol is a run of Hangul syllables U+AC00-D7A3 plus bare jamo
|
|
15
|
+
U+1100-11FF/U+3130-318F that escaped IME composition) and the
|
|
16
|
+
Dubeolsik keyboard grid keyed on jamo per the KS X 5002 standard 2-set
|
|
17
|
+
layout, so typo proximity lands on the unit vowel mergers and
|
|
18
|
+
fortis/lenis confusions operate on. `ne` ships the Devanagari
|
|
19
|
+
tokenizer (matras U+093E-094C, the virama U+094D and vowel signs
|
|
20
|
+
stay attached to the base consonant; the danda and Devanagari
|
|
21
|
+
digits are separators) and the Devanagari InScript grid (the Indian
|
|
22
|
+
national standard, mirrored from the models repo eval harness and
|
|
23
|
+
drift-checked against it; the Nepali Traditional Romanized layout
|
|
24
|
+
is more common on the ground but is not a documented standard).
|
|
25
|
+
Both dictionaries were already staged, so `kotoshu setup ko ne`
|
|
26
|
+
serves them; full-feature languages grow 33 to 35.
|
|
27
|
+
- **Basic support for every staged dictionary language** (plan 107) -
|
|
28
|
+
`kotoshu setup` now serves all 95 languages staged in the
|
|
29
|
+
kotoshu/dictionaries manifest, not just the ones with gem modules.
|
|
30
|
+
`LanguageCache::AVAILABLE_LANGUAGES` derives from a vendored
|
|
31
|
+
snapshot of the manifest (Cache::StagedLanguages, refreshed by
|
|
32
|
+
`rake kotoshu:staged_languages:sync`), keeping the hot path
|
|
33
|
+
network-free. Languages without a module run on the basic tier:
|
|
34
|
+
script classification (Language::Script) picks the tokenizer whose
|
|
35
|
+
word regex drives check/suggest word extraction (shared Latin /
|
|
36
|
+
Cyrillic / Greek tokenizers plus a generic regex tokenizer for
|
|
37
|
+
Arabic, Hebrew, Armenian, Georgian, Devanagari and Hangul), and the
|
|
38
|
+
keyboard registry falls back to the script's standard grid (JCUKEN
|
|
39
|
+
for Cyrillic, Arabic 101, Hebrew SI-1452, QWERTY otherwise) instead
|
|
40
|
+
of QWERTY for everything. Two documented tiers: full feature =
|
|
41
|
+
module (registered keyboard + verified specimens,
|
|
42
|
+
`LanguageCache.full_feature_languages`), basic = manifest-present.
|
|
43
|
+
`nn` (Norwegian Nynorsk) is wired as a full-feature module while
|
|
44
|
+
here - dictionary and model are both staged. Modules keep winning
|
|
45
|
+
over the fallbacks; the 32 existing modules behave identically
|
|
46
|
+
(conformance vectors untouched).
|
|
47
|
+
- **Personal dictionary in the check path** (plan 105) -
|
|
48
|
+
`Spellchecker#check` (and therefore `Kotoshu.check`,
|
|
49
|
+
`Kotoshu.check_file`, and `kotoshu check`) no longer flags words
|
|
50
|
+
present in the user personal dictionary
|
|
51
|
+
(~/.config/kotoshu/personal.dic, KOTOSHU_PERSONAL_DIC override),
|
|
52
|
+
matching the semantics the LSP already applies to diagnostics:
|
|
53
|
+
case-insensitive, filtered at result assembly without suppression
|
|
54
|
+
metadata, words still counted, and `suggest`/`correct?` untouched so
|
|
55
|
+
personal words stay spellcheckable. The dictionary is loaded once
|
|
56
|
+
per process (no hot reload; the LSP keeps its mtime-based reload).
|
|
57
|
+
Opt out with `Kotoshu.configure { |c| c.personal_dictionary = false }`
|
|
58
|
+
/ KOTOSHU_PERSONAL_DICTIONARY=false / `kotoshu check --no-personal`.
|
|
59
|
+
- **Native language detection** (plan 106) - `Kotoshu.detect_language(text)`
|
|
60
|
+
now returns a `Kotoshu::Language::Detection` (`code` + `score`) scored by
|
|
61
|
+
the Rust extension over the `kotoshu://models/lid/lid-176` registry
|
|
62
|
+
artifact pair (the ONNX-converted lid.176 model, 176 languages,
|
|
63
|
+
bit-faithful to the fastText bindings - parity 55/55 against the frozen
|
|
64
|
+
corpus shared with kotoshu-rs). Fetch the pair once with
|
|
65
|
+
`Kotoshu.setup_lid` (registry-resolved, sha-verified, cached under
|
|
66
|
+
`models-cache/lid/`); detect resolves cache-only and never downloads.
|
|
67
|
+
The 7-language `Language::Detector` heuristic stays the fallback when
|
|
68
|
+
the extension is absent, the model is not set up, or
|
|
69
|
+
`KOTOSHU_BACKEND=ruby` opts out - so pure-Ruby installs are unaffected
|
|
70
|
+
(no new gemspec dependency, no onnxruntime). The magnus shim
|
|
71
|
+
(ext/kotoshu_native) gains `Kotoshu::Native::LidModel.load` +
|
|
72
|
+
`#detect` behind the kotoshu-rs `model` feature; the workspace
|
|
73
|
+
Cargo.lock moves to the kotoshu-rs revision carrying the reader.
|
|
74
|
+
`Kotoshu.detect_language_with_confidence` keeps its `[code, score]`
|
|
75
|
+
shape and now routes through the same engine.
|
|
76
|
+
- **Full-feature language batch 3** (plan 100) - twelve languages gain the
|
|
77
|
+
downloadable spelling dictionary, a keyboard layout, and a gem module:
|
|
78
|
+
`ar` `id` `fa` `he` `bg` `sr` `hr` `sk` `sl` `lt` `lv` `et` (the RTL
|
|
79
|
+
ar/fa/he modules are promoted from tokenizer-only wiring; the other nine
|
|
80
|
+
are new thin modules - id hr sl sk et lt lv on `LatinBase`, bg sr with
|
|
81
|
+
the Cyrillic tokenizer). Six national key grids join the layout registry
|
|
82
|
+
(Arabic 101, Persian standard ISIRI 9147, Hebrew SI-1452, Bulgarian BDS,
|
|
83
|
+
Serbian Cyrillic, and the shared Croatian/Slovenian QWERTZ), all mirrored
|
|
84
|
+
from the models repo eval harness and drift-checked against it; five new
|
|
85
|
+
Latin family members (Indonesian Slovak Estonian Lithuanian Latvian)
|
|
86
|
+
type on the US QWERTY grid like the eval model. Bulgarian and Serbian
|
|
87
|
+
register ahead of JCUKEN so their Cyrillic grids win the lookup. The
|
|
88
|
+
eval-grid drift fixture now covers all nine national grids. Specimen
|
|
89
|
+
misspelling -> correction pairs for every language are engine-verified
|
|
90
|
+
against the staged dictionaries in a `:network` spec.
|
|
91
|
+
|
|
92
|
+
### Fixed
|
|
93
|
+
- **RTL word extraction on the check path** - the plan-91 script-aware
|
|
94
|
+
word regex never reached the Arabic, Persian and Hebrew tokenizers, so
|
|
95
|
+
`Kotoshu.check` / `kotoshu check -l ar|fa|he` extracted zero words and
|
|
96
|
+
silently passed every RTL document. The three tokenizers now declare
|
|
97
|
+
their script sets (`\p{Arabic}`, `\p{Arabic}` + ZWNJ for Persian
|
|
98
|
+
compounds like میروم, `\p{Hebrew}` + geresh/gershayim), and RTL
|
|
99
|
+
misspellings surface with ranked suggestions.
|
|
100
|
+
- **Suggestion candidate sweep missed transpositions, substitutions, and
|
|
101
|
+
dictionary forms** - `EditDistanceStrategy` only scored raw dictionary
|
|
102
|
+
stems, so valid dictionary forms could never be suggested no matter how
|
|
103
|
+
close: an adjacent transposition paired with a case difference ("Teh" ->
|
|
104
|
+
"The") was charged as two operations, and affixed forms ("definately" ->
|
|
105
|
+
"definitely", a stem + `-ly` suffix never present as a stem) were absent
|
|
106
|
+
from the candidate set entirely. The strategy now also sweeps the
|
|
107
|
+
distance-1 edits of the misspelling itself - adjacent transposition
|
|
108
|
+
(restricted Damerau operation, cost 1), substitution and insertion over
|
|
109
|
+
the dictionary TRY string, and deletion - validating each against the
|
|
110
|
+
full affix-aware dictionary lookup, and case-variant duplicates of the
|
|
111
|
+
same word are deduplicated keeping the best-scoring form. The phonetic
|
|
112
|
+
and keyboard-proximity strategies now measure candidates with
|
|
113
|
+
`Algorithms::EditDistance` (Damerau-Levenshtein; previously each carried
|
|
114
|
+
a private plain-Levenshtein copy charging transpositions 2), and the
|
|
115
|
+
keyboard variant generator enumerates adjacent transpositions as a
|
|
116
|
+
single edit. `Dictionary::Base#try_string` / `Dictionary::Hunspell#try_string`
|
|
117
|
+
expose the Hunspell TRY directive for the sweep alphabet. Conformance
|
|
118
|
+
suggest vectors regenerated (84 of 1315 rows improved - affixed, compound,
|
|
119
|
+
and correctly-cased forms now appear; all 1315 correct vectors unchanged).
|
|
120
|
+
- **Dictionary mutation correctness** (from suleman-uzair, PR rescue of #93) -
|
|
121
|
+
`remove_word` deleted the wrong word after any prior removal because the
|
|
122
|
+
word set stored array indices that went stale; `Custom`, `PlainText`, and
|
|
123
|
+
`UnixWords` now track membership (word => true) and remove by value, so
|
|
124
|
+
every copy of a duplicate word is gone and `words` / `include?` stay
|
|
125
|
+
consistent across interleaved add/remove sequences. `PlainText#add_word` /
|
|
126
|
+
`#remove_word` now also keep the length-bucketed index in sync, so
|
|
127
|
+
`find_by_length_range` (the edit-distance candidate filter) reflects
|
|
128
|
+
post-construction mutations instead of returning stale buckets.
|
|
129
|
+
- **Edit-distance threshold on empty strings** (from suleman-uzair, PR rescue
|
|
130
|
+
of #93) - `Algorithms::EditDistance.distance_with_threshold` returned the
|
|
131
|
+
other string's length for empty inputs even when that exceeded the
|
|
132
|
+
threshold; the length pre-filter now runs before the empty-string
|
|
133
|
+
shortcuts, so the nil-above-threshold contract holds for empty strings.
|
|
134
|
+
- **Offline mode enforcement and cache-only suggestion hot path** (from
|
|
135
|
+
suleman-uzair, PR rescue of #93) - `Kotoshu::Configuration.offline` /
|
|
136
|
+
`KOTOSHU_OFFLINE=1` previously did not stop downloads. Every cache
|
|
137
|
+
download path now refuses in offline mode (`BaseCache#download` and the
|
|
138
|
+
`download_url` / `download_file` transports raise
|
|
139
|
+
`ResourceNotCachedError`; `Integrity::NetHTTP.get` refuses before opening
|
|
140
|
+
a connection; `LanguageCache#download_spelling` / `#download_grammar`
|
|
141
|
+
raise). The resolve paths (`ResourceManager`, `FrequencyProvider`) read
|
|
142
|
+
via the public cache-only `load_cached` instead of the download-on-miss
|
|
143
|
+
`get`, so the suggestion hot path never triggers a download.
|
|
144
|
+
`Configuration.default` no longer re-applies `DEFAULTS` as explicit
|
|
145
|
+
settings, which clobbered every `KOTOSHU_*` ENV-derived value including
|
|
146
|
+
`KOTOSHU_OFFLINE`. `kotoshu cache download` surfaces offline and
|
|
147
|
+
integrity failures as `ResourceUnavailable` instead of a stack trace.
|
|
148
|
+
- **Kelly frequency loader dead branch** (from suleman-uzair, PR rescue of
|
|
149
|
+
#93) - `CommonWordsLoader.load_from_frequency_file` kept a legacy
|
|
150
|
+
array-format branch that crashed on its own probe and had no producer;
|
|
151
|
+
it now parses the Kelly shape only and degrades unknown shapes to empty
|
|
152
|
+
tiers.
|
|
153
|
+
- Remote `kotoshu setup` for staged languages: the dictionaries repo ships only
|
|
154
|
+
`en` under the `{lang}/spelling/` layout while staged languages sit flat at
|
|
155
|
+
`{lang}/index.*`; downloads now try the sublayout first and fall back to the
|
|
156
|
+
flat layout (verified end-to-end for `it`).
|
|
157
|
+
- `require "kotoshu/tasks"` / `require "kotoshu/jekyll"` standalone now load
|
|
158
|
+
without needing `require "kotoshu"` first.
|
|
159
|
+
- Baseline paths compare canonically: entries recorded as `docs/a.md` match a
|
|
160
|
+
directory-walk check over `./docs/a.md`, and recorded entries are stored
|
|
161
|
+
canonical.
|
|
162
|
+
### Fixed
|
|
163
|
+
- **Unicode word detection** (plan 91 Track A) - `Spellchecker` word
|
|
164
|
+
extraction accepted only ASCII letters, so Greek and Ukrainian users
|
|
165
|
+
could not check any text through the CLI. Extraction now follows the
|
|
166
|
+
configured language's tokenizer: Greek words for `el`, Cyrillic for
|
|
167
|
+
`uk` (in-word apostrophe included, for names like Мар'яна), and
|
|
168
|
+
Latin-script letters including capitals such as Å and Ä for Latin
|
|
169
|
+
languages. Wrong-script words stay uncheckable; languages without a
|
|
170
|
+
script tokenizer (en, de, ...) keep the historical ASCII behavior;
|
|
171
|
+
the frozen conformance vectors are unchanged.
|
|
172
|
+
- **Swedish aff parsing** (plan 91 Track B) - `Readers::AffReader`
|
|
173
|
+
raised `RegexpError: unmatched close parenthesis` on the Swedish
|
|
174
|
+
dictionary (dictionaries pin 1829a3e). `COMPOUNDRULE )k` uses `)`
|
|
175
|
+
itself as a flag character, but the rule compiler escaped it only in
|
|
176
|
+
some positions. Every flag character is now escaped at compile time,
|
|
177
|
+
so the dictionary loads and its compound rules match.
|
|
178
|
+
### Added
|
|
179
|
+
- **Framework integrations** (plan 89) - four opt-in integration
|
|
180
|
+
layers, all inside the gem with NO new runtime dependencies
|
|
181
|
+
(soft-dependency pattern, like onnxruntime/suika):
|
|
182
|
+
- `Kotoshu::Validators::SpellingValidator`, an ActiveModel
|
|
183
|
+
EachValidator (`validates :body, spelling: true` after the
|
|
184
|
+
documented one-line alias, or `validates_with` directly): one
|
|
185
|
+
validation error per misspelling with the top suggestion in the
|
|
186
|
+
message; `language:` and `personal_words:` options. Loads the
|
|
187
|
+
real ActiveModel standalone - Rails is not required.
|
|
188
|
+
- RSpec matchers (`require "kotoshu/rspec"` +
|
|
189
|
+
`include Kotoshu::Rspec::Matchers`):
|
|
190
|
+
`expect_words("helo").to all_be_spelled_correctly`, and
|
|
191
|
+
`expect(text) / expect_document(path).to be_spelled_correctly`
|
|
192
|
+
for words or whole documents, with failure messages that list
|
|
193
|
+
each misspelling and its suggestions.
|
|
194
|
+
- Rake task (`require "kotoshu/tasks"`): `rake kotoshu:check`
|
|
195
|
+
over the repository text files using the plan-88 file selection
|
|
196
|
+
(known extensions, .gitignore/.ignore, default skips);
|
|
197
|
+
`Kotoshu::Tasks::CheckTask` exposes files/language/fail_on_error
|
|
198
|
+
/baseline configuration.
|
|
199
|
+
- Jekyll generator (`Kotoshu::Jekyll::Generator`, `safe`):
|
|
200
|
+
checks posts and drafts and fails the build on new spelling
|
|
201
|
+
errors; a `.kotoshu-baseline.json` in the site source is
|
|
202
|
+
respected so baselined errors do not block builds.
|
|
203
|
+
- **CLI directory mode** (plan 88) - `kotoshu check DIR [DIR ...]`
|
|
204
|
+
walks trees and checks every file with a known text extension
|
|
205
|
+
(md markdown asciidoc adoc txt rst mdx) unless `--include` /
|
|
206
|
+
`--exclude` globs say otherwise. `.gitignore` and `.ignore` files
|
|
207
|
+
are honored through a standard glob subset implemented in Ruby
|
|
208
|
+
(`*`, `?`, `**` as a whole segment, `!` negation with last-match
|
|
209
|
+
wins, trailing `/` for directory-only patterns, anchoring, nested
|
|
210
|
+
scoping; files inside ignored directories cannot be re-included).
|
|
211
|
+
Hidden files and `.git` / `node_modules` / `vendor` / `target` are
|
|
212
|
+
skipped by default. Text output prints one section per file plus a
|
|
213
|
+
summary; JSON and SARIF emit one combined document with per-file
|
|
214
|
+
entries/runs; `--baseline` applies per file as shipped; exit codes
|
|
215
|
+
are unchanged; interactive mode stays file-only with an explicit
|
|
216
|
+
notice for directory targets. Single-file and stdin behavior is
|
|
217
|
+
byte-identical to before.
|
|
218
|
+
- **Wave-1 language modules and keyboard layouts** (plan 84) -
|
|
219
|
+
language modules for the thirteen model-registry newcomers (ca cs
|
|
220
|
+
da el hu it nl pl ro sv tr uk vi): thin compositions over the shared
|
|
221
|
+
Latin tokenizer/normalizer plus registry entries, with real care
|
|
222
|
+
where the script needs it — Turkish dotless-i case folding
|
|
223
|
+
(`İZMİR` -> `izmir`, `ISTANBUL` -> `ıstanbul`), Greek-aware
|
|
224
|
+
normalization (final sigma `ΕΛΛΑΣ` -> `ελλας`, opt-in accent
|
|
225
|
+
stripping) and Greek/Cyrillic script tokenizers that keep
|
|
226
|
+
non-Latin words. Keyboard layouts grow from 5 to 19: Turkish-Q,
|
|
227
|
+
Ukrainian-JCUKEN and Greek-Phonetic national grids mirrored from
|
|
228
|
+
the models repo eval harness (a drift spec asserts the gem grids
|
|
229
|
+
cover the eval grids; `scripts/extract_eval_grids.rb` refreshes the
|
|
230
|
+
committed snapshot), plus one parameterized Latin family
|
|
231
|
+
(`Keyboard::Layouts::Latin` and members) over the qwerty/qwertz
|
|
232
|
+
base grids with real Nordic å/æ/ø and å/ä/ö keys. The pre-existing
|
|
233
|
+
five layouts and ten language modules are unchanged.
|
|
234
|
+
- **Inline ignore directives** (plan 82) - `kotoshu:disable-line`,
|
|
235
|
+
`kotoshu:disable-next-line [WORDS]`, and the nestable
|
|
236
|
+
`kotoshu:disable-file` / `kotoshu:enable-file` block, recognized in
|
|
237
|
+
each format's comment syntax: HTML comments in Markdown, `//` line
|
|
238
|
+
comments in AsciiDoc, bare lines in plain text (trailing directives
|
|
239
|
+
in Markdown/AsciiDoc). One shared scanner in `Documents::Suppressions`
|
|
240
|
+
attaches them to every `Document`, and the `Spellchecker` facade
|
|
241
|
+
filters them out of `check`: suppressed words move to
|
|
242
|
+
`DocumentResult#suppressed_errors` (marked `suppressed: true`,
|
|
243
|
+
`suppressed_by: "inline"`), and `kotoshu check --show-suppressed`
|
|
244
|
+
lists them. Result JSON gains `suppressed` / `suppressed_by` on
|
|
245
|
+
errors and `suppressedCount` / `suppressedErrors` at the top level.
|
|
246
|
+
- **CI baselines** (plan 82) - `kotoshu baseline init FILE ...` writes
|
|
247
|
+
`.kotoshu-baseline.json` (canonical `{version, entries: [{file, line,
|
|
248
|
+
word, count}]}` via lutaml-model; count-based, so it survives
|
|
249
|
+
reformatting); `kotoshu check --baseline FILE` lets covered errors
|
|
250
|
+
pass, fails on new ones, and reports stale entries. SARIF output
|
|
251
|
+
marks baseline-suppressed results with a `suppressions` entry
|
|
252
|
+
(`kind: external`, `justification: "baseline"`), and JSON output
|
|
253
|
+
adds a `baseline` block.
|
|
254
|
+
- **pre-commit hook** (plan 81) - `.pre-commit-hooks.yaml` ships hook
|
|
255
|
+
id `kotoshu` (`language: system`, entry `kotoshu check`, text-format
|
|
256
|
+
file pattern); requires Ruby plus the gem, documented honestly in the
|
|
257
|
+
README.
|
|
258
|
+
|
|
259
|
+
## [0.7.0] — 2026-09-05
|
|
260
|
+
|
|
261
|
+
The universal-kotoshu cut (plan 65's roadmap milestone): tiered
|
|
262
|
+
models, the Rust core, and the native extension.
|
|
263
|
+
|
|
264
|
+
### Added
|
|
265
|
+
|
|
266
|
+
- **Model tiers** — `full` (~120 MB) / `fluency` (~15 MB, the new
|
|
267
|
+
default) / `mini` (~3 MB), resolved through the models repo's
|
|
268
|
+
registry (`kotoshu://models/{lang}/{tier}`) with sha256-verified
|
|
269
|
+
primary→mirror→vocab downloads; `KOTOSHU_MODEL_TIER`, `setup
|
|
270
|
+
LANG --model --tier`, and a tier-less legacy-cache bridge.
|
|
271
|
+
- **Native extension** (`ext/kotoshu_native`, optional Rust
|
|
272
|
+
accelerator): `KOTOSHU_BACKEND=ruby|native|auto` (default ruby;
|
|
273
|
+
pure-Ruby installs unchanged), `Kotoshu::Native.available?`, and
|
|
274
|
+
`rake kotoshu:conformance:compare` — 2630 vectors, zero divergences
|
|
275
|
+
between the Ruby and Rust engines.
|
|
276
|
+
- **Conformance export** — `rake kotoshu:conformance:export` freezes
|
|
277
|
+
engine behavior as golden vectors (2630 lines) consumed by
|
|
278
|
+
kotoshu-rs CI.
|
|
279
|
+
- **Confidence cascade** — `semantic_cascade_threshold`
|
|
280
|
+
(`KOTOSHU_SEMANTIC_CASCADE_THRESHOLD`): skip the ONNX rerank when
|
|
281
|
+
the composite strategies are already confident.
|
|
282
|
+
- Hunspell correctness: CHECKCOMPOUNDPATTERN replacement support,
|
|
283
|
+
dot-split casing, INITCAP ngram-root skip, AF alias flag-format
|
|
284
|
+
parsing, and a long tail of fixture fixes (integrational suite
|
|
285
|
+
176 examples, 0 failures, 1 documented pending).
|
|
286
|
+
|
|
287
|
+
### Fixed
|
|
288
|
+
|
|
289
|
+
- Model downloads cached 134-byte LFS pointer stubs whose own sha256
|
|
290
|
+
verified forever — the model source now fetches from the media host
|
|
291
|
+
and rejects stubs at download and verify.
|
|
292
|
+
- `Embeddings::Vocabulary` crashed on the shipped wrapped vocab
|
|
293
|
+
format; shared normalizer added.
|
|
294
|
+
- `Models::OnnxModel` used a nonexistent ONNX Runtime API and wrong
|
|
295
|
+
IO names — rewritten against `InferenceSession`.
|
|
296
|
+
- CLI JSON output restored to the documented camelCase contract;
|
|
297
|
+
`DictionaryNotFoundError` mapping restored at the cache boundary.
|
|
298
|
+
|
|
299
|
+
## [0.5.0] — 2026-06-30
|
|
300
|
+
|
|
301
|
+
Tier 2.5 release. Adds the personal-dictionary CLI, completes direct
|
|
302
|
+
spec coverage for every previously-untested namespace (~370 new
|
|
303
|
+
examples), eliminates every `respond_to?` / `send`-to-private /
|
|
304
|
+
`instance_variable_*` violation from `lib/`, and fixes a long list of
|
|
305
|
+
production bugs the refactors surfaced.
|
|
306
|
+
|
|
307
|
+
### Added
|
|
308
|
+
|
|
309
|
+
- **Cache eviction** (`Kotoshu::Cache::EvictionPolicy` +
|
|
310
|
+
`BaseCache#evict` + `kotoshu cache evict`). A pure value object
|
|
311
|
+
decides which entries to evict (LRU by `cached_at`, oldest first)
|
|
312
|
+
to fit a configured size cap; `BaseCache#evict` collects on-disk
|
|
313
|
+
entries (one record per discovered `metadata.json`) and executes
|
|
314
|
+
the plan. `--dry-run` returns the plan without touching disk. The
|
|
315
|
+
cap defaults to `max_cache_size` (1 GB, `KOTOSHU_MAX_CACHE_SIZE`)
|
|
316
|
+
and is now wired through `Configuration` instead of a hardcoded
|
|
317
|
+
constant. Corrupt metadata (missing `cached_at`) sorts oldest so it
|
|
318
|
+
is evicted first.
|
|
319
|
+
- **Audit log rotation** (`Kotoshu::Integrity::RotationPolicy`). When the
|
|
320
|
+
current `audit.log` exceeds `audit_max_bytes` (default 10 MB,
|
|
321
|
+
`KOTOSHU_AUDIT_MAX_BYTES`), the log rotates through `audit_rotations`
|
|
322
|
+
historical files (default 5, `KOTOSHU_AUDIT_ROTATIONS`). Total on-disk
|
|
323
|
+
footprint is bounded at `max_bytes * (rotations + 1)`. The rotation
|
|
324
|
+
policy is a pure value object — `AuditLog#record` consults it on every
|
|
325
|
+
write and executes the returned rename plan under an exclusive flock
|
|
326
|
+
on a sibling lockfile (`audit.log.lock`) so concurrent writers cannot
|
|
327
|
+
race the rename chain. `AuditLog#entries` now walks both the current
|
|
328
|
+
log and all rotations, newest-first.
|
|
329
|
+
- **Shell completion** (`kotoshu completions bash|zsh|fish`). Emits a
|
|
330
|
+
shell-specific completion script that completes top-level subcommands
|
|
331
|
+
and dynamically completes language codes for `setup` / `fetch` by
|
|
332
|
+
shelling out to `kotoshu completions languages`. README documents
|
|
333
|
+
install paths for bash, zsh, and fish.
|
|
334
|
+
- **Load-time model cache integrity** (`Cache::ModelCache`). A cached
|
|
335
|
+
ONNX model file is re-verified against the SHA-256 recorded in its
|
|
336
|
+
`metadata.json` every time it is loaded — not just at download time.
|
|
337
|
+
Tampering, truncation, or partial writes are caught before the file
|
|
338
|
+
reaches the ONNX runtime, and the resulting `Kotoshu::IntegrityError`
|
|
339
|
+
includes a `remediation:` hint pointing the user at
|
|
340
|
+
`kotoshu cache download :LANG --model`. Legacy caches whose metadata
|
|
341
|
+
predates the checksum field continue to load (graceful degradation).
|
|
342
|
+
- **`IntegrityError#remediation`** — the error now accepts an optional
|
|
343
|
+
`remediation:` keyword whose text is appended to the message and
|
|
344
|
+
exposed via a reader, so callers can surface the user-facing fix
|
|
345
|
+
(e.g. "Run `kotoshu cache download :en --model` to re-download").
|
|
346
|
+
Existing callers that omit it see no change in the message format.
|
|
347
|
+
- **Personal dictionary CLI** (`kotoshu personal SUBCOMMAND`). Thor
|
|
348
|
+
subcommand wrapping `Kotoshu::PersonalDictionary`: `add`, `remove`,
|
|
349
|
+
`list`, `import` (with `--dry-run`), `path`, and `clear` (with
|
|
350
|
+
`--yes`). Errors raise `Kotoshu::Cli::Errors::UsageError` (exit 2),
|
|
351
|
+
matching the existing CLI error contract.
|
|
352
|
+
- **Direct specs for previously-untested namespaces.** ~370 new
|
|
353
|
+
examples pinning the public contract of `Integrity`, `Embeddings`,
|
|
354
|
+
`Language`, `Dictionary`, and `Cache` namespaces — each spec uses
|
|
355
|
+
real instances and tmpdir/fixture IO, never `double()`.
|
|
356
|
+
- **`Kotoshu::Dictionary::Unified`** is now loadable. The class
|
|
357
|
+
was previously declared as `class Kotoshu::Dictionary`, colliding
|
|
358
|
+
with the existing `module Kotoshu::Dictionary` namespace and
|
|
359
|
+
raising `TypeError` on autoload. Renamed to `Unified` to match the
|
|
360
|
+
autoload entry. Two follow-on bugs (mismatched `Suggester.new`
|
|
361
|
+
kwargs and a custom `build_dic_structure` that produced the wrong
|
|
362
|
+
shape) fixed in the same change.
|
|
363
|
+
|
|
364
|
+
### Changed
|
|
365
|
+
|
|
366
|
+
- **`Embeddings::LruCache#fetch`** now yields the *key* to the provided
|
|
367
|
+
block (matching Ruby's `Hash#fetch` convention) instead of invoking
|
|
368
|
+
the block with no arguments. Callers that compute the missing value
|
|
369
|
+
from the key no longer need a closure over an outer variable.
|
|
370
|
+
- **`Embeddings::SimilarityEngine`** exposes a `pre_normalize?`
|
|
371
|
+
predicate (replacing a dead `attr_reader :pre_normalize` that was
|
|
372
|
+
silently shadowed by a private method of the same name) and now
|
|
373
|
+
raises `ArgumentError` with a clear dimension-mismatch message
|
|
374
|
+
instead of crashing with a `TypeError` from `Array#zip` when given
|
|
375
|
+
vectors of differing lengths. `nil` and empty inputs still return
|
|
376
|
+
`0.0` (early-return path).
|
|
377
|
+
- **Eliminated `respond_to?` across `lib/`.** Replaced every
|
|
378
|
+
occurrence with `is_a?` / `case-when` / `method_defined?` / direct
|
|
379
|
+
calls. Duck-typing via `respond_to?` hid real type errors — the
|
|
380
|
+
production bugs surfaced by this refactor (see Fixed) had been
|
|
381
|
+
silently no-op'ing for as long as the code existed.
|
|
382
|
+
- **Eliminated `send` to private/protected methods** in `lib/`. The
|
|
383
|
+
`cache` CLI subcommand's `cache.send(:read_metadata, ...)` is now
|
|
384
|
+
`cache.read_metadata(...)` after promoting the method to public
|
|
385
|
+
(`public def`). `SemanticStrategy#embedding_for` was dead code that
|
|
386
|
+
called `@search.send(:get_embedding, ...)` on a method that didn't
|
|
387
|
+
exist — removed entirely.
|
|
388
|
+
- **Eliminated `instance_variable_set/get` across `lib/`.**
|
|
389
|
+
`BloomFilter#merge` now reads `other.bits` (public `attr_reader`)
|
|
390
|
+
instead of `other.instance_variable_get(:@bits)`.
|
|
391
|
+
`Dictionary::PlainText` and `Dictionary::CSpell` constructors now
|
|
392
|
+
accept a `words:` keyword instead of `from_words` poking ivars via
|
|
393
|
+
`allocate` + `instance_variable_set`. `Readers::LookupBuilder`
|
|
394
|
+
gained `aff_data:` / `words:` keyword args to replace the same
|
|
395
|
+
pattern in `from_data`.
|
|
396
|
+
- **Configuration dispatch is type-checked, not duck-typed.**
|
|
397
|
+
`Configuration#apply_*` previously called
|
|
398
|
+
`send("#{key}=", value) if respond_to?("#{key}=")`; now uses
|
|
399
|
+
`public_send(setter, value) if self.class.method_defined?(setter)`.
|
|
400
|
+
- **`Dictionary::Base` exposes a default `path` method** returning
|
|
401
|
+
`nil` (overridden by file-backed subclasses). `Spellchecker`
|
|
402
|
+
previously guarded with `dictionary.respond_to?(:path)`; now calls
|
|
403
|
+
unconditionally.
|
|
404
|
+
- **`Language::Registry.info` checks the type bound.** Was
|
|
405
|
+
`klass.respond_to?(:instance)`; now
|
|
406
|
+
`klass.is_a?(Class) && klass <= Kotoshu::Language::Base`.
|
|
407
|
+
|
|
408
|
+
### Fixed
|
|
409
|
+
|
|
410
|
+
- **`AuditLog#each` now iterates newest-first across current and
|
|
411
|
+
rotations.** The docstring promised newest-first but the
|
|
412
|
+
implementation yielded each file in write order (oldest line first)
|
|
413
|
+
then walked current → oldest file, producing a jumbled order. Each
|
|
414
|
+
file's lines are now read into memory and yielded in reverse before
|
|
415
|
+
walking to the next older rotation.
|
|
416
|
+
- **`Embeddings::Protocol` is no longer broken.** The module declared
|
|
417
|
+
`required_methods` and `required_methods(*names)` (a reader + a
|
|
418
|
+
writer with the same name). Ruby's last-def-wins shadowed the
|
|
419
|
+
reader, so every protocol registration silently dropped its
|
|
420
|
+
arguments and `compliance_errors` returned `[]` unconditionally —
|
|
421
|
+
every class appeared to conform to every protocol. Renamed the
|
|
422
|
+
writer to `required` (and `optional_methods` → `optional`); also
|
|
423
|
+
fixed `compliance_errors` to use `klass.method_defined?` instead of
|
|
424
|
+
`klass.respond_to?` (which checks class methods, not instance
|
|
425
|
+
methods).
|
|
426
|
+
- **`Embeddings::OnnxRuntimeModel` defines `loaded?`.** The protocol
|
|
427
|
+
requires `loaded?` but the class declared only `attr_reader :loaded`
|
|
428
|
+
(no question mark). `EmbeddingPipeline#stats` and `#to_s` called
|
|
429
|
+
`@model.loaded?` — would have raised `NoMethodError` on the real
|
|
430
|
+
model.
|
|
431
|
+
- **`Dictionary::Base` aliases dispatch dynamically.** Ruby's `alias`
|
|
432
|
+
keyword binds to the parent method body at definition time,
|
|
433
|
+
bypassing subclass overrides. `has_word?`/`include?`/`contains?`/
|
|
434
|
+
`lookup?`/`<<`/`all_words` on a concrete subclass (e.g.,
|
|
435
|
+
`UnixWords`) would invoke the abstract `Base#lookup` /
|
|
436
|
+
`Base#add_word` and raise `NotImplementedError`. Replaced with
|
|
437
|
+
one-line method definitions that call the abstract method (which
|
|
438
|
+
then dispatches dynamically to the subclass override).
|
|
439
|
+
- **`Dictionary::CSpell#add_word` no longer crashes.** Tried to
|
|
440
|
+
`@trie.insert` on a frozen `Trie` (frozen by `Builder#build`),
|
|
441
|
+
raising `FrozenError`. Aligned with `remove_word`'s existing
|
|
442
|
+
"always returns false (immutable after load)" contract.
|
|
443
|
+
- **`Cache::FrequencyCache#get_frequency` no longer raises
|
|
444
|
+
`ArgumentError`.** Called `get(language_code, force_download)`
|
|
445
|
+
positionally, but `BaseCache#get` declares `force_download:` as
|
|
446
|
+
keyword. Fixed to use the keyword form.
|
|
447
|
+
- **`Cache::FrequencyCache#cached_resources` no longer reports the
|
|
448
|
+
`tmp/` scratch slot.** `BaseCache#initialize` creates
|
|
449
|
+
`<cache_path>/tmp/` as a working directory; the directory-walking
|
|
450
|
+
implementation in `FrequencyCache` listed it as a cached language.
|
|
451
|
+
Added an explicit `basename != "tmp"` exclusion.
|
|
452
|
+
- **`Cache::FrequencyCache#install_local` exists.** `ResourceManager`
|
|
453
|
+
called `freq_cache.install_local(...)` behind a `respond_to?` guard
|
|
454
|
+
that always returned `false`, so local frequency installs were
|
|
455
|
+
silently skipped — `setup(from:)` reported `frequency_status: :local`
|
|
456
|
+
even though no install happened. Added the method, mirroring
|
|
457
|
+
`LanguageCache#install_local` but taking a single `path:` kwarg.
|
|
458
|
+
- **`PhonetTable::Rule#match?` is no longer dead code.**
|
|
459
|
+
`Algorithms::PhonetSuggest` had its own `match_rule` that
|
|
460
|
+
duplicated the rule-matching logic, ignoring `Rule#match?`
|
|
461
|
+
entirely. Consolidated into `Rule#match_length` (returning
|
|
462
|
+
`Integer|nil` — the metaphone walker needs the length to advance
|
|
463
|
+
its position) with `Rule#match?` as a boolean wrapper;
|
|
464
|
+
`PhonetSuggest#match_rule` now delegates.
|
|
465
|
+
- **`DoubleNegativeMatcher` recognizes the "not only...but also"
|
|
466
|
+
correlative.** The previous check fired only on the narrow shape
|
|
467
|
+
`not [current_negative] only`, missing the actual idiom (where
|
|
468
|
+
"not" and "only" are adjacent). Replaced with a region-based
|
|
469
|
+
matcher that finds every exception phrase in the token stream and
|
|
470
|
+
suppresses any negative whose index falls inside a region. Supports
|
|
471
|
+
both `"prefix...suffix"` (open-ended) and `"literal n-gram"` (no
|
|
472
|
+
ellipsis) phrase shapes, driven by the rule's `exceptions.phrases`
|
|
473
|
+
config.
|
|
474
|
+
- **`SemanticAnalyzer` no longer returns `[]` for OOV words.**
|
|
475
|
+
`suggest_corrections` called `@model.nearest_neighbors(word)`,
|
|
476
|
+
which returns `[]` for OOV inputs — but OOV is precisely what the
|
|
477
|
+
analyzer exists to handle. Added an edit-distance fallback: when
|
|
478
|
+
the model returns `[]`, walk the vocabulary and return words within
|
|
479
|
+
Levenshtein distance 2, scored by `1/(1+distance)`. Drive-by fix:
|
|
480
|
+
`Models::Suggestion.new` was called with `word:` as keyword but
|
|
481
|
+
the constructor takes `word` positional — exposed once the OOV
|
|
482
|
+
fallback actually reached the `Suggestion.new` call.
|
|
483
|
+
|
|
484
|
+
### Removed
|
|
485
|
+
|
|
486
|
+
- **Orphan files emptied to no-op stubs.**
|
|
487
|
+
`lib/kotoshu/embeddings/protocols.rb` (plural) had
|
|
488
|
+
`require_relative 'protocols/embedding_model'` pointing at
|
|
489
|
+
nonexistent files (loading it raised `LoadError`).
|
|
490
|
+
`lib/kotoshu/readers/readers.rb` (nested) duplicated the autoload
|
|
491
|
+
setup in `lib/kotoshu/readers.rb` with `require_relative` calls
|
|
492
|
+
(violating the project rule against `require_relative` in lib/).
|
|
493
|
+
Both reduced to documentation-only stubs; no callers were affected.
|
|
494
|
+
The real protocol definitions live in
|
|
495
|
+
`lib/kotoshu/embeddings/protocol.rb`; the real reader autoloads
|
|
496
|
+
live in `lib/kotoshu/readers.rb`.
|
|
497
|
+
|
|
498
|
+
## [0.4.0] — 2026-06-29
|
|
499
|
+
|
|
500
|
+
Tier 2 release. Completes the Hunspell correctness work (morphological
|
|
501
|
+
rules, REP/phonet/ngram ranking, compound handling), migrates the entire
|
|
502
|
+
library from `require_relative` to Ruby `autoload`, and tightens the
|
|
503
|
+
cache interface alignment that 0.3.1 sketched.
|
|
504
|
+
|
|
505
|
+
### Added
|
|
506
|
+
|
|
507
|
+
- **Hunspell morphological correctness** (T2 Phases 2A–2D):
|
|
508
|
+
- `FLAG long|num|utf-8` parsing and `AF` alias resolution.
|
|
509
|
+
- `COMPOUNDRULE`, `CHECKCOMPOUNDPATTERN` (with empty-flag fix),
|
|
510
|
+
`CIRCUMFIX` handling.
|
|
511
|
+
- `ICONV`/`OCONV` `ConvTable` with stable sort — preserves Nepali
|
|
512
|
+
`ZWNJ`/`ZWJ` ordering.
|
|
513
|
+
- `KEEPCASE` suggestion flag with `CHECKSHARPS` exception, plus the
|
|
514
|
+
German eszet rule (`ss` ↔ `SS`).
|
|
515
|
+
- `REP` table, phonet ordering, ngram ranking wired end-to-end.
|
|
516
|
+
- Edge cases: `IJ` digraph, `i58202` case preservation,
|
|
517
|
+
`opentaal_forbiddenword2`, `breakdefault` `BREAK` pattern.
|
|
518
|
+
- **Hunspell Suggester** is now wired into the Hunspell dictionary; the
|
|
519
|
+
default affix directives are honoured when generating suggestions.
|
|
520
|
+
- **`BaseCache#max_cache_size`** attribute (1 GB default) — the
|
|
521
|
+
forward-looking hook for cache eviction work (see
|
|
522
|
+
`TODO.impl/34-cache-eviction.md`).
|
|
523
|
+
- **`LanguageCache#language_path(lang, type)`** public helper; resource
|
|
524
|
+
and metadata path resolution is now DRY'd through it.
|
|
525
|
+
- **`Kotoshu::Language` suika soft-load module** with specs — keeps
|
|
526
|
+
tokenization lazy and optional.
|
|
527
|
+
- **Multi-language EditDistanceStrategy spec coverage** — pins
|
|
528
|
+
per-language keyboard selection (QWERTZ/AZERTY/QWERTY/JCUKEN) and
|
|
529
|
+
verifies German and French typo correction end-to-end. The previous
|
|
530
|
+
`skip 'Multi-language support not yet implemented'` hook is removed;
|
|
531
|
+
the strategy has been language-aware via `Keyboard::Registry` +
|
|
532
|
+
`Cache::FrequencyCache` for several releases.
|
|
533
|
+
|
|
534
|
+
### Changed
|
|
535
|
+
|
|
536
|
+
- **Library-wide `autoload` migration.** Every `require_relative` (and
|
|
537
|
+
in-library `require`) under `lib/kotoshu/` is replaced with a Ruby
|
|
538
|
+
`autoload` declared in the immediate parent namespace's file. This
|
|
539
|
+
fixes load-order races, makes the eager/opt-in split explicit, and
|
|
540
|
+
matches the contributor guidance in `CLAUDE.md`. Behaviour is
|
|
541
|
+
unchanged for callers who only `require "kotoshu"`, but extenders who
|
|
542
|
+
reached past the public facade may need to update their load entry
|
|
543
|
+
points.
|
|
544
|
+
- **`Embeddings` namespace consolidation.** All embedding classes are
|
|
545
|
+
nested under `Kotoshu::Embeddings` (was scattered with several
|
|
546
|
+
`require_relative` shims).
|
|
547
|
+
- **Grammar rules bundled in the gem.** English `rules.yaml` ships
|
|
548
|
+
inside the gem package so a `gem install kotoshu` is self-contained
|
|
549
|
+
for grammar checking without a separate download.
|
|
550
|
+
- **Plain-text edit distance is now case-insensitive**, matching the
|
|
551
|
+
Hunspell backend's behaviour for `PlainText` dictionaries.
|
|
552
|
+
|
|
553
|
+
### Fixed
|
|
554
|
+
|
|
555
|
+
- **`Cli::CacheCommand` wired to the real `LanguageCache` API.** The
|
|
556
|
+
previous implementation called nonexistent methods (`cache_status`,
|
|
557
|
+
`get_frequency_data`, `get_language_info`, `purge_all`); `kotoshu cache`
|
|
558
|
+
now correctly routes through `available?`, `cached_resources`,
|
|
559
|
+
`clear_all`, `clean`, `stats`, `get_spelling`, `get_grammar`, and the
|
|
560
|
+
standalone `Cache::FrequencyCache` for frequency downloads.
|
|
561
|
+
- **`Keyboard::Registry` lazy reload.** `clear` now marks
|
|
562
|
+
`@languages_loaded = true` to prevent an infinite lazy-reload loop
|
|
563
|
+
when the registry is reset at runtime.
|
|
564
|
+
- **German common-words YAML.** `data/common_words/de.yml` line 466 had
|
|
565
|
+
`-ecke` (missing space after dash), a syntax error that prevented the
|
|
566
|
+
file from loading and silently broke German strategy instantiation.
|
|
567
|
+
- **`Readers::AffReader` Hunspell edge cases** — encoding fallback for
|
|
568
|
+
Latin-1 fixtures, `REP` table splitting, `MultiWord` dash variants,
|
|
569
|
+
default affix directives honoured when the `.aff` file omits them.
|
|
570
|
+
- **RuboCop 3.x / plugins migration.** `rubocop-rspec` is bumped to
|
|
571
|
+
3.x and the config uses the new plugins syntax; safe auto-corrections
|
|
572
|
+
applied across `lib/` and `spec/` (`Layout/ExtraSpacing`,
|
|
573
|
+
`Layout/EmptyLine`).
|
|
574
|
+
- **Cross-platform CI.** `bundle` is invoked via `Gem.ruby -S` for
|
|
575
|
+
Windows compatibility; Ruby 4.0 compatibility fixes applied.
|
|
576
|
+
|
|
577
|
+
### Internal
|
|
578
|
+
|
|
579
|
+
- `TODO.impl/` planning documents (00–40) are the tiered execution
|
|
580
|
+
plan. Tier 2 is complete; Tier 3 (`audit-log-rotation`,
|
|
581
|
+
`cache-eviction`, `shell-completion`) and content-repo tasks
|
|
582
|
+
(`onnx-vocab-json-generation`) remain deferred past 0.4.0.
|
|
583
|
+
|
|
584
|
+
## [0.3.1] — 2026-06-28
|
|
585
|
+
|
|
586
|
+
Bug-fix release. Closes the regressions introduced by the 0.3.0
|
|
587
|
+
lutaml-model migration and tightens up spec gating so the default suite
|
|
588
|
+
is green on the non-network, non-ONNX subset.
|
|
589
|
+
|
|
590
|
+
### Fixed
|
|
591
|
+
|
|
592
|
+
- `WordResult#suggestions` is `Array<Suggestion>` after the lutaml
|
|
593
|
+
migration; the integration spec now reads it as
|
|
594
|
+
`suggestions.map(&:word)` instead of the removed `SuggestionSet#words`.
|
|
595
|
+
- `WordResult#to_h` was removed by lutaml; the integration spec now uses
|
|
596
|
+
the framework-supplied `to_hash`.
|
|
597
|
+
- `Readers::AffReader#detect_encoding` now handles a nil path so the
|
|
598
|
+
reader can be constructed from an in-memory `StringReader` (used by
|
|
599
|
+
the unit tests and by programmatic callers that build affix data
|
|
600
|
+
without a file on disk).
|
|
601
|
+
- `Embeddings::SimilaritySearch` is now eagerly loaded alongside the
|
|
602
|
+
rest of the embeddings namespace; previously the constant was
|
|
603
|
+
uninitialized when `SemanticStrategy` tried to use it.
|
|
604
|
+
- `Embeddings::Vocabulary.from_cache` is implemented (was missing). It
|
|
605
|
+
resolves the `vocab.json` sibling of the cached ONNX model and
|
|
606
|
+
returns nil when the model or vocab is unavailable, so callers can
|
|
607
|
+
degrade gracefully.
|
|
608
|
+
- `Embeddings::OnnxRuntimeModel.from_cache` now accepts the `cache:`
|
|
609
|
+
keyword (matching `Vocabulary.from_cache` and
|
|
610
|
+
`SimilaritySearch.from_cache`); the previous positional signature
|
|
611
|
+
silently swallowed the cache as a Hash.
|
|
612
|
+
- `Suggestions::Strategies::SemanticStrategy` now surfaces its named
|
|
613
|
+
constructor kwargs (`min_semantic_similarity`,
|
|
614
|
+
`semantic_boost_weight`, `max_context_window`) through `get_config`,
|
|
615
|
+
so the strategy's configurable knobs are queryable the same way as
|
|
616
|
+
the base strategy's `**config` bag.
|
|
617
|
+
|
|
618
|
+
### Changed
|
|
619
|
+
|
|
620
|
+
- `spec/spec_helper.rb` excludes `:onnx`-tagged examples unless
|
|
621
|
+
`ONNX_TESTS=1` is set, mirroring the existing `:network` and `:slow`
|
|
622
|
+
opt-in pattern. `spec/kotoshu/suggestions/strategies/semantic_strategy_spec.rb`
|
|
623
|
+
is the sole consumer; its `:integration` tags are now `:onnx`.
|
|
624
|
+
- The SymSpell benchmark spec and the edit-distance / symspell
|
|
625
|
+
performance describe blocks are tagged `:slow`, removing
|
|
626
|
+
environment-dependent timing flakes from the default run.
|
|
627
|
+
- Arabic language detection is marked `pending` with a pointer to
|
|
628
|
+
`TODO.impl/30-language-auto-detection.md` (FastText LID missing the
|
|
629
|
+
Arabic vector).
|
|
630
|
+
- `edit_distance_strategy_spec` now uses the `from_source?` predicate
|
|
631
|
+
instead of asserting `source == :edit_distance`, matching the
|
|
632
|
+
lutaml-era contract that `Suggestion#source` is a string.
|
|
633
|
+
|
|
634
|
+
### Internal
|
|
635
|
+
|
|
636
|
+
- Added `TODO.impl/36-cleanup-regressions-0.3.1.md` (this release),
|
|
637
|
+
`TODO.impl/37-hunspell-correctness-tier2.md`,
|
|
638
|
+
`TODO.impl/38-onnx-semantic-gating.md`, and
|
|
639
|
+
`TODO.impl/39-tier3-and-beyond.md` as the tiered execution plan.
|
|
640
|
+
|
|
641
|
+
## [0.3.0] — 2026-06-27
|
|
642
|
+
|
|
643
|
+
The two-stage release. Resources are now downloaded explicitly via
|
|
644
|
+
`Kotoshu.setup(:en)`; the hot path (`correct?`, `suggest`, `check`) reads
|
|
645
|
+
only from cache and raises a typed error when a language is missing instead
|
|
646
|
+
of triggering a network download. The CLI adds `setup`, `status`, language
|
|
647
|
+
auto-detection, SARIF/JSON output, and an interactive auto-setup prompt.
|
|
648
|
+
`onnxruntime` is now a soft dependency, so `gem install kotoshu` succeeds on
|
|
649
|
+
hosts that can't load native ONNX runtime.
|
|
650
|
+
|
|
651
|
+
### Added
|
|
652
|
+
|
|
653
|
+
- **Two-stage resource model** (`Kotoshu::ResourceManager`):
|
|
654
|
+
`Kotoshu.setup(:en, want: %i[spelling frequency model])` writes into the
|
|
655
|
+
cache; `Kotoshu::ResourceManager.resolve(language:, want:)` is instant and
|
|
656
|
+
cache-only, raising `ResourceNotSetupError` on miss. `Kotoshu.setup?` is
|
|
657
|
+
the predicate for "is this language already cached?". The library never
|
|
658
|
+
triggers a surprise download; the CLI prompts the user via `AutoSetup`.
|
|
659
|
+
- **`SourceRegistry`** — single source of truth for the three content repos'
|
|
660
|
+
URLs and per-repo pins. `kotoshu/dictionaries` is pinned to the `v1`
|
|
661
|
+
branch; `frequency-list-kelly` and `models-fasttext-onnx` are on `main`.
|
|
662
|
+
Override at runtime via `KOTOSHU_REPOS_BASE_URL`, `KOTOSHU_DICTIONARIES_PIN`,
|
|
663
|
+
`KOTOSHU_FREQUENCY_PIN`, `KOTOSHU_MODELS_PIN`.
|
|
664
|
+
- **XDG Base Directory layout** (`Kotoshu::Paths`): dictionaries, frequency
|
|
665
|
+
lists, ONNX models under `$XDG_CACHE_HOME/kotoshu/`; personal dictionary
|
|
666
|
+
and `kotoshu.cfg` under `$XDG_CONFIG_HOME/kotoshu/`; audit log under
|
|
667
|
+
`$XDG_DATA_HOME/kotoshu/audit.log`. Override per-axis with
|
|
668
|
+
`KOTOSHU_CACHE_PATH`, `KOTOSHU_CONFIG_PATH`, `KOTOSHU_DATA_PATH`.
|
|
669
|
+
- **Integrity verification** — `Kotoshu::Integrity::Manifest` (SHA-256) is
|
|
670
|
+
fetched per content repo and matched against every download. Mismatches
|
|
671
|
+
raise `Kotoshu::IntegrityError`. Outcomes (verified / unverified / mismatch)
|
|
672
|
+
are written to the audit log. Missing manifests degrade gracefully.
|
|
673
|
+
- **CLI `setup` command** — `kotoshu setup LANG [--force] [--no-frequency]
|
|
674
|
+
[--no-model]` writes the requested resources into the cache with progress
|
|
675
|
+
reporting.
|
|
676
|
+
- **CLI `status` command** — `kotoshu status [--json]` summarises installed
|
|
677
|
+
resources, sizes, mtimes, and ONNX runtime availability.
|
|
678
|
+
- **CLI `check --language auto`** — auto-detects document language via
|
|
679
|
+
FastText LID; falls back to the configured default language when detection
|
|
680
|
+
is unavailable or the detected language is not set up.
|
|
681
|
+
- **CLI `check --format json|sarif`** — machine-readable output. SARIF
|
|
682
|
+
follows v2.1.0 with `kotoshu/spelling` rule id, JSON exposes
|
|
683
|
+
`success`/`wordCount`/`errorCount`/`uniqueErrorCount`/`errors`/`source`.
|
|
684
|
+
- **CLI auto-setup prompt** — when the hot path raises
|
|
685
|
+
`ResourceNotSetupError` in an interactive session, the user is prompted to
|
|
686
|
+
run setup now and the original command is retried on success. Non-TTY,
|
|
687
|
+
offline (`--offline`), and `--no-prompt` invocations skip the prompt and
|
|
688
|
+
surface the error as before.
|
|
689
|
+
- **Download progress reporting** (`Kotoshu::Cli::ProgressReporter`) — TTY
|
|
690
|
+
mode renders a determinate/indeterminate progress bar; non-TTY mode prints
|
|
691
|
+
a periodic line every 10 MiB. `Kotoshu.configuration.download_reporter=`
|
|
692
|
+
exposes the reporter for programmatic use.
|
|
693
|
+
- **End-to-end smoke spec** (`spec/integration/end_to_end_spec.rb`) covers
|
|
694
|
+
install → setup → `correct?` → `suggest.to_words` → `check` →
|
|
695
|
+
`setup?` predicate → `ResourceNotSetupError` → idempotent re-setup.
|
|
696
|
+
Tagged `:network`, opted into via `NETWORK_TESTS=1`.
|
|
697
|
+
- **CLI format spec** (`spec/kotoshu/cli/check_format_spec.rb`) shells out to
|
|
698
|
+
the real `kotoshu` CLI and asserts JSON / SARIF structure and exit codes.
|
|
699
|
+
|
|
700
|
+
### Changed
|
|
701
|
+
|
|
702
|
+
- **`onnxruntime` is a soft dependency.** Removed from `kotoshu.gemspec`.
|
|
703
|
+
`Kotoshu::Models::OnnxModel` soft-requires it at load time and exposes
|
|
704
|
+
`ONNX_LOADED`. When false, semantic methods raise
|
|
705
|
+
`Kotoshu::Models::OnnxModel::OnnxUnavailable` with a caller-friendly
|
|
706
|
+
message. `KOTOSHU_NO_ONNX=1` forces semantic off even when the gem is
|
|
707
|
+
present. The traditional spell-checking path never touches `onnxruntime`.
|
|
708
|
+
- **Loading strategy** — `lib/kotoshu.rb` eagerly loads only the facade
|
|
709
|
+
dependencies; heavier or optional pieces (ONNX models, interactive CLI,
|
|
710
|
+
caches, language detection) are wired through Ruby `autoload` registered
|
|
711
|
+
in their immediate parent namespace.
|
|
712
|
+
- **Public API** — `suggest` returns a `SuggestionSet`; call `.to_words` for
|
|
713
|
+
an `Array<String>`. `Kotoshu.check` returns a `DocumentResult`; iterate
|
|
714
|
+
`errors` for `WordResult` instances with `word`, `position`, `line`,
|
|
715
|
+
`column`, `suggestions`.
|
|
716
|
+
- **README quickstart** — reflects the two-stage API; documents XDG paths;
|
|
717
|
+
marks `onnxruntime` as optional.
|
|
718
|
+
|
|
719
|
+
### Fixed
|
|
720
|
+
|
|
721
|
+
- `gem install kotoshu` no longer requires `onnxruntime` or its native
|
|
722
|
+
toolchain.
|
|
723
|
+
- Resource resolution no longer triggers downloads from inside the hot path.
|
|
724
|
+
- Per-repo pins are honoured — the `v1` branch of `kotoshu/dictionaries` is
|
|
725
|
+
fetched instead of `main`.
|
|
726
|
+
|
|
727
|
+
### Known limitations (carried from 0.1.0, scope reduced)
|
|
728
|
+
|
|
729
|
+
- **Hunspell correctness**: compound rules, circumfix, ICONV/OCONV, German ß,
|
|
730
|
+
Turkish dotless-i remain partial. See `TODO.impl/01-hunspell-correctness.md`.
|
|
731
|
+
- **CJK and RTL**: tokenizer, normalizer, and keyboard layouts exist for
|
|
732
|
+
supported languages; full CJK/RTL support deferred past 0.3.
|
|
733
|
+
See `TODO.impl/06-cjk-support.md` and `TODO.impl/07-rtl-support.md`.
|
|
734
|
+
- **Grammar rules**: the rule engine exists; no rule packs are shipped.
|
|
735
|
+
See `TODO.impl/08-grammar-engine.md`.
|
|
736
|
+
- **Audit log rotation, cache eviction policy, and shell completion** are
|
|
737
|
+
deferred past 0.3 (T3 TODOs).
|
|
738
|
+
|
|
739
|
+
### Internal
|
|
740
|
+
|
|
741
|
+
- 9 logical commits on `release-0.3` cover the T1 (architectural) and T2
|
|
742
|
+
(user-facing) work for this release.
|
|
743
|
+
- `SourceRegistry`, `Paths`, `ResourceManager`, `ResourceBundle`,
|
|
744
|
+
`SetupResult`, `Integrity::Manifest`, `Integrity::AuditLog`,
|
|
745
|
+
`Cli::AutoSetup`, `Cli::StatusReport`, `Cli::LanguageResolver`,
|
|
746
|
+
`Cli::ProgressReporter` are new model-driven types.
|
|
747
|
+
- 73 new specs added (source_registry, end_to_end, check_format,
|
|
748
|
+
progress_reporter, language_resolver, status_report, auto_setup).
|
|
749
|
+
|
|
750
|
+
### Contributors
|
|
751
|
+
|
|
752
|
+
- Ribose Inc.
|
|
753
|
+
|
|
754
|
+
## [0.1.0] — 2026-06-25
|
|
755
|
+
|
|
756
|
+
First public release. Kotoshu is a pure-Ruby spellchecker that combines a
|
|
757
|
+
Ruby port of the Hunspell algorithm with optional FastText ONNX embeddings
|
|
758
|
+
for semantic reranking. This release establishes the public Ruby API, the
|
|
759
|
+
basic CLI, and the cache layer.
|
|
760
|
+
|
|
761
|
+
### Working
|
|
762
|
+
|
|
763
|
+
- **Ruby API**: `Kotoshu.correct?`, `Kotoshu.suggest`, `Kotoshu.check`,
|
|
764
|
+
`Kotoshu.check_file`, `Kotoshu.detect_language`
|
|
765
|
+
- **CLI**: `kotoshu check TARGET`, `kotoshu dict SUBCOMMAND`, `kotoshu cache
|
|
766
|
+
SUBCOMMAND`, `kotoshu version`
|
|
767
|
+
- **Dictionary backends**: Hunspell (`.aff`/`.dic`), CSpell, UnixWords
|
|
768
|
+
(`/usr/share/dict/words`), PlainText, Custom
|
|
769
|
+
- **Suggestion strategies**: edit distance, phonetic (Phonet), keyboard
|
|
770
|
+
proximity, n-gram, symspell, composite pipeline
|
|
771
|
+
- **Configuration**: `Kotoshu.configure`, CLI > ENV (`KOTOSHU_*`) >
|
|
772
|
+
programmatic > defaults via `Configuration::Resolver`
|
|
773
|
+
- **Cache layer**: `LanguageCache`, `FrequencyCache`, `ModelCache` with TTLs
|
|
774
|
+
and download from `kotoshu/dictionaries`, `kotoshu/frequency-list-kelly`,
|
|
775
|
+
`kotoshu/models-fasttext-onnx`
|
|
776
|
+
- **Language detection**: FastText LID, 127 languages
|
|
777
|
+
- **Documents**: Plain text, Markdown (Kramdown), AsciiDoc (Asciidoctor)
|
|
778
|
+
- **Test suite**: 803 of 866 examples passing (92.7%), 6 pending
|
|
779
|
+
|
|
780
|
+
### Known limitations (not blocking 0.1)
|
|
781
|
+
|
|
782
|
+
- **Hunspell correctness**: compound rules, circumfix, ICONV/OCONV, German
|
|
783
|
+
ß, Turkish dotless-i are not fully implemented. Single-word lookup and
|
|
784
|
+
basic affixes work. See `TODO.impl/01-hunspell-correctness.md`.
|
|
785
|
+
- **CLI surface**: `--interactive`, `--format sarif|json|yaml|csv`,
|
|
786
|
+
`--model fasttext|hybrid`, `--language auto` exist in
|
|
787
|
+
`lib/kotoshu/commands/check_command.rb` but are not wired through
|
|
788
|
+
`exe/kotoshu`. See `TODO.impl/02-cli-unification.md`.
|
|
789
|
+
- **Semantic path**: gated behind `ENV['KOTOSHU_REQUIRE_ONNX']` because
|
|
790
|
+
`onnxruntime` loads eagerly otherwise. Hybrid mode is not the default.
|
|
791
|
+
See `TODO.impl/05-semantic-path.md`.
|
|
792
|
+
- **Dynamic resolution**: the three caches exist independently; there is
|
|
793
|
+
no unified `ResourceManager` that takes arbitrary text and yields the
|
|
794
|
+
full resource bundle. See `TODO.impl/03-dynamic-download.md`.
|
|
795
|
+
- **Languages**: code is wired for English by default. The
|
|
796
|
+
`dictionaries` repo has 98 language directories but the gem's
|
|
797
|
+
`lib/kotoshu/languages/` has only 7 modules (de, en, es, fr, ja, pt,
|
|
798
|
+
ru). See `TODO.impl/04-language-modules.md`.
|
|
799
|
+
- **CJK, RTL**: not implemented. See `TODO.impl/06-cjk-support.md`
|
|
800
|
+
and `TODO.impl/07-rtl-support.md`.
|
|
801
|
+
- **Grammar rules**: the rule engine exists; no rule packs are shipped.
|
|
802
|
+
See `TODO.impl/08-grammar-engine.md`.
|
|
803
|
+
- **Integrity verification**: downloaded resources are not currently
|
|
804
|
+
checksummed. See `TODO.impl/09-integrity-security.md`.
|
|
805
|
+
|
|
806
|
+
### Internal
|
|
807
|
+
|
|
808
|
+
- 12 plans under `TODO.impl/` define the path to 1.0
|
|
809
|
+
- Architecture documentation consolidated under `docs/`
|
|
810
|
+
|
|
811
|
+
### Contributors
|
|
812
|
+
|
|
813
|
+
- Ribose Inc.
|