kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
# 36 — Cleanup Regressions + 0.3.1 Bug-Fix Release
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Ship `kotoshu-0.3.1` — a focused bug-fix release that closes the
|
|
6
|
+
regressions introduced by the 0.3.0 lutaml-model migration and a small
|
|
7
|
+
set of pre-existing spec failures. This is the **only** Tier-1 work;
|
|
8
|
+
its acceptance criterion is a clean spec run on the non-network,
|
|
9
|
+
non-ONNX subset.
|
|
10
|
+
|
|
11
|
+
## Scope (locked)
|
|
12
|
+
|
|
13
|
+
| # | Failure cluster | Root cause | Fix shape |
|
|
14
|
+
|---|---|---|---|
|
|
15
|
+
| 1 | `spec/integration/walking_skeleton_spec.rb:22` — `result.suggestions.words` | Lutaml migration changed `WordResult#suggestions` from `SuggestionSet` to `Array<Suggestion>`. | Update spec to `result.suggestions.map(&:word)`. |
|
|
16
|
+
| 2 | `spec/integration/walking_skeleton_spec.rb:265` — `result.to_h` | Lutaml-model replaced hand-rolled `to_h`. The instance method is gone; framework supplies `to_hash`. | Update spec to call `result.to_hash` (lutaml-supplied). |
|
|
17
|
+
| 3 | `spec/kotoshu/suggestions/strategies/semantic_strategy_spec.rb` (22 failures) | Specs assume `onnxruntime` is installed and a model is loaded. Without ONNX, every example errors. | Tag the spec file `:onnx` and exclude by default; document the opt-in env (`ONNX_TESTS=1`) in CLAUDE.md. Mirror the `:network` opt-in pattern. |
|
|
18
|
+
| 4 | `spec/unit/hunspell/read_aff_spec.rb` (7 failures) | Real parser bugs in REP, MAP, PFX directive parsing and UTF-8 / numeric / long flag formats. | Fix each parser branch with a targeted spec; do **not** rewrite the whole reader. |
|
|
19
|
+
| 5 | `spec/kotoshu/language/detector_spec.rb:56` — Arabic detection | Pre-existing FastText LID gap. | Mark `pending` with a TODO referencing the upstream model; do not block the release on it. |
|
|
20
|
+
| 6 | `spec/benchmark/symspell_benchmark_spec.rb` (1 failure) | Likely environment-dependent timing. | Audit; if real bug, fix; if env noise, mark `:slow`. |
|
|
21
|
+
|
|
22
|
+
**Out of scope:** the 90+ `spec/integrational/{lookup,suggest}_spec.rb`
|
|
23
|
+
Hunspell-fixture failures. Those are T2 (`TODO.impl/37-hunspell-correctness-tier2.md`).
|
|
24
|
+
|
|
25
|
+
## Phase A — Lutaml regression cleanups
|
|
26
|
+
|
|
27
|
+
### A1. walking_skeleton_spec
|
|
28
|
+
|
|
29
|
+
- Line 28: `expect(result.suggestions.words).to include("hello")`
|
|
30
|
+
→ `expect(result.suggestions.map(&:word)).to include("hello")`
|
|
31
|
+
- Line 268: `hash = result.to_h`
|
|
32
|
+
→ `hash = result.to_hash` (lutaml-model supplies this)
|
|
33
|
+
|
|
34
|
+
### A2. semantic_strategy_spec gating
|
|
35
|
+
|
|
36
|
+
- Add `describe ..., :onnx do` or `RSpec.describe ... do; before { skip "set ONNX_TESTS=1" unless ENV["ONNX_TESTS"] } end`.
|
|
37
|
+
- Update `spec/spec_helper.rb` to exclude `onnx: true` unless `ONNX_TESTS=1`.
|
|
38
|
+
- Update `CLAUDE.md` "Notes" section to mention the opt-in.
|
|
39
|
+
|
|
40
|
+
### A3. Audit other lutaml fallout
|
|
41
|
+
|
|
42
|
+
- `grep -rn "\.suggestions\.words\|\.suggestions\.to_a\|\.to_h\b" spec/`
|
|
43
|
+
(only the two walking_skeleton hits — confirmed.)
|
|
44
|
+
- `grep -rn "\.as_json\b" spec/` — already replaced in cli.rb; verify
|
|
45
|
+
no spec calls the removed instance method.
|
|
46
|
+
|
|
47
|
+
**Acceptance:** `bundle exec rspec --tag ~network --tag ~onnx` shows
|
|
48
|
+
**0 failures** attributable to lutaml migration.
|
|
49
|
+
|
|
50
|
+
## Phase B — Hunspell .aff parser fixes
|
|
51
|
+
|
|
52
|
+
For each failing directive, add a focused spec covering the exact
|
|
53
|
+
shape from the failing fixture, then fix the parser branch.
|
|
54
|
+
|
|
55
|
+
### B1. REP directive (read_aff_spec:12)
|
|
56
|
+
|
|
57
|
+
Repairs — substitution pairs used by the suggester. Currently parsed
|
|
58
|
+
incorrectly.
|
|
59
|
+
|
|
60
|
+
### B2. MAP directive (read_aff_spec:31)
|
|
61
|
+
|
|
62
|
+
Character equivalence classes (e.g. `MAP uü`, `MAP aàáâã`). Sets up
|
|
63
|
+
related-character substitution in suggestions.
|
|
64
|
+
|
|
65
|
+
### B3. PFX directive (read_aff_spec:47)
|
|
66
|
+
|
|
67
|
+
Prefix rules. Currently fails to parse multi-rule PFX blocks. Likely
|
|
68
|
+
the same bug as SFX since they share the parser.
|
|
69
|
+
|
|
70
|
+
### B4. Long flag format (read_aff_spec:62)
|
|
71
|
+
|
|
72
|
+
`FLAG long` mode — two-character flags.
|
|
73
|
+
|
|
74
|
+
### B5. UTF-8 flag format (read_aff_spec:106)
|
|
75
|
+
|
|
76
|
+
`FLAG UTF-8` mode — flags are single Unicode codepoints.
|
|
77
|
+
|
|
78
|
+
### B6. Numeric flag format (read_aff_spec:125)
|
|
79
|
+
|
|
80
|
+
`FLAG num` mode — integer flags.
|
|
81
|
+
|
|
82
|
+
### B7. AF directive / flag aliases (read_aff_spec:125)
|
|
83
|
+
|
|
84
|
+
`AF` directive defines aliases used to keep `.dic` files compact.
|
|
85
|
+
|
|
86
|
+
**Acceptance:** all 7 `read_aff_spec` examples green; no regression in
|
|
87
|
+
existing `algorithms/` specs.
|
|
88
|
+
|
|
89
|
+
## Phase C — Pre-existing pending/skips
|
|
90
|
+
|
|
91
|
+
- Arabic detection: `pending "FastText LID missing Arabic vector — see TODO.impl/30-language-auto-detection.md"`.
|
|
92
|
+
- Symspell benchmark: audit; if real, fix; if env noise, `:slow`.
|
|
93
|
+
|
|
94
|
+
## Phase D — Cut 0.3.1
|
|
95
|
+
|
|
96
|
+
1. Bump `lib/kotoshu/version.rb` → `0.3.1`.
|
|
97
|
+
2. Update CHANGELOG (or create if missing).
|
|
98
|
+
3. Update README quickstart if it references any 0.3.0-broken API.
|
|
99
|
+
4. Run full suite (`bundle exec rspec --tag ~network --tag ~onnx`)
|
|
100
|
+
green; `bundle exec rubocop` clean.
|
|
101
|
+
5. PR (not direct-to-main — see global rule).
|
|
102
|
+
6. After merge: tag `v0.3.1` (the **user** does this; never the agent).
|
|
103
|
+
|
|
104
|
+
## Release-level acceptance criteria
|
|
105
|
+
|
|
106
|
+
- `bundle exec rspec --tag ~network --tag ~onnx` → 0 failures
|
|
107
|
+
(excluding the explicitly-pending FastText Arabic case).
|
|
108
|
+
- `bundle exec rubocop` → clean.
|
|
109
|
+
- Lutaml migration regressions: 0.
|
|
110
|
+
- Pre-existing Hunspell `.aff` parser bugs: 0.
|
|
111
|
+
- No new public API changes — 0.3.1 is strictly a bug-fix release.
|
|
112
|
+
|
|
113
|
+
## Dependencies
|
|
114
|
+
|
|
115
|
+
- **Blocked by:** nothing.
|
|
116
|
+
- **Blocks:** `TODO.impl/37-hunspell-correctness-tier2.md` (the next
|
|
117
|
+
tier of Hunspell-fixture work).
|
|
118
|
+
|
|
119
|
+
## Status (audited 2026-09-03, gem is 0.6.x)
|
|
120
|
+
|
|
121
|
+
All clusters were closed long ago at 0.3.1 (commit `364a8fa`) or since;
|
|
122
|
+
re-audited against main today — nothing in this plan still reproduced.
|
|
123
|
+
|
|
124
|
+
| Cluster | State |
|
|
125
|
+
|---|---|
|
|
126
|
+
| 1 walking_skeleton `suggestions.words` | Done at 0.3.1 (`map(&:word)`); spec green today. |
|
|
127
|
+
| 2 walking_skeleton `to_h` → `to_hash` | Done at 0.3.1; spec green today. |
|
|
128
|
+
| 3 semantic_strategy_spec gating | Gating wired in `spec_helper.rb`; done at 0.3.1. (A different, later failure in this file — vocab.json format drift — was fixed today under plan 38.) |
|
|
129
|
+
| 4 read_aff REP/MAP/PFX/flag bugs | Fixed long ago; 7/7 examples green today. kotoshu-rs subsequently ported these parsers against 1315 conformance vectors exported from this gem. |
|
|
130
|
+
| 5 Arabic detection | Still `pending` as planned (upstream LID gap); suite green. |
|
|
131
|
+
| 6 symspell benchmark | Green today; no `:slow` change needed. |
|
|
132
|
+
| Phase D 0.3.1 release | Shipped long ago (gem is now 0.6.x). Versioning/release decisions remain owner-only. |
|
|
133
|
+
|
|
134
|
+
Nothing remains open in this plan.
|
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
# 37 — Hunspell Correctness (Tier 2)
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Drive the integrational Hunspell-fixture suite (`spec/integrational/
|
|
6
|
+
{lookup,suggest}_spec.rb`) from its current ~90-failure state to a
|
|
7
|
+
**bounded, categorized, steadily green** state. T2 is *correctness
|
|
8
|
+
work on the reader/lookup/suggest core*, not a rewrite.
|
|
9
|
+
|
|
10
|
+
## Why this is T2
|
|
11
|
+
|
|
12
|
+
The fixtures are ported from
|
|
13
|
+
[`spylls`](https://github.com/neolithos/spylls) (Python) which itself
|
|
14
|
+
mirrors upstream Hunspell's `tests/` directory. Each fixture exercises
|
|
15
|
+
a single directive (`FLAG long`, `COMPOUNDRULE`, `CHECKCOMPOUNDPATTERN`,
|
|
16
|
+
…). The current Ruby port passes the easy ones and fails the ones that
|
|
17
|
+
exercise less-trafficked code paths.
|
|
18
|
+
|
|
19
|
+
This work is **not** a release blocker for 0.3.1 (see
|
|
20
|
+
`TODO.impl/36-cleanup-regressions-0.3.1.md`). It is the next tier.
|
|
21
|
+
|
|
22
|
+
## Source of truth
|
|
23
|
+
|
|
24
|
+
- Reference: `/Users/mulgogi/src/external/hunspell/` (C++ upstream).
|
|
25
|
+
- Reference: `spylls` (Python port) — `algorithms/` is a Ruby line-by-line
|
|
26
|
+
port of `spylls/spylls/hunspell/`.
|
|
27
|
+
- Fixtures live in `spec/integrational/fixtures/` and are loaded via
|
|
28
|
+
`SpyllsTestHelper` (mixed into every spec through `spec_helper.rb`).
|
|
29
|
+
|
|
30
|
+
## Phase structure
|
|
31
|
+
|
|
32
|
+
Each phase ends with a **green run on the fixtures it targets** plus a
|
|
33
|
+
commit on the T2 branch. The phases are independent — partial completion
|
|
34
|
+
still ships value.
|
|
35
|
+
|
|
36
|
+
### Phase 2A — Flag-format correctness
|
|
37
|
+
|
|
38
|
+
**Fixtures affected:** `flaglong`, `flagnum`, `flagutf8`, `alias`,
|
|
39
|
+
`alias2`, `alias3`.
|
|
40
|
+
|
|
41
|
+
**Root causes (hypothesized, verify before fixing):**
|
|
42
|
+
|
|
43
|
+
1. `FLAG long` mode: `parse_flags` in `aff_reader.rb` uses
|
|
44
|
+
`string.scan(/../)` which fails on **odd-length** flag runs (the last
|
|
45
|
+
char gets dropped). Real Hunspell pads/aligns differently per context
|
|
46
|
+
(single flag = 2 chars, multi-flag = even run).
|
|
47
|
+
2. `FLAG num` mode: integer flags are parsed as strings, but downstream
|
|
48
|
+
`Set` operations sometimes compare against integers.
|
|
49
|
+
3. `FLAG UTF-8` mode: surrogate-pair / multi-byte handling is incorrect
|
|
50
|
+
on non-BMP characters (see also `utf8_nonbmp`).
|
|
51
|
+
4. `AF` flag aliases: aliases are looked up *per-rule* but should be
|
|
52
|
+
resolved once at load time and stashed on the `Affix` instance.
|
|
53
|
+
|
|
54
|
+
**Acceptance:** the 10 fixtures above pass in both `lookup_spec` and
|
|
55
|
+
`suggest_spec` (where a `.sug` fixture exists).
|
|
56
|
+
|
|
57
|
+
### Phase 2B — Compounding
|
|
58
|
+
|
|
59
|
+
**Fixtures affected:** `compoundflag`, `onlyincompound`, `compoundaffix`,
|
|
60
|
+
`compoundrule{1..8}`, `checkcompoundcase{,2,utf}`,
|
|
61
|
+
`checkcompounddup`, `checkcompoundpattern{,2,3,4}`,
|
|
62
|
+
`checkcompoundrep`, `checkcompoundtriple`, `compoundforbid`,
|
|
63
|
+
`simplifiedtriple`, `wordpair`, `forceucase`, `utfcompound`,
|
|
64
|
+
`fogemorpheme`, `opentaal_cpdpat{,2}`, `opentaal_forbiddenword{1,2}`,
|
|
65
|
+
`germancompounding{,old}`.
|
|
66
|
+
|
|
67
|
+
This is the largest single cluster — roughly 30 fixtures.
|
|
68
|
+
|
|
69
|
+
**Approach:**
|
|
70
|
+
|
|
71
|
+
- Bring the Ruby `algorithms/suggest.rb` / `algorithms/lookup.rb` back
|
|
72
|
+
in line with `spylls` commit-by-commit. Each fixture failure has a
|
|
73
|
+
corresponding code path in spylls; find the divergence.
|
|
74
|
+
- The `COMPOUNDRULE` regex matcher (`CompoundRule#fullmatch` in
|
|
75
|
+
`aff_data.rb`) has known issues with parenthesized flag groups —
|
|
76
|
+
audit against spylls' `algorithms/compoundrule.py`.
|
|
77
|
+
- `CHECKCOMPOUNDPATTERN` with replacement (`/`-separated third field)
|
|
78
|
+
is `pending` in `lookup_spec.rb` already; **finish** that work.
|
|
79
|
+
|
|
80
|
+
**Acceptance:** all 30 fixtures green. The three already-`pending`
|
|
81
|
+
`replacement in pattern` cases move from pending to passing.
|
|
82
|
+
|
|
83
|
+
### Phase 2C — Suggestion-quality
|
|
84
|
+
|
|
85
|
+
**Fixtures affected:** every fixture that has a `.sug` file in
|
|
86
|
+
`suggest_spec.rb`.
|
|
87
|
+
|
|
88
|
+
**Note:** suggestion output is order-sensitive and Hunspell's actual
|
|
89
|
+
output is the reference. We don't need *identical* output to upstream;
|
|
90
|
+
we need **the same set** with reasonable ordering. Where spylls asserts
|
|
91
|
+
ordering, mirror it; where Hunspell docs are silent, accept any
|
|
92
|
+
permutation.
|
|
93
|
+
|
|
94
|
+
**Approach:**
|
|
95
|
+
|
|
96
|
+
- `algorithms/ngram_suggest.rb`: port the latest ngram ranking from
|
|
97
|
+
spylls.
|
|
98
|
+
- `algorithms/phonet_suggest.rb`: audit against spylls; the PhonetTable
|
|
99
|
+
rule parser in `aff_data.rb` is the likely source of phantom
|
|
100
|
+
suggestions.
|
|
101
|
+
- `REP` patterns are applied too eagerly in the suggester (fixed in
|
|
102
|
+
T1 at the *reader* level; here we fix the *suggester* application).
|
|
103
|
+
|
|
104
|
+
**Acceptance:** every `suggest_spec` example either passes or carries
|
|
105
|
+
a documented `pending` with a concrete hypothesis.
|
|
106
|
+
|
|
107
|
+
### Phase 2D — Edge cases & language-specific
|
|
108
|
+
|
|
109
|
+
**Fixtures affected:** `1463589`, `1592880`, `1695964`, `1706659`,
|
|
110
|
+
`1975530`, `2970240`, `2970242`, `2999225`, `i35725`, `i53643`,
|
|
111
|
+
`i54633`, `i54980`, `i58202`, `slash`, `morph`, `ngram_utf_fix`,
|
|
112
|
+
`ph2`, `warn`, `korean`, `utf8_nonbmp`, `checksharps{,utf}`,
|
|
113
|
+
`dotless_i`, `IJ`, `nepali`, `allcaps{,2,3,_utf}`, `forbiddenword`,
|
|
114
|
+
`keepcase`, `nosuggest`, `iconv{,2}`, `oconv{,2}`,
|
|
115
|
+
`breakdefault`, `break`, `breakoff`, `fullstrip`, `zeroaffix`,
|
|
116
|
+
`needaffix{,2,3,4,5}`, `circumfix`, `condition{,_utf}`,
|
|
117
|
+
`conditionalprefix`, `complexprefixes{,2,utf}`, `affixes`,
|
|
118
|
+
`base{,_utf}`, `encoding`, `utf8{,_bom,_bom2}`,
|
|
119
|
+
`right_to_left_mark`, `ignore{,sug,_utf}`, `opentaal_keepcase`.
|
|
120
|
+
|
|
121
|
+
These are the bug-tracker cases (numbers are Hunspell GitHub issue
|
|
122
|
+
numbers) plus a long tail of single-directive edge cases.
|
|
123
|
+
|
|
124
|
+
**Approach:** bucket by directive; fix one bucket per PR.
|
|
125
|
+
|
|
126
|
+
**Acceptance:** `bundle exec rspec spec/integrational --tag ~network`
|
|
127
|
+
shows ≤ 5 failing examples (each with a documented `pending` and
|
|
128
|
+
an upstream link).
|
|
129
|
+
|
|
130
|
+
## Cross-cutting work
|
|
131
|
+
|
|
132
|
+
### C1. `SpyllsTestHelper` audit
|
|
133
|
+
|
|
134
|
+
The helper currently `include`s in every spec. The `read_dictionary`
|
|
135
|
+
helper loads fixtures by name and constructs a `Hunspell` dictionary.
|
|
136
|
+
Audit:
|
|
137
|
+
|
|
138
|
+
- Does it use `double()` anywhere? (Forbidden per global rule.)
|
|
139
|
+
- Does it bypass the public `Dictionary::Hunspell` API to poke at
|
|
140
|
+
internals? If so, the API is the wrong shape — fix the API.
|
|
141
|
+
- Are the fixture paths resolved correctly under `bundle exec` versus
|
|
142
|
+
direct invocation?
|
|
143
|
+
|
|
144
|
+
### C2. Reader/lookup OO refactor
|
|
145
|
+
|
|
146
|
+
The current `AffReader` returns a raw `Hash` with string keys. As T2
|
|
147
|
+
progresses, consider:
|
|
148
|
+
|
|
149
|
+
- A `Readers::AffData` value object (not a `Hash`) with typed accessors.
|
|
150
|
+
- A `Readers::DicData` parallel.
|
|
151
|
+
- The `Dictionary::Hunspell` constructor consumes `AffData`/`DicData`,
|
|
152
|
+
not hashes.
|
|
153
|
+
|
|
154
|
+
This is **optional** for T2 correctness — only do it if a fixture
|
|
155
|
+
demands it. If pursued, do it as the **last** step so the bulk of T2
|
|
156
|
+
remains "fix the bug, don't reshape the code".
|
|
157
|
+
|
|
158
|
+
## Acceptance criteria for T2 release
|
|
159
|
+
|
|
160
|
+
- `bundle exec rspec spec/integrational --tag ~network` → ≤ 5 failing
|
|
161
|
+
examples, each `pending` with an upstream issue link.
|
|
162
|
+
- `bundle exec rspec --tag ~network --tag ~onnx` → all green (T1 still
|
|
163
|
+
green).
|
|
164
|
+
- `bundle exec rubocop` → clean.
|
|
165
|
+
- **No new public API changes.** T2 is internal correctness.
|
|
166
|
+
|
|
167
|
+
## Dependencies
|
|
168
|
+
|
|
169
|
+
- **Blocked by:** `TODO.impl/36-cleanup-regressions-0.3.1.md`. The
|
|
170
|
+
integrational suite needs to be runnable on a clean baseline before
|
|
171
|
+
we can track fixture-level regressions.
|
|
172
|
+
- **Blocks:** nothing in T3. T3 features (CJK, RTL, grammar) build on a
|
|
173
|
+
correct core, but they can be developed in parallel branches against
|
|
174
|
+
the current (pre-T2) state if needed.
|
|
175
|
+
|
|
176
|
+
## Out of scope
|
|
177
|
+
|
|
178
|
+
- Performance optimization (`spec/benchmark/`, `spec/performance/`).
|
|
179
|
+
That's T4.
|
|
180
|
+
- New languages beyond what the fixtures cover. That's T3.
|
|
181
|
+
- Grammar engine. That's T3 (`TODO.impl/39-tier3-and-beyond.md`).
|
|
182
|
+
- ONNX semantic path correctness. That's `TODO.impl/38-onnx-semantic-gating.md`.
|
|
183
|
+
|
|
184
|
+
## Phase log (2026-09-04)
|
|
185
|
+
|
|
186
|
+
- **2A** — gem PR #104: AF alias values decode with the active FLAG
|
|
187
|
+
format (real FLAG-num bug; other hypotheses audited stale). Rust
|
|
188
|
+
re-synced same day (kotoshu-rs PR #4); vectors unchanged.
|
|
189
|
+
- **2B** — resolved via the owner collaborator PR #94 (HassanAkbar):
|
|
190
|
+
CHECKCOMPOUNDPATTERN replacement support ported from Hunspell +
|
|
191
|
+
compound checks module refactor + dedicated fixtures + perf spec;
|
|
192
|
+
merged by rebase after its stale-base failures cleared against
|
|
193
|
+
post-#106 main. A parallel independent implementation remains on
|
|
194
|
+
branch fix/t2-phase2b-compounding for reference.
|
|
195
|
+
- **2C** — gem PR #111: dot-split casing + INITCAP ngram-root skip
|
|
196
|
+
ported from hunspell suggestmgr (spylls never had them); the three
|
|
197
|
+
silently-excused pendings now pass for real. Full fixture dump:
|
|
198
|
+
exactly 3 vectors changed, zero collateral.
|
|
199
|
+
- Conformance vectors regenerated on post-#94 main: 2630 lines,
|
|
200
|
+
~13 vectors changed (replacement fixtures now produce corrected
|
|
201
|
+
outputs). kotoshu-rs re-sync (this pack + the behavior ports) is
|
|
202
|
+
the follow-up once 2D lands.
|
|
203
|
+
|
|
@@ -0,0 +1,218 @@
|
|
|
1
|
+
# 38 — ONNX Semantic Path Gating & Hardening
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Make the semantic path (ONNX FastText embeddings) a **first-class opt-in
|
|
6
|
+
feature** with a clear contract: when ONNX is absent, the traditional
|
|
7
|
+
path is unaffected; when ONNX is present, the semantic path is
|
|
8
|
+
testable, debuggable, and measurable.
|
|
9
|
+
|
|
10
|
+
## Why this is its own tier
|
|
11
|
+
|
|
12
|
+
The semantic path is the differentiator — it's the "semantic" in
|
|
13
|
+
"kotoshu is a semantic spell checker." But it depends on
|
|
14
|
+
`onnxruntime` (a native extension), a multi-megabyte model download per
|
|
15
|
+
language, and a non-trivial embedding pipeline. None of those can be
|
|
16
|
+
assumed present in CI or on a fresh `gem install`.
|
|
17
|
+
|
|
18
|
+
Right now the path is half-gated: specs that need ONNX fail noisily
|
|
19
|
+
rather than skipping; the runtime gating is correct but the test
|
|
20
|
+
gating are missing.
|
|
21
|
+
|
|
22
|
+
## Phase structure
|
|
23
|
+
|
|
24
|
+
### Phase A — Spec gating (the T1 deliverable)
|
|
25
|
+
|
|
26
|
+
This is also referenced from `TODO.impl/36-cleanup-regressions-0.3.1.md`
|
|
27
|
+
Phase A2; details here.
|
|
28
|
+
|
|
29
|
+
**Actions:**
|
|
30
|
+
|
|
31
|
+
1. Add `:onnx` to the `:integration`-tagged contexts in
|
|
32
|
+
`spec/kotoshu/suggestions/strategies/semantic_strategy_spec.rb`
|
|
33
|
+
(currently tagged `:integration` only — repurpose to `:onnx` since
|
|
34
|
+
the file is the sole user).
|
|
35
|
+
2. Update `spec/spec_helper.rb`:
|
|
36
|
+
```ruby
|
|
37
|
+
config.filter_run_excluding :onnx unless ENV.fetch("ONNX_TESTS", nil)
|
|
38
|
+
```
|
|
39
|
+
3. Document `ONNX_TESTS=1` in `CLAUDE.md` Notes section, mirroring the
|
|
40
|
+
`NETWORK_TESTS=1` pattern.
|
|
41
|
+
4. Add a "smoke" spec (always-run, no `:onnx` tag) that asserts
|
|
42
|
+
`Kotoshu::Models::OnnxModel::ONNX_LOADED` matches
|
|
43
|
+
`Gem::Specification.find_all_by_name("onnxruntime").any?`. This
|
|
44
|
+
catches the silent regression where the soft-require path breaks.
|
|
45
|
+
|
|
46
|
+
**Acceptance:**
|
|
47
|
+
|
|
48
|
+
- `bundle exec rspec` (default) → 0 ONNX-related failures.
|
|
49
|
+
- `ONNX_TESTS=1 bundle exec rspec spec/kotoshu/suggestions/strategies/semantic_strategy_spec.rb`
|
|
50
|
+
→ either all green (if onnxruntime installed + model cached) or all
|
|
51
|
+
skipped with a clear "model not cached; run `kotoshu setup :en --model`"
|
|
52
|
+
message.
|
|
53
|
+
|
|
54
|
+
### Phase B — Runtime gating hardening
|
|
55
|
+
|
|
56
|
+
**Audit each entry point into the semantic path:**
|
|
57
|
+
|
|
58
|
+
1. `Kotoshu::Analyzers::SemanticAnalyzer` — already raises
|
|
59
|
+
`Models::OnnxModel::OnnxUnavailable`. Verify the message is
|
|
60
|
+
actionable.
|
|
61
|
+
2. `Suggestions::Strategies::SemanticStrategy#generate` — returns an
|
|
62
|
+
empty `SuggestionSet` when ONNX is unavailable. This is silent;
|
|
63
|
+
consider logging at debug level.
|
|
64
|
+
3. `Kotoshu.correct?` / `Kotoshu.suggest` / `Kotoshu.check` — these
|
|
65
|
+
should *never* hit the semantic path unless explicitly requested
|
|
66
|
+
(e.g. via `Kotoshu.suggest(word, semantic: true)`). Verify.
|
|
67
|
+
4. `KOTOSHU_NO_ONNX=1` is documented; verify it short-circuits all
|
|
68
|
+
semantic entry points.
|
|
69
|
+
|
|
70
|
+
**Acceptance:** with `onnxruntime` *installed* but
|
|
71
|
+
`KOTOSHU_NO_ONNX=1` set, `bundle exec rspec --tag ~network --tag ~onnx`
|
|
72
|
+
passes. With `onnxruntime` *uninstalled*, same suite passes.
|
|
73
|
+
|
|
74
|
+
### Phase C — Model cache correctness
|
|
75
|
+
|
|
76
|
+
The semantic path depends on `Cache::ModelCache` finding a model file
|
|
77
|
+
under `~/.cache/kotoshu/models/{lang}/`. Issues to address:
|
|
78
|
+
|
|
79
|
+
1. **Manifest validation.** `kotoshu/models-fasttext-onnx` ships a
|
|
80
|
+
`manifest.json` (18 entries per `kotoshu-content-repos-state.md`
|
|
81
|
+
memory). Verify `ModelCache` reads the manifest, not a glob, and
|
|
82
|
+
rejects unknown shapes.
|
|
83
|
+
2. **Version pinning.** Each manifest entry should have a content hash
|
|
84
|
+
or version; verify the cache checks it.
|
|
85
|
+
3. **Stale-model recovery.** If the model file is truncated or the
|
|
86
|
+
onnxruntime session can't deserialize, the error message should
|
|
87
|
+
point to `kotoshu cache download :en --model` (or equivalent).
|
|
88
|
+
|
|
89
|
+
**Acceptance:** deliberately corrupt a cached model file; the next
|
|
90
|
+
`Kotoshu.suggest(word, semantic: true)` raises a clear, actionable
|
|
91
|
+
error and points the user to the cache subcommand.
|
|
92
|
+
|
|
93
|
+
### Phase D — Vocabulary / embedding pipeline tests
|
|
94
|
+
|
|
95
|
+
The `Embeddings::*` namespace (vocabulary, similarity engine, search,
|
|
96
|
+
LRU cache) is undertested. Add specs for:
|
|
97
|
+
|
|
98
|
+
1. `Embeddings::Vocabulary` — token → id, id → token round-trip; OOV
|
|
99
|
+
behavior; boundary indices.
|
|
100
|
+
2. `Embeddings::SimilarityEngine` — cosine similarity of orthogonal
|
|
101
|
+
vs identical vectors; dimension mismatch handling.
|
|
102
|
+
3. `Embeddings::LruCache` — eviction at capacity; hit/miss accounting.
|
|
103
|
+
|
|
104
|
+
**These specs do not require ONNX** — they test the math, not the
|
|
105
|
+
runtime. They should be in the always-run suite.
|
|
106
|
+
|
|
107
|
+
**Acceptance:** the three classes above have direct unit specs (no
|
|
108
|
+
`:onnx` tag), all green in default `bundle exec rspec`.
|
|
109
|
+
|
|
110
|
+
### Phase E — End-to-end semantic smoke test
|
|
111
|
+
|
|
112
|
+
A single `:onnx`-tagged spec that:
|
|
113
|
+
|
|
114
|
+
1. Resolves the English model from cache (skip if missing).
|
|
115
|
+
2. Builds a `SemanticAnalyzer` for English.
|
|
116
|
+
3. Asserts that `analyze("their")` vs `analyze("there")` returns
|
|
117
|
+
distinguishable contexts.
|
|
118
|
+
4. Asserts that `Kotoshu.suggest("helo", semantic: true)` returns
|
|
119
|
+
"hello" in the top 5.
|
|
120
|
+
|
|
121
|
+
This is the "does the semantic path actually work end-to-end" guard.
|
|
122
|
+
|
|
123
|
+
**Acceptance:** spec passes when ONNX is available; skips with a clear
|
|
124
|
+
message otherwise.
|
|
125
|
+
|
|
126
|
+
## Cross-cutting: OCP/MECE audit
|
|
127
|
+
|
|
128
|
+
While doing the above, audit the embeddings namespace for:
|
|
129
|
+
|
|
130
|
+
- **OCP violations:** places where adding a new model type (e.g.
|
|
131
|
+
Word2Vec, GloVe) requires editing a switch instead of registering.
|
|
132
|
+
- **MECE violations:** overlap between `Embeddings::SimilarityEngine`
|
|
133
|
+
and `Embeddings::SimilaritySearch` — if they have overlapping
|
|
134
|
+
responsibilities, consolidate or split cleanly.
|
|
135
|
+
- **DRY:** the embedding-pipeline code is the prime suspect for
|
|
136
|
+
copy-paste between FastText and ONNX paths.
|
|
137
|
+
|
|
138
|
+
Document findings; address in T3 if scope creep risks the release.
|
|
139
|
+
|
|
140
|
+
## Acceptance criteria
|
|
141
|
+
|
|
142
|
+
- Default `bundle exec rspec` → 0 ONNX-related failures, 0 ONNX-related
|
|
143
|
+
skips leaking into the non-`:onnx` suite.
|
|
144
|
+
- `ONNX_TESTS=1 bundle exec rspec --tag onnx` → all green (with model
|
|
145
|
+
cached) or all gracefully skipped (without).
|
|
146
|
+
- `KOTOSHU_NO_ONNX=1 bundle exec rspec` → identical pass rate to the
|
|
147
|
+
default run.
|
|
148
|
+
- Semantic path is documented end-to-end in a single place (probably
|
|
149
|
+
`docs/semantic-path.md`).
|
|
150
|
+
|
|
151
|
+
## Dependencies
|
|
152
|
+
|
|
153
|
+
- **Blocked by:** nothing.
|
|
154
|
+
- **Blocks:** `TODO.impl/39-tier3-and-beyond.md` (the grammar / CJK
|
|
155
|
+
work uses the embedding pipeline; the gating must be solid first).
|
|
156
|
+
|
|
157
|
+
## Status (re-audited 2026-09-03)
|
|
158
|
+
|
|
159
|
+
Done long ago: Phase A gating (`spec_helper.rb` `:onnx` exclusion,
|
|
160
|
+
`ONNX_TESTS=1` opt-in documented in CLAUDE.md), Phase D unit specs
|
|
161
|
+
(`embeddings/vocabulary_spec`, `similarity_engine_spec`, `lru_cache_spec`,
|
|
162
|
+
`models/models_spec`), Phase E e2e spec (`spec/kotoshu/semantic_e2e_spec.rb`,
|
|
163
|
+
tagged `:onnx` with skip guards).
|
|
164
|
+
|
|
165
|
+
Verified today:
|
|
166
|
+
|
|
167
|
+
- Phase B: `KOTOSHU_NO_ONNX=1 bundle exec rspec --tag ~network --tag ~onnx`
|
|
168
|
+
→ 3285 examples, 0 failures (one extra skip: the new load-failure spec
|
|
169
|
+
correctly skips when the runtime is off). `SemanticStrategy` empty-return
|
|
170
|
+
still only warns at construction (`warn … if $VERBOSE`); there is no
|
|
171
|
+
debug-log infrastructure in the gem — left as-is (building one is out of
|
|
172
|
+
scope for this residue pass).
|
|
173
|
+
|
|
174
|
+
Fixed today (branch `fix/plan36-38-residue`):
|
|
175
|
+
|
|
176
|
+
- **vocab.json format drift** (the live 5 failures in
|
|
177
|
+
`semantic_strategy_spec`): the models repo / `fasttext_to_onnx.py` ship
|
|
178
|
+
vocab.json as `{"vocab_size", "word_to_idx"}`; `Embeddings::Vocabulary`
|
|
179
|
+
and `Models::OnnxModel.from_file` only parsed flat maps, so
|
|
180
|
+
`SemanticStrategy.new(language_code: "en")` crashed in
|
|
181
|
+
`Vocabulary#initialize` (`TypeError` via `Hash#<`). Both loaders now
|
|
182
|
+
share `Vocabulary.word_to_index_from_document` (flat, wrapped, and array
|
|
183
|
+
shapes); `Vocabulary#initialize` raises a clear ArgumentError for
|
|
184
|
+
non-Integer index values instead of the cryptic TypeError.
|
|
185
|
+
- **LFS pointer corruption** (Phase C root cause):
|
|
186
|
+
`raw.githubusercontent.com` serves 134-byte LFS pointer stubs for the
|
|
187
|
+
LFS-tracked `*.onnx`; the legacy full-tier download cached a stub and
|
|
188
|
+
recorded the stub's own sha256, so checksum verification passed forever
|
|
189
|
+
and the runtime died with a raw `OnnxRuntime::Error: Protobuf parsing
|
|
190
|
+
failed`. Fixes: `SourceRegistry` fetches `:model` from
|
|
191
|
+
`media.githubusercontent.com/media` (LFS host; custom mirrors untouched),
|
|
192
|
+
`ModelCache` refuses pointer stubs at download time and at
|
|
193
|
+
`verify_cached_integrity!` (purges stub + metadata, raises an error
|
|
194
|
+
pointing at `kotoshu cache download :{lang} --model`), and
|
|
195
|
+
`Embeddings::OnnxRuntimeModel#load!` wraps session-creation failures in
|
|
196
|
+
the same actionable `Kotoshu::Error`.
|
|
197
|
+
- **`Models::OnnxModel` session rot** (previously masked by the vocab
|
|
198
|
+
crash): `OnnxRuntime::Session` (nonexistent class) →
|
|
199
|
+
`OnnxRuntime::InferenceSession` with graph-detected IO names
|
|
200
|
+
(`word_index`/`embedding`); tensor I/O now uses real values instead of
|
|
201
|
+
`pack('q<')`/`unpack('e*')` byte surgery; batching executes sequential
|
|
202
|
+
single lookups (the converted graph's input is fixed shape [1]).
|
|
203
|
+
- **Spec drift**: strategy config examples no longer construct against the
|
|
204
|
+
developer's real `en` cache (use `xx`, per the file's own convention —
|
|
205
|
+
construction could otherwise trigger a 120MB download mid-spec);
|
|
206
|
+
identical-word similarity compared with float tolerance; removed the
|
|
207
|
+
stale `#embedding_for` describe (that API lives on the model, not the
|
|
208
|
+
strategy). New direct specs: `models/onnx_model_spec` (both vocab shapes),
|
|
209
|
+
`embeddings/onnx_runtime_model_spec` (actionable corrupt-model error).
|
|
210
|
+
|
|
211
|
+
`ONNX_TESTS=1 bundle exec rspec` on the semantic specs: 32 examples,
|
|
212
|
+
0 failures against the real cached English model.
|
|
213
|
+
|
|
214
|
+
Remains (noted, not blocking): `Embeddings::OnnxRuntimeModel`'s
|
|
215
|
+
`run_batch_inference` feeds >1 indices to a fixed-shape-[1] graph, so
|
|
216
|
+
multi-index `get_embeddings` / `preload_embeddings!` fail at runtime; the
|
|
217
|
+
live single-lookup path is unaffected. Belongs with the tiered-model work
|
|
218
|
+
(plans 39/67).
|