kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
data/Rakefile
ADDED
|
@@ -0,0 +1,150 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "bundler/gem_tasks"
|
|
4
|
+
require "rspec/core/rake_task"
|
|
5
|
+
|
|
6
|
+
RSpec::Core::RakeTask.new(:spec)
|
|
7
|
+
|
|
8
|
+
require "rubocop/rake_task"
|
|
9
|
+
|
|
10
|
+
RuboCop::RakeTask.new
|
|
11
|
+
|
|
12
|
+
task default: %i[spec rubocop]
|
|
13
|
+
|
|
14
|
+
# Conformance failure reporters (kept as lambdas so nothing leaks into
|
|
15
|
+
# Object from the Rakefile).
|
|
16
|
+
report_failures = lambda do |failures, label = nil|
|
|
17
|
+
failures.first(10).each do |failure|
|
|
18
|
+
prefix = label ? "[#{label}] " : ""
|
|
19
|
+
puts "#{prefix}##{failure.index} #{failure.kind} #{failure.input.inspect} " \
|
|
20
|
+
"on #{failure.dictionary}: expected #{failure.expected.inspect}, got #{failure.actual.inspect}"
|
|
21
|
+
end
|
|
22
|
+
puts "... and #{failures.size - 10} more" if failures.size > 10
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
report_divergences = lambda do |divergences|
|
|
26
|
+
divergences.first(10).each do |divergence|
|
|
27
|
+
puts "[divergence] ##{divergence.index} #{divergence.kind} #{divergence.input.inspect} " \
|
|
28
|
+
"on #{divergence.dictionary}: ruby #{divergence.expected.inspect}, " \
|
|
29
|
+
"native #{divergence.actual.inspect}"
|
|
30
|
+
end
|
|
31
|
+
puts "... and #{divergences.size - 10} more" if divergences.size > 10
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
namespace :kotoshu do
|
|
35
|
+
namespace :staged_languages do
|
|
36
|
+
desc <<~DESC
|
|
37
|
+
Regenerate the vendored staged-language list (plan 107). Fetches \
|
|
38
|
+
the dictionaries manifest at the SourceRegistry pin, extracts the \
|
|
39
|
+
distinct language codes from its resources, and rewrites \
|
|
40
|
+
lib/kotoshu/cache/staged_languages.rb in place so \
|
|
41
|
+
LanguageCache::AVAILABLE_LANGUAGES tracks the manifest. Run after \
|
|
42
|
+
the dictionaries repo stages new languages.
|
|
43
|
+
DESC
|
|
44
|
+
task :sync do
|
|
45
|
+
require "kotoshu"
|
|
46
|
+
require "json"
|
|
47
|
+
|
|
48
|
+
url = Kotoshu::SourceRegistry.new.url_for(:dict_manifest)
|
|
49
|
+
puts "Fetching #{url}"
|
|
50
|
+
bytes = Kotoshu::Integrity::NetHTTP.get(url)
|
|
51
|
+
abort "manifest not found at #{url}" if bytes.nil?
|
|
52
|
+
|
|
53
|
+
manifest = JSON.parse(bytes)
|
|
54
|
+
codes = manifest.fetch("resources").values.map { |r| r.fetch("language") }.uniq.sort
|
|
55
|
+
abort "manifest carried no languages" if codes.empty?
|
|
56
|
+
|
|
57
|
+
target = File.expand_path("lib/kotoshu/cache/staged_languages.rb", __dir__)
|
|
58
|
+
source = File.read(target)
|
|
59
|
+
updated = source.sub(/CODES = %w\[\n(?: {8}\S+(?: \S+)*\n)+ {6}\]\.freeze/) do
|
|
60
|
+
wrapped = codes.each_slice(10).map { |slice| " #{slice.join(' ')}" }.join("\n")
|
|
61
|
+
"CODES = %w[\n#{wrapped}\n ].freeze"
|
|
62
|
+
end
|
|
63
|
+
abort "CODES block not found in #{target}" if updated == source
|
|
64
|
+
|
|
65
|
+
File.write(target, updated)
|
|
66
|
+
puts "Rewrote #{target}: #{codes.size} languages (was #{Kotoshu::Cache::StagedLanguages::CODES.size})"
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
namespace :conformance do
|
|
71
|
+
desc <<~DESC
|
|
72
|
+
Export conformance vectors for kotoshu-rs (plan 67 M3). Walks the Spylls \
|
|
73
|
+
fixtures (spec/integrational/fixtures) plus the spec fixture corpus, runs \
|
|
74
|
+
the real engine over every fixture word, and writes one JSON object per \
|
|
75
|
+
line. `expected` values are FROZEN engine behavior (byte-reproducible \
|
|
76
|
+
cross-implementation), not claims about linguistic correctness. \
|
|
77
|
+
Output path: ENV["CONFORMANCE_VECTORS_PATH"] or conformance/vectors.jsonl \
|
|
78
|
+
(repo root). The generated file IS committed -- conformance/ is \
|
|
79
|
+
deliberately NOT gitignored -- and kotoshu-rs copies it into \
|
|
80
|
+
tests/conformance/ via its documented sync step so its CI needs no gem.
|
|
81
|
+
DESC
|
|
82
|
+
task :export do
|
|
83
|
+
path = ENV.fetch("CONFORMANCE_VECTORS_PATH", nil) || "conformance/vectors.jsonl"
|
|
84
|
+
require "kotoshu"
|
|
85
|
+
result = Kotoshu::ConformanceExporter.export(path: path)
|
|
86
|
+
puts "Wrote #{result.row_count} vectors from #{result.corpus_count} corpora to #{result.path}"
|
|
87
|
+
unless result.skipped_corpora.empty?
|
|
88
|
+
puts "Skipped corpora:"
|
|
89
|
+
result.skipped_corpora.each { |id, reason| puts " #{id}: #{reason}" }
|
|
90
|
+
end
|
|
91
|
+
result.skipped_words.each { |(id, word), reason| puts " Skipped word #{id} #{word.inspect}: #{reason}" }
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
desc <<~DESC
|
|
95
|
+
Replay conformance/vectors.jsonl through the pure-Ruby engine \
|
|
96
|
+
(plan 66 dual-backend suite). Every vector must reproduce its frozen \
|
|
97
|
+
expectation; exits nonzero on any mismatch.
|
|
98
|
+
DESC
|
|
99
|
+
task :ruby do
|
|
100
|
+
require "kotoshu"
|
|
101
|
+
result = Kotoshu::ConformanceRunner.new.run(backend: :ruby)
|
|
102
|
+
puts "ruby backend: #{result.row_count} vectors, #{result.failures.size} failures"
|
|
103
|
+
report_failures.call(result.failures)
|
|
104
|
+
abort "kotoshu:conformance:ruby: #{result.failures.size} of #{result.row_count} vectors diverged" unless result.ok?
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
desc <<~DESC
|
|
108
|
+
Replay conformance/vectors.jsonl through the native (Rust) engine \
|
|
109
|
+
(plan 66 dual-backend suite). Requires the extension to be built \
|
|
110
|
+
(rake compile); exits nonzero on any mismatch.
|
|
111
|
+
DESC
|
|
112
|
+
task :native do
|
|
113
|
+
require "kotoshu"
|
|
114
|
+
unless Kotoshu::Native.available?
|
|
115
|
+
abort "kotoshu:conformance:native requires the native extension -- run `rake compile` first"
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
result = Kotoshu::ConformanceRunner.new.run(backend: :native)
|
|
119
|
+
puts "native backend: #{result.row_count} vectors, #{result.failures.size} failures"
|
|
120
|
+
report_failures.call(result.failures)
|
|
121
|
+
abort "kotoshu:conformance:native: #{result.failures.size} of #{result.row_count} vectors diverged" unless result.ok?
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
desc <<~DESC
|
|
125
|
+
Run the conformance vectors through BOTH backends and diff every row \
|
|
126
|
+
(plan 66 acceptance: zero diffs). Also fails when either backend \
|
|
127
|
+
diverges from the frozen expectations. Requires the extension \
|
|
128
|
+
(rake compile).
|
|
129
|
+
DESC
|
|
130
|
+
task :compare do
|
|
131
|
+
require "kotoshu"
|
|
132
|
+
unless Kotoshu::Native.available?
|
|
133
|
+
abort "kotoshu:conformance:compare requires the native extension -- run `rake compile` first"
|
|
134
|
+
end
|
|
135
|
+
|
|
136
|
+
result = Kotoshu::ConformanceRunner.new.compare
|
|
137
|
+
puts "compare: #{result.ruby.row_count} vectors -- ruby failures: " \
|
|
138
|
+
"#{result.ruby.failures.size}, native failures: #{result.native.failures.size}, " \
|
|
139
|
+
"backend divergences: #{result.divergences.size}"
|
|
140
|
+
report_failures.call(result.ruby.failures, "ruby")
|
|
141
|
+
report_failures.call(result.native.failures, "native")
|
|
142
|
+
report_divergences.call(result.divergences)
|
|
143
|
+
unless result.ok?
|
|
144
|
+
abort "kotoshu:conformance:compare: backends diverged " \
|
|
145
|
+
"(ruby #{result.ruby.failures.size}, native #{result.native.failures.size}, " \
|
|
146
|
+
"cross #{result.divergences.size} of #{result.ruby.row_count})"
|
|
147
|
+
end
|
|
148
|
+
end
|
|
149
|
+
end
|
|
150
|
+
end
|
data/SECURITY.md
ADDED
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# Security Policy
|
|
2
|
+
|
|
3
|
+
## Reporting a Vulnerability
|
|
4
|
+
|
|
5
|
+
Email security@kotoshu.org with details and a reproduction. You will
|
|
6
|
+
receive an acknowledgement within 72 hours. Please do not file public
|
|
7
|
+
issues for security reports.
|
|
8
|
+
|
|
9
|
+
## Supported Versions
|
|
10
|
+
|
|
11
|
+
The latest 0.x minor is supported. Older minors are not maintained.
|
|
12
|
+
|
|
13
|
+
## Threat Model
|
|
14
|
+
|
|
15
|
+
Kotoshu downloads dictionaries, frequency lists, and (optionally)
|
|
16
|
+
embedding models from public GitHub repositories on first use:
|
|
17
|
+
|
|
18
|
+
| Resource | Source repo | Cached at |
|
|
19
|
+
|---|---|---|
|
|
20
|
+
| Spelling dictionaries | `kotoshu/dictionaries` | `~/.kotoshu/languages/{code}/` |
|
|
21
|
+
| Kelly frequency lists | `kotoshu/frequency-list-kelly` | `~/.kotoshu/frequency-lists/{code}/` |
|
|
22
|
+
| FastText / ONNX models | `kotoshu/models-fasttext-onnx` | `~/.kotoshu/models/{code}/` |
|
|
23
|
+
|
|
24
|
+
Downloads flow over HTTPS from `raw.githubusercontent.com`. The threat
|
|
25
|
+
model assumes GitHub serves the bytes the repository owner committed,
|
|
26
|
+
and treats anyone with push access to those repos as trusted.
|
|
27
|
+
|
|
28
|
+
## Integrity Verification
|
|
29
|
+
|
|
30
|
+
Each content repo may ship a `manifest.json` at its root listing every
|
|
31
|
+
file with its SHA-256 hash, size, and language/type tags. When a
|
|
32
|
+
manifest is present, every download is verified against it:
|
|
33
|
+
|
|
34
|
+
1. The manifest is fetched once per cache session.
|
|
35
|
+
2. Each downloaded file's SHA-256 is computed locally and compared to
|
|
36
|
+
the manifest entry.
|
|
37
|
+
3. On mismatch, `Kotoshu::IntegrityError` is raised with the expected
|
|
38
|
+
and actual hashes. The download is rejected and the cache is left
|
|
39
|
+
untouched.
|
|
40
|
+
4. Every verification outcome (verified / unverified / mismatch /
|
|
41
|
+
missing) is appended to `~/.kotoshu/audit.log` as one JSON object
|
|
42
|
+
per line.
|
|
43
|
+
|
|
44
|
+
### Graceful Degradation
|
|
45
|
+
|
|
46
|
+
When a manifest is **absent** (HTTP 404), verification silently
|
|
47
|
+
downgrades to `"unverified"` status and the download proceeds. This
|
|
48
|
+
preserves forward compatibility with repos that have not yet shipped a
|
|
49
|
+
manifest. The audit log records the difference.
|
|
50
|
+
|
|
51
|
+
### Strict Mode
|
|
52
|
+
|
|
53
|
+
`Kotoshu.spellchecker_for(lang, strict: true)` (and the CLI's
|
|
54
|
+
`--strict` flag) re-raise on any optional-resource failure — including
|
|
55
|
+
integrity mismatches on frequency data — instead of silently
|
|
56
|
+
degrading. Spelling-dictionary integrity is always enforced.
|
|
57
|
+
|
|
58
|
+
## Cache Layout
|
|
59
|
+
|
|
60
|
+
The cache is written under `$KOTOSHU_HOME` (default `~/.kotoshu/`).
|
|
61
|
+
Files are created with the user's default umask. Cache contents are
|
|
62
|
+
not encrypted at rest.
|
|
63
|
+
|
|
64
|
+
## Audit Log
|
|
65
|
+
|
|
66
|
+
`~/.kotoshu/audit.log` is append-only, JSON-per-line, and never
|
|
67
|
+
auto-rotated. Operators in multi-user environments should rotate it
|
|
68
|
+
via logrotate or equivalent. To inspect:
|
|
69
|
+
|
|
70
|
+
[source,bash]
|
|
71
|
+
----
|
|
72
|
+
cat ~/.kotoshu/audit.log | jq .
|
|
73
|
+
----
|
|
74
|
+
|
|
75
|
+
To clear the audit log:
|
|
76
|
+
|
|
77
|
+
[source,ruby]
|
|
78
|
+
----
|
|
79
|
+
Kotoshu::Integrity::AuditLog.new(path: "#{ENV['HOME']}/.kotoshu/audit.log").clear!
|
|
80
|
+
----
|
|
81
|
+
|
|
82
|
+
## Network Egress
|
|
83
|
+
|
|
84
|
+
`offline: true` (or `KOTOSHU_OFFLINE=1` or `--offline`) disables all
|
|
85
|
+
network egress and only reads from the on-disk cache. If a required
|
|
86
|
+
resource is not cached, the call raises `Kotoshu::ResourceNotCachedError`
|
|
87
|
+
(CLI exits 3). Use `kotoshu fetch LANGUAGE` to pre-warm the cache in
|
|
88
|
+
environments without outbound network access.
|
|
89
|
+
|
|
90
|
+
## Scope
|
|
91
|
+
|
|
92
|
+
This policy covers the kotoshu gem itself. Vulnerabilities in
|
|
93
|
+
dependencies (Thor, suika, onnxruntime) should be reported upstream.
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
# 00-cut-0.1 — The 0.1.0 Release Cut
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Ship `kotoshu-0.1.0.gem` to rubygems.org with a clear, honest scope: the
|
|
6
|
+
pure-Ruby API and CLI work for English via system dictionaries; everything
|
|
7
|
+
else is documented as in-progress.
|
|
8
|
+
|
|
9
|
+
This is **not** v1.0. The point of 0.1 is to be installable, runnable,
|
|
10
|
+
and discoverable — to claim the name on rubygems.org and establish the
|
|
11
|
+
public API shape.
|
|
12
|
+
|
|
13
|
+
## 0.1.0 scope (locked)
|
|
14
|
+
|
|
15
|
+
In scope for 0.1.0:
|
|
16
|
+
|
|
17
|
+
| Capability | Status |
|
|
18
|
+
|---|---|
|
|
19
|
+
| `Kotoshu.correct?(word)` / `suggest` / `check` / `check_file` | ✅ Working |
|
|
20
|
+
| `Kotoshu.detect_language(text)` (FastText LID) | ✅ Working |
|
|
21
|
+
| `kotoshu check TARGET` basic CLI | ✅ Working |
|
|
22
|
+
| `kotoshu dict list/info`, `kotoshu cache list/info/...` | ✅ Working |
|
|
23
|
+
| Hunspell dictionary backend (Ruby port) | ✅ Working for common cases |
|
|
24
|
+
| CSpell, UnixWords, PlainText, Custom backends | ✅ Working |
|
|
25
|
+
| Edit distance / phonetic / keyboard / n-gram / symspell strategies | ✅ Working |
|
|
26
|
+
| Configuration via `Kotoshu.configure` + ENV (`KOTOSHU_*`) + CLI | ✅ Working |
|
|
27
|
+
| 3-layer cache (LanguageCache, FrequencyCache, ModelCache) | ✅ Working |
|
|
28
|
+
| Document parsing (Plain text, Markdown, AsciiDoc) | ✅ Working |
|
|
29
|
+
| LICENSE (BSD-2-Clause), CHANGELOG.md, accurate README | ✅ Shipped |
|
|
30
|
+
|
|
31
|
+
Explicitly **deferred** to later 0.x or 1.0:
|
|
32
|
+
|
|
33
|
+
| Capability | Target | Plan |
|
|
34
|
+
|---|---|---|
|
|
35
|
+
| Unified `ResourceManager` for "any language, just works" | 0.3 | `03-dynamic-download.md` |
|
|
36
|
+
| `--interactive`, `--format sarif`, `--model`, `--language auto` wired | 0.2 | `02-cli-unification.md` |
|
|
37
|
+
| Hybrid (Hunspell + ONNX) as default | 0.4 | `05-semantic-path.md` |
|
|
38
|
+
| Hunspell compound rules, circumfix, ICONV/OCONV, ß, dotless-i | 0.5 | `01-hunspell-correctness.md` |
|
|
39
|
+
| ≥30 language modules wired | 0.6 | `04-language-modules.md` |
|
|
40
|
+
| Grammar rule packs | 0.7 | `08-grammar-engine.md` |
|
|
41
|
+
| CJK (Japanese, Chinese) | 0.8 | `06-cjk-support.md` |
|
|
42
|
+
| RTL (Arabic, Hebrew, Persian, Urdu) | 0.9 | `07-rtl-support.md` |
|
|
43
|
+
| Integrity verification (checksums, signed manifests) | 0.4 | `09-integrity-security.md` |
|
|
44
|
+
| v1.0 | — | `11-release-v1.md` |
|
|
45
|
+
|
|
46
|
+
## Tasks executed for 0.1.0
|
|
47
|
+
|
|
48
|
+
1. ✅ Audit current state — `bundle install`, smoke-test `Kotoshu.correct?`,
|
|
49
|
+
run rspec. Result: 803/866 (92.7%) passing; VERSION already `0.1.0`.
|
|
50
|
+
2. ✅ Add `LICENSE` (BSD-2-Clause — was missing despite gemspec claim).
|
|
51
|
+
3. ✅ Add `CHANGELOG.md` with full 0.1.0 scope and known-limitation list.
|
|
52
|
+
4. ✅ Fix README accuracy: license, status section at top, ONNX env gate.
|
|
53
|
+
5. ✅ Tighten CI workflow — rubocop no longer `continue-on-error`.
|
|
54
|
+
6. ✅ Prepare release checklist (see "Release execution" below).
|
|
55
|
+
|
|
56
|
+
## Release execution (manual — requires maintainer)
|
|
57
|
+
|
|
58
|
+
The global project rule is "AI never pushes tags, never pushes to main,
|
|
59
|
+
never publishes." The following must be run by a human:
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
# 1. On a clean main, verify everything passes
|
|
63
|
+
git checkout main
|
|
64
|
+
git pull
|
|
65
|
+
bundle install
|
|
66
|
+
bundle exec rspec
|
|
67
|
+
bundle exec rubocop
|
|
68
|
+
|
|
69
|
+
# 2. Verify VERSION
|
|
70
|
+
grep VERSION lib/kotoshu/version.rb
|
|
71
|
+
# Should read: VERSION = "0.1.0"
|
|
72
|
+
|
|
73
|
+
# 3. Build the gem
|
|
74
|
+
gem build kotoshu.gemspec
|
|
75
|
+
# Produces kotoshu-0.1.0.gem
|
|
76
|
+
|
|
77
|
+
# 4. Sanity-check the gem contents
|
|
78
|
+
gem contents kotoshu-0.1.0.gem --tarball
|
|
79
|
+
|
|
80
|
+
# 5. Tag (the human decides when)
|
|
81
|
+
git tag -a v0.1.0 -m "kotoshu 0.1.0 — first public release"
|
|
82
|
+
git push origin v0.1.0
|
|
83
|
+
|
|
84
|
+
# 6. Publish to rubygems.org
|
|
85
|
+
gem push kotoshu-0.1.0.gem
|
|
86
|
+
|
|
87
|
+
# 7. Cut a GitHub Release
|
|
88
|
+
gh release create v0.1.0 --notes-from-tag
|
|
89
|
+
|
|
90
|
+
# 8. Smoke test from a clean environment
|
|
91
|
+
docker run --rm -it ruby:3.3 bash -c '
|
|
92
|
+
gem install kotoshu
|
|
93
|
+
ruby -e "require \"kotoshu\"; puts Kotoshu.correct?(\"hello\")"
|
|
94
|
+
echo "Hello wrold" | ruby -e "require \"kotoshu\"; puts Kotoshu.check(STDIN.read).errors.map(&:word)"
|
|
95
|
+
'
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
## Post-release
|
|
99
|
+
|
|
100
|
+
- Update `TODO.impl/00-vision.md` to reflect that 0.1 shipped; the
|
|
101
|
+
critical path now runs through `03-dynamic-download` →
|
|
102
|
+
`04-language-modules` → `11-release-v1`.
|
|
103
|
+
- Start `02-cli-unification.md` (next-highest user-visible value).
|
|
104
|
+
- Start `01-hunspell-correctness.md` task 1 (audit) to get an honest
|
|
105
|
+
fixture-pass number.
|
|
106
|
+
|
|
107
|
+
## Acceptance criteria
|
|
108
|
+
|
|
109
|
+
- `gem install kotoshu` works on Ruby 3.1+
|
|
110
|
+
- `Kotoshu.correct?("hello")` returns `true` on a fresh install with
|
|
111
|
+
no env vars set
|
|
112
|
+
- `Kotoshu.correct?("helo")` returns `false`
|
|
113
|
+
- `kotoshu check README.adoc` runs without crashing
|
|
114
|
+
- README has no false claims (license, model count, CLI flags)
|
|
115
|
+
- LICENSE and CHANGELOG.md exist
|
|
116
|
+
- rubygems.org page shows v0.1.0 with correct metadata
|
|
117
|
+
|
|
118
|
+
## Status
|
|
119
|
+
|
|
120
|
+
v0.1.0 release-prep complete on this branch (2026-06-25). Release
|
|
121
|
+
execution pending maintainer sign-off.
|
|
@@ -0,0 +1,175 @@
|
|
|
1
|
+
# 00-cut-0.2 — The First Functional Library
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Ship `kotoshu-0.2.0` — the first release where `gem install kotoshu`
|
|
6
|
+
works on any Ruby 3.1+ system with **no system dictionary dependency**.
|
|
7
|
+
One call, `Kotoshu.check("Hello wrold")`, auto-downloads the English
|
|
8
|
+
Hunspell dictionary on first use and returns correct results. The CLI
|
|
9
|
+
works end-to-end with proper exit codes.
|
|
10
|
+
|
|
11
|
+
0.1.0 proved the API shape. 0.2.0 delivers the promise: "just works,
|
|
12
|
+
no setup." For 0.2.0 the resolution machinery is English-only; the
|
|
13
|
+
code is language-agnostic so adding more languages is a content task
|
|
14
|
+
(shipping dictionaries in `kotoshu/dictionaries`), not a code change.
|
|
15
|
+
|
|
16
|
+
## 0.2.0 scope (locked)
|
|
17
|
+
|
|
18
|
+
| Capability | Source plan |
|
|
19
|
+
|---|---|
|
|
20
|
+
| `ResourceManager.resolve` (spelling + frequency) | `03` tasks 1–2 |
|
|
21
|
+
| `ResourceBundle` value object | `03` task 1 |
|
|
22
|
+
| en_US Hunspell auto-download on first use | `03` task 2 |
|
|
23
|
+
| Spellchecker accepts `ResourceBundle` (Strangler-Fig) | `03` |
|
|
24
|
+
| One canonical `CheckCommand` wired into CLI | `02` task 1 |
|
|
25
|
+
| Exit codes 0/1/2/3 | `02` task 6 |
|
|
26
|
+
| `--language en\|auto`, `--format text\|json` | `02` task 5 |
|
|
27
|
+
| en_US Hunspell fixtures at 100 % | `01` tasks 1–2 |
|
|
28
|
+
| Offline mode (`offline: true`, `KOTOSHU_OFFLINE=1`) | `03` task 4 |
|
|
29
|
+
| Fake-server integration spec | `03` task 9 |
|
|
30
|
+
|
|
31
|
+
Explicitly **deferred** to 0.3+:
|
|
32
|
+
|
|
33
|
+
| Capability | Target | Plan |
|
|
34
|
+
|---|---|---|
|
|
35
|
+
| Checksum / integrity verification | 0.3 | `09-integrity-security` |
|
|
36
|
+
| `--interactive`, `--format sarif` | 0.3 | `02` tasks 4, 5 |
|
|
37
|
+
| `kotoshu fetch` pre-warm command | 0.3 | `03` task 6 |
|
|
38
|
+
| Strict mode | 0.3 | `03` task 5 |
|
|
39
|
+
| ONNX model auto-download (semantic rerank) | 0.4 | `05-semantic-path` |
|
|
40
|
+
| ≥30 language modules wired | 0.6 | `04-language-modules` |
|
|
41
|
+
| Grammar rule packs | 0.7 | `08-grammar-engine` |
|
|
42
|
+
| CJK (Japanese, Chinese) | 0.8 | `06-cjk-support` |
|
|
43
|
+
| RTL (Arabic, Hebrew, Persian, Urdu) | 0.9 | `07-rtl-support` |
|
|
44
|
+
| Hunspell MECE 4-layer refactor | 1.0 | `01` task 6 |
|
|
45
|
+
|
|
46
|
+
## Critical path — three phases
|
|
47
|
+
|
|
48
|
+
### Phase A — ResourceManager (foundation)
|
|
49
|
+
|
|
50
|
+
The Spellchecker today takes a single `Dictionary::Base` the user must
|
|
51
|
+
configure manually. Phase A introduces `ResourceManager` as the thing
|
|
52
|
+
that produces that dictionary (plus frequency data) from a language
|
|
53
|
+
code, using the existing `LanguageCache`.
|
|
54
|
+
|
|
55
|
+
1. **`ResourceBundle`** (`lib/kotoshu/resource_bundle.rb`). Plain
|
|
56
|
+
value object — Struct or class with `language`, `dictionary`,
|
|
57
|
+
`frequency`, `model` (nil for 0.2), `rules` (nil for 0.2),
|
|
58
|
+
`cached`, `source_urls`. No hand-rolled `to_h` (project rule).
|
|
59
|
+
2. **`ResourceManager`** (`lib/kotoshu/resource_manager.rb`,
|
|
60
|
+
autoloaded from `lib/kotoshu.rb`). API:
|
|
61
|
+
```ruby
|
|
62
|
+
Kotoshu::ResourceManager.resolve(
|
|
63
|
+
text: "Hello wrold",
|
|
64
|
+
language: :auto, # or "en", "de", nil
|
|
65
|
+
want: %i[spelling frequency],
|
|
66
|
+
offline: false
|
|
67
|
+
) # => ResourceBundle
|
|
68
|
+
```
|
|
69
|
+
Internals: detect language via `Language::Identifier` when `:auto`;
|
|
70
|
+
fetch dictionary via `LanguageCache.get(language)`; fetch frequency
|
|
71
|
+
via `FrequencyCache.get(language)` when wanted; construct
|
|
72
|
+
`ResourceBundle`. No new network code — the caches already
|
|
73
|
+
download from `kotoshu/dictionaries` and `kotoshu/frequency-list-kelly`.
|
|
74
|
+
3. **Spellchecker accepts `ResourceBundle`.** Modify
|
|
75
|
+
`lib/kotoshu/spellchecker.rb` to accept either a `ResourceBundle`
|
|
76
|
+
(new path) or a `Dictionary::Base` (legacy path). Strangler-Fig —
|
|
77
|
+
don't break the 0.1 API. Extract `dictionary` and `frequency` from
|
|
78
|
+
the bundle; ignore `model`/`rules` (nil for 0.2).
|
|
79
|
+
4. **First-use UX.** When `LanguageCache` downloads, print
|
|
80
|
+
`[en] downloading dictionary (X.X MB)…` to STDERR. One line per
|
|
81
|
+
resource, not per byte. Suppress when `$stdout.tty?` is false and
|
|
82
|
+
`KOTOSHU_QUIET` is set.
|
|
83
|
+
5. **Configuration.** Add to `Configuration::SCHEMA`:
|
|
84
|
+
- `offline` (boolean, default `false`) — also via `KOTOSHU_OFFLINE=1`
|
|
85
|
+
- `default_language` (string, default `"en"`) — used when
|
|
86
|
+
`language: nil` and detection is inconclusive
|
|
87
|
+
6. **Offline mode.** `ResourceManager.resolve(..., offline: true)`
|
|
88
|
+
only uses cached resources; raises `ResourceNotCachedError` with a
|
|
89
|
+
message naming what's missing and how to pre-fetch.
|
|
90
|
+
|
|
91
|
+
**Phase A acceptance:** `Kotoshu.correct?("hello")` works on a Docker
|
|
92
|
+
`ruby:3.3` container with no `/usr/share/dict/words`, auto-downloading
|
|
93
|
+
en_US on first call.
|
|
94
|
+
|
|
95
|
+
### Phase B — English correctness
|
|
96
|
+
|
|
97
|
+
7. **Audit en_US fixtures.** Run
|
|
98
|
+
`bundle exec rspec spec/hunspell_tests/ --tag lang:en` (or filter
|
|
99
|
+
by fixture path). Record which examples fail and categorize:
|
|
100
|
+
compound words, capitalization, affix combinations, special
|
|
101
|
+
characters. This is `01` task 1 scoped to en_US only.
|
|
102
|
+
8. **Fix en_US failures.** Target the categories from the audit. The
|
|
103
|
+
full MECE 4-layer refactor (`01` task 6) is explicitly **not**
|
|
104
|
+
required for 0.2 — patch the procedural `algorithms/lookup.rb`
|
|
105
|
+
where needed. Each fix gets a named spec example.
|
|
106
|
+
9. **Suggestion sanity.** Verify `Kotoshu.suggest("helo").first.word`
|
|
107
|
+
== `"hello"` for the top-20 common misspellings. Tune
|
|
108
|
+
`EditDistanceStrategy` weight or `frequency_bonus` if the top
|
|
109
|
+
suggestion is wrong.
|
|
110
|
+
|
|
111
|
+
**Phase B acceptance:** 100 % of en_US Hunspell fixtures pass.
|
|
112
|
+
`suggest("helo")[0]` == `"hello"`.
|
|
113
|
+
|
|
114
|
+
### Phase C — CLI wired
|
|
115
|
+
|
|
116
|
+
10. **Wire `CheckCommand` as canonical `check`.** Remove the basic
|
|
117
|
+
`check` from `lib/kotoshu/cli.rb`; register
|
|
118
|
+
`Kotoshu::CheckCommand` (from `commands/check_command.rb`) via
|
|
119
|
+
`subcommand "check", Kotoshu::CheckCommand`. This is `02` task 1.
|
|
120
|
+
11. **Exit codes** (`02` task 6):
|
|
121
|
+
- `0` — no errors found
|
|
122
|
+
- `1` — errors found
|
|
123
|
+
- `2` — usage error
|
|
124
|
+
- `3` — dictionary fetch failure (network down, not cached + offline)
|
|
125
|
+
12. **Flag reconciliation** (`02` task 5). For 0.2, support exactly:
|
|
126
|
+
- `kotoshu check FILE` — default check, text output
|
|
127
|
+
- `--language en|de|...|auto` — language selection (`auto` =
|
|
128
|
+
detect via `Language::Identifier`)
|
|
129
|
+
- `--format text|json` — output format (sarif deferred to 0.3)
|
|
130
|
+
- `--offline` — use cached resources only
|
|
131
|
+
Remove or hide every other flag the README mentions but code
|
|
132
|
+
doesn't support yet. Update README to match.
|
|
133
|
+
13. **End-to-end CLI spec.** Under `spec/kotoshu/cli/`, using a temp
|
|
134
|
+
dir (not real `~/.kotoshu`): write a file with known misspellings,
|
|
135
|
+
run `kotoshu check`, assert exit code + output content for both
|
|
136
|
+
`text` and `json` formats.
|
|
137
|
+
|
|
138
|
+
**Phase C acceptance:** `kotoshu check README.adoc` exits 1 with
|
|
139
|
+
readable error output; `kotoshu check clean_file.txt` exits 0.
|
|
140
|
+
|
|
141
|
+
## Release-level acceptance criteria
|
|
142
|
+
|
|
143
|
+
- Fresh `docker run --rm ruby:3.3` → `gem install kotoshu` →
|
|
144
|
+
`ruby -e "require 'kotoshu'; puts Kotoshu.correct?('hello')"`
|
|
145
|
+
prints `true` (auto-downloads en_US on first call)
|
|
146
|
+
- `Kotoshu.correct?("helo")` returns `false`
|
|
147
|
+
- `Kotoshu.suggest("helo").first.word` == `"hello"`
|
|
148
|
+
- `kotoshu check README.adoc` exits 1, reports errors with suggestions
|
|
149
|
+
- `kotoshu check clean.txt` exits 0
|
|
150
|
+
- en_US Hunspell fixtures: 100 % pass
|
|
151
|
+
- No system dictionary dependency (no `/usr/share/dict/words` read)
|
|
152
|
+
- `Kotoshu.check(text, offline: true)` raises clear error if not cached
|
|
153
|
+
- `ENV['KOTOSHU_OFFLINE']` == `"1"` respected throughout
|
|
154
|
+
- README accurately reflects 0.2 capabilities (no false claims)
|
|
155
|
+
- `bundle exec rspec` green; `bundle exec rubocop` clean
|
|
156
|
+
|
|
157
|
+
## Dependencies
|
|
158
|
+
|
|
159
|
+
- **Blocked by:** nothing — 0.1.0 is shipped.
|
|
160
|
+
- **Blocks:** `04-language-modules` (needs `ResourceManager`),
|
|
161
|
+
`05-semantic-path` (needs `ResourceManager` for model resolution).
|
|
162
|
+
|
|
163
|
+
## Out of scope for 0.2.0
|
|
164
|
+
|
|
165
|
+
Everything in the "deferred" table above. Additionally:
|
|
166
|
+
- The Hunspell MECE 4-layer refactor (`01` task 6) — patch, don't
|
|
167
|
+
rewrite, for 0.2.
|
|
168
|
+
- `DictCommand` / `CacheCommand` relocation to `commands/` (`02`
|
|
169
|
+
task 2) — cosmetic, defer.
|
|
170
|
+
- `model` subcommand (`02` task 3) — needs ONNX, defer to 0.4.
|
|
171
|
+
- `--from-stdin` (`02` task 4) — nice-to-have, defer to 0.3.
|
|
172
|
+
|
|
173
|
+
## Status
|
|
174
|
+
|
|
175
|
+
Starting now. Phase A (`ResourceManager`) is the first task.
|