kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,251 @@
|
|
|
1
|
+
# The 1.0 Readiness Audit
|
|
2
|
+
|
|
3
|
+
Plan 109. Analysis and documentation only — no behavior change, no version
|
|
4
|
+
bump, no release. The version cut itself is an owner decision.
|
|
5
|
+
|
|
6
|
+
- Audited at: gem 0.10.0 (`lib/kotoshu/version.rb:4`), main @ 73b7fac.
|
|
7
|
+
- Companion policy draft: `docs/STABILITY-1.0.md`.
|
|
8
|
+
- Method: every inventory line cites the file that defines the surface.
|
|
9
|
+
Reachability was verified by grep, not by assumption: a class being
|
|
10
|
+
present under `lib/` is not the same as being wired to `exe/kotoshu`.
|
|
11
|
+
|
|
12
|
+
## Inventory
|
|
13
|
+
|
|
14
|
+
Verdicts: **stable** (freeze at 1.0), **deprecate-before-1.0** (needs a
|
|
15
|
+
deprecation cycle first), **accidentally-public** (internal or dead, 1.0
|
|
16
|
+
should privatize or delete).
|
|
17
|
+
|
|
18
|
+
### Ruby gem — module functions (`lib/kotoshu.rb`)
|
|
19
|
+
|
|
20
|
+
| Surface | Where defined | Verdict | Note |
|
|
21
|
+
|---|---|---|---|
|
|
22
|
+
| `configure` / `configuration` / `configuration=` | `lib/kotoshu.rb:98,109,127` | stable | Type-checked global swap (127-131). |
|
|
23
|
+
| `spellchecker` / `spellchecker_for` | `lib/kotoshu.rb:139,157` | stable | Cache-only; README-documented (README.adoc:18). |
|
|
24
|
+
| `resolve` | `lib/kotoshu.rb:179` | stable | Stage-2 resolve; README.adoc:364. |
|
|
25
|
+
| `setup` / `setup?` / `languages_setup` | `lib/kotoshu.rb:220,245,257` | stable | Stage-1 entry; idempotent; `want:`/`tier:`/local-source opts frozen. |
|
|
26
|
+
| `reset_spellchecker` | `lib/kotoshu.rb:268` | stable | Test/embedding seam. |
|
|
27
|
+
| `correct?` / `misspelled?` / `suggest` / `check` / `check_file` / `check_files` | `lib/kotoshu.rb:286,297,313,331,349,363` | stable | The hot path; strict two-stage contract is the product identity. |
|
|
28
|
+
| `detect_language` / `detect_language_with_confidence` / `setup_lid` / `setup_lid?` | `lib/kotoshu.rb:465,481,501,508` | stable | Native LID with heuristic fallback (plan 106). |
|
|
29
|
+
| `get_language` / `language_registered?` / `supported_languages` / `language` | `lib/kotoshu.rb:519,530,540,443` | stable | `language` is a trivial namespace accessor; keep. |
|
|
30
|
+
| `dictionary` / `trie` / `suggestion_pipeline` | `lib/kotoshu.rb:371,388,409` | stable | Convenience factories; exercised by `examples/04..06`. |
|
|
31
|
+
| `register_dictionary_type` / `register_suggestion_algorithm` | `lib/kotoshu.rb:422,433` | stable | The extension points; 1.0 must not narrow them. |
|
|
32
|
+
| `VERSION` | `lib/kotoshu/version.rb:4` | stable | |
|
|
33
|
+
|
|
34
|
+
### Ruby gem — core classes
|
|
35
|
+
|
|
36
|
+
| Surface | Where defined | Verdict | Note |
|
|
37
|
+
|---|---|---|---|
|
|
38
|
+
| `Kotoshu::Spellchecker` public methods: `correct?`, `incorrect?`, `suggest`, `check_word`, `check`, `check_file`, `check_directory`, `tokenize`, `dictionary`, `reload_dictionary` | `lib/kotoshu/spellchecker.rb:118,130,143,164,196,242,265,280,308,315` | stable | Constructor kwargs (`dictionary:`, `config:`, `resource_bundle:`) frozen at 65. |
|
|
39
|
+
| `Kotoshu::Spellchecker::ASCII_WORD_REGEX` | `lib/kotoshu/spellchecker.rb:39` | accidentally-public | Internal fallback word-char set exposed as a constant. Freeze + document, or privatize (see deprecations). |
|
|
40
|
+
| `Kotoshu::Configuration`: `SCHEMA`, `DEFAULTS`, accessors, `get`, `key?`, `load_dictionary`, `reset_dictionary`, `source_registry`, `self.default/instance/instance=/reset` | `lib/kotoshu/configuration.rb:38,222,423,431,447,474,517,541,565,576,585` | stable | ENV contract (`KOTOSHU_*`) rides on SCHEMA — every key there is public surface. |
|
|
41
|
+
| `Configuration#dictionaries_url` / `#models_url` | `lib/kotoshu/configuration.rb:63-74` (SCHEMA), `:318-321` (attrs) | deprecate-before-1.0 | SCHEMA descriptions already say "Deprecated: use repos_base_url + pins via SourceRegistry". |
|
|
42
|
+
| `ResourceManager.setup/resolve/setup?/languages_setup`, `DEFAULT_WANT` | `lib/kotoshu/resource_manager.rb:69,97,21` | stable | Facade delegates here; two-stage model frozen. |
|
|
43
|
+
| `ResourceBundle` (`dictionary`/`frequency`/`model`/`rules`) | `lib/kotoshu/resource_bundle.rb` | stable | Consumed by `Kotoshu.resolve` (kotoshu.rb:179). |
|
|
44
|
+
| Exception hierarchy: `Error` + `DictionaryNotFoundError`, `InvalidDictionaryFormatError`, `ConfigurationError`, `SpellcheckError`, `AffixRuleError`, `ResourceNotCachedError`, `ResourceNotSetupError`, `ResourceResolutionError`, `IntegrityError`, `SuikaUnavailable` | `lib/kotoshu/core/exceptions.rb:8,14,32,54,72,90,106,122,136,156,177`; autoloaded `lib/kotoshu.rb:65-75` | stable | Message text is not frozen; class names are. |
|
|
45
|
+
| `Kotoshu::Paths` (XDG + `KOTOSHU_{CACHE,CONFIG,DATA}_PATH`) | `lib/kotoshu/paths.rb` | stable | On-disk layout is user-visible contract. |
|
|
46
|
+
| Autoloaded namespaces behind `Kotoshu::*` | `lib/kotoshu.rb:14-62` | stable (as namespaces) | Namespaces stay; member-by-member stability is YARD-documented per class. 1.0 freezes namespace existence, not every leaf. |
|
|
47
|
+
| Top-level singleton aliases `Kotoshu::LanguageCache`, `Kotoshu::LanguageIdentifier`, `Kotoshu::ModelCache`, `Kotoshu::SemanticAnalyzer` | `lib/kotoshu.rb:80,81,84,85` | accidentally-public | Zero hits in lib/, spec/, README; canonical homes are `Kotoshu::Cache::LanguageCache`, `Kotoshu::Language::*`, `Kotoshu::Cache::ModelCache`, `Kotoshu::Analyzers::SemanticAnalyzer`. |
|
|
48
|
+
| `Kotoshu::Debug`, `DebugLogger`, `Metrics`, `MetricsCollector` | `lib/kotoshu.rb:78-83`; `lib/kotoshu/debug_mode.rb`, `metrics_module.rb`, `metrics_collector.rb` | stable | Documented in their doc comments; `Kotoshu::Metrics.stats` is README-documented (README.adoc:211). |
|
|
49
|
+
| `Kotoshu::FluentChecker` / `Kotoshu::MultiLanguageChecker` | `lib/kotoshu/fluent_checker.rb`, `lib/kotoshu/multi_language_checker.rb` | deprecate-before-1.0 (or document) | Spec-covered (`spec/kotoshu/{fluent,multi_language}_checker_spec.rb`) but absent from README.adoc — decide: document or deprecate. |
|
|
50
|
+
|
|
51
|
+
### Ruby gem — framework integrations (all soft deps, plan 89)
|
|
52
|
+
|
|
53
|
+
| Surface | Where defined | Verdict | Note |
|
|
54
|
+
|---|---|---|---|
|
|
55
|
+
| `require "kotoshu/validators"` → `Kotoshu::Validators::SpellingValidator` (+ `ACTIVE_MODEL_LOADED`, `ActiveModelUnavailable`) | `lib/kotoshu/validators/spelling_validator.rb:10-99`; autoload `lib/kotoshu.rb:60` | stable | Placeholder class keeps constant resolvable without Rails (91-99). |
|
|
56
|
+
| `require "kotoshu/rspec"` → `Kotoshu::Rspec::Matchers` (`expect_words`, `expect_document`, `all_be_spelled_correctly`, `be_spelled_correctly(in:)`) | `lib/kotoshu/rspec.rb:26-160`; autoload `lib/kotoshu.rb:61` | stable | Matcher names are the contract. |
|
|
57
|
+
| `require "kotoshu/tasks"` → default `rake kotoshu:check`; `Kotoshu::Tasks::CheckTask` | `lib/kotoshu/tasks.rb:11`, `lib/kotoshu/tasks/check_task.rb:22` | stable | require-only (defining a task on load is the point — kotoshu.rb:56-59). |
|
|
58
|
+
| `require "kotoshu/jekyll"` → `Kotoshu::Jekyll::Generator` (+ `JEKYLL_LOADED`, `JekyllUnavailable`) | `lib/kotoshu/jekyll.rb:9-96`; autoload `lib/kotoshu.rb:62` | stable | Baseline honored via `Baseline::Store::DEFAULT_FILENAME`. |
|
|
59
|
+
|
|
60
|
+
### Ruby gem — conformance surface (the behavioral contract)
|
|
61
|
+
|
|
62
|
+
| Surface | Where defined | Verdict | Note |
|
|
63
|
+
|---|---|---|---|
|
|
64
|
+
| `conformance/vectors.jsonl` (2630 rows, committed) | `conformance/vectors.jsonl`; sync contract documented in `lib/kotoshu/conformance_exporter.rb:54-60` | stable — frozen expectations | `expected` records what the Ruby engine ACTUALLY returns, not linguistic truth (conformance_exporter.rb:31-37). |
|
|
65
|
+
| Row shape v0: `kind/language/dictionary/input/limit/expected`; suggestion keys `word,distance,confidence,source` | `lib/kotoshu/conformance_exporter.rb:16-29,80` | stable | Four-key shape shared with wasm and `ffi::ruby`. |
|
|
66
|
+
| `Kotoshu::ConformanceRunner` (`run`, `compare`, rake tasks) | `lib/kotoshu/conformance_runner.rb:26,72,97` | stable | Guards Ruby-vs-vectors and native-vs-Ruby equality. |
|
|
67
|
+
| `Kotoshu::ConformanceExporter` (`export`, corpora selection, `EXCLUDED_FIXTURES`) | `lib/kotoshu/conformance_exporter.rb:61-254` | stable | Deterministic re-export (44-53). |
|
|
68
|
+
|
|
69
|
+
### CLI (`exe/kotoshu` → `Kotoshu::Cli::Cli`)
|
|
70
|
+
|
|
71
|
+
| Surface | Where defined | Verdict | Note |
|
|
72
|
+
|---|---|---|---|
|
|
73
|
+
| Commands: `check`, `setup`, `dict`, `cache`, `personal`, `completions`, `baseline`, `status`, `version` (`--version`/`-V`) | `lib/kotoshu/cli.rb:178,241,317-329,349,358-367` | stable | |
|
|
74
|
+
| Class options `--language/-l`, `--format/-f` (text,json,sarif), `--interactive/-i`, `--verbose/-v` | `lib/kotoshu/cli.rb:102-125` | stable | |
|
|
75
|
+
| `check` options `--baseline`, `--show-suppressed`, `--include`, `--exclude`, `--personal` | `lib/kotoshu/cli.rb:151-168` | stable | |
|
|
76
|
+
| `setup` options `--aff --dic --from --frequency --want --model --tier --force --strict --list` | `lib/kotoshu/cli.rb:214-240` | stable | |
|
|
77
|
+
| Exit codes 0/1/2/3 | `lib/kotoshu/cli.rb:92-97,145-149` | stable | Consumed by action-kotoshu (action.yml:178-179). |
|
|
78
|
+
| JSON output top-level keys; SARIF 2.1.0 shape | `lib/kotoshu/cli.rb:624-651,686-715` | stable | Machine-readable contract; additive keys only. |
|
|
79
|
+
| `cache` subcommands `list/info/clean/evict/download/purge/validate` | `lib/kotoshu/cli/cache_command.rb:27,50,79,101,133,166,184` | stable | |
|
|
80
|
+
| `personal` subcommands `add/remove/list/import/path/clear` | `lib/kotoshu/cli/personal_command.rb:28-89` | stable | |
|
|
81
|
+
| `baseline init` (`--output`, `--language`) | `lib/kotoshu/cli/baseline_command.rb:15-36` | stable | |
|
|
82
|
+
| `completions bash/zsh` | `lib/kotoshu/cli/completions_command.rb:13-26` | stable | Catalog is hand-maintained — must track command changes. |
|
|
83
|
+
| `fetch` (hidden alias of `setup`) | `lib/kotoshu/cli.rb:280-315` | deprecate-before-1.0 | Already hidden and labeled deprecated; remove at 1.0. |
|
|
84
|
+
|
|
85
|
+
### Ruby gem — accidentally-public legacy commands (the hazard block)
|
|
86
|
+
|
|
87
|
+
| Surface | Where defined | Verdict | Note |
|
|
88
|
+
|---|---|---|---|
|
|
89
|
+
| `::DictCommand` (top-level constant!) | `lib/kotoshu/cli.rb:10` | accidentally-public | Defined on `Object`, outside `module Kotoshu`. Pollutes the global namespace of every process that requires the gem; collides with any other gem defining `DictCommand`. Move under `Kotoshu::Cli` before 1.0. |
|
|
90
|
+
| `Kotoshu::Commands::CheckCommand` | `lib/kotoshu/commands/check_command.rb:4`; autoload `lib/kotoshu/commands.rb:9` | accidentally-public | Not wired into `Cli` (no `subcommand "check"` at cli.rb:317-329 — `check` is a method on `Cli` itself, cli.rb:178). Half-placeholder implementation; its hunspell path calls `SpellChecker.new` (check_command.rb:198) which resolves to nonexistent `Kotoshu::SpellChecker` (the real class is `Kotoshu::Spellchecker`) → `NameError` at runtime. Declares flags that do not exist on the real CLI (`--model`, `-o`, `csv/yaml` formats), contradicting cli.rb:108-113. Delete or privatize. |
|
|
91
|
+
| `Kotoshu::CacheCommand` | `lib/kotoshu/commands/cache_command.rb:4` | accidentally-public | Dead duplicate of the wired `Kotoshu::Cli::CacheCommand` (autoload cli.rb:70; subcommand cli.rb:321). Nothing requires this file (verified by grep); acknowledged as historical in `lib/kotoshu/commands.rb:7-8`. Still shipped in the gem (`spec.files` = `git ls-files`, kotoshu.gemspec:30-38). Delete. |
|
|
92
|
+
| `Kotoshu::Commands::ModelCommand` | `lib/kotoshu/commands/model_command.rb:4`; autoload `lib/kotoshu/commands.rb:10` | accidentally-public | No `kotoshu model` subcommand is registered anywhere. `convert` shells out to `scripts/convert_fasttext_to_onnx.py` (model_command.rb:33-36) — untracked in git, therefore NOT shipped in the gem, so the command cannot work as packaged. Delete. |
|
|
93
|
+
|
|
94
|
+
### Ruby gem — native extension
|
|
95
|
+
|
|
96
|
+
| Surface | Where defined | Verdict | Note |
|
|
97
|
+
|---|---|---|---|
|
|
98
|
+
| Optional Rust ext build (`extconf.rb`) | `kotoshu.gemspec:53`; `ext/kotoshu_native/extconf.rb` | stable | Soft: pure-Ruby is the complete fallback (gemspec:44-52). |
|
|
99
|
+
| `Kotoshu::Native.available?`, `Native.suggestion(row)`, `Native::Unavailable`, `Native::EXTENSION` | `lib/kotoshu/native.rb:49,65,35,39` | stable | Extension defines `Native::VERSION`, `Native::Error`, `Native::Dictionary.load/correct?/suggest` (native.rb:9-14). |
|
|
100
|
+
| `Kotoshu::NativeBackend.resolve` boundary (ruby/auto/native) | `lib/kotoshu/native_backend.rb:45-73` | stable | Accelerator-only boundary is documented and frozen. |
|
|
101
|
+
|
|
102
|
+
### wasm package (kotoshu-rs)
|
|
103
|
+
|
|
104
|
+
| Surface | Where defined | Verdict | Note |
|
|
105
|
+
|---|---|---|---|
|
|
106
|
+
| `KotoshuWasm` constructor `(affSource, dicSource)`, `correct(word)`, `suggest(word, limit?)`, static `VERSION` | `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:165-174,178,185,152-155` | stable | Contents-not-paths loading; errors reject with the Rust message. |
|
|
107
|
+
| `loadModel(modelBytes, vocabBytes, bucketsBytes?)` → `KotoshuModel`; `.free()` | mod.rs:220-235,207-209 | stable | Optional third arg added by plan 103 — additive pattern to keep. |
|
|
108
|
+
| `rerank(model, word, context)` → number in [-1,1] | mod.rs:241-244 | stable | Mean-cosine contract documented (mod.rs:73-82). |
|
|
109
|
+
| `semanticSuggest(model, word, k?)` → `{word, score}[]` | mod.rs:252-266 | stable | |
|
|
110
|
+
| `loadLid(onnxBytes, vocabBytes)` → `KotoshuLid`; `detectLanguage(lid, text)` → `{code, score}` | mod.rs:281-287,293-300 | stable | Parity contract with the gem documented (mod.rs:84-94). |
|
|
111
|
+
| Suggestion row shape: exactly `word/distance/confidence/source` | mod.rs:48-54,190-198 | stable — frozen | Same four keys as the conformance vectors (`conformance_exporter.rb:80`). |
|
|
112
|
+
| npm identity | `kotoshu-rs/kotoshu-wasm/pkg/package.json:2,5` | needs a decision | Package name is `kotoshu-wasm` but every doc says `@kotoshu/wasm` (mod.rs:2-3); publication still blocked on npm org credentials. Decide the final name before 1.0 — npm names are sticky. |
|
|
113
|
+
|
|
114
|
+
### HTTP server (kotoshu-server)
|
|
115
|
+
|
|
116
|
+
| Surface | Where defined | Verdict | Note |
|
|
117
|
+
|---|---|---|---|
|
|
118
|
+
| `GET /` (metadata), `GET /v1/health`, `GET /v1/version`, `GET /v1/languages` | `kotoshu-server/openapi.yaml:15,25,35,45`; routes `lib/kotoshu/server/app.rb:290,300,306,311` | stable | |
|
|
119
|
+
| `POST /v1/check` (`text`, `language`, `format`, `model`), `POST /v1/suggest` (`word`, `language`, `max` 1-50), `POST /v1/detect` (`text`) | openapi.yaml:55-117,179-203; routes app.rb:318,350,366 | stable | Errors: 400/422/503 shapes (openapi.yaml:118-137). |
|
|
120
|
+
| Response schemas `DocumentResult/WordError/Suggestions/Suggestion/Detection` (incl. `engine: lid-176|heuristic`) | openapi.yaml:204-245 | stable | openapi.yaml is the contract source of truth; app.rb matches it 1:1 (verified). |
|
|
121
|
+
| `/v1` prefix as the contract version | openapi.yaml:25-100 | stable | Breaking changes require `/v2`, not field mutations. |
|
|
122
|
+
|
|
123
|
+
### GitHub action (action-kotoshu)
|
|
124
|
+
|
|
125
|
+
| Surface | Where defined | Verdict | Note |
|
|
126
|
+
|---|---|---|---|
|
|
127
|
+
| Inputs: `files, language, format, baseline, include, exclude, show_suppressed, fail_on_error, category, version, prewarm_languages, offline, output_path` | `action-kotoshu/action.yml:8-59` | stable — additive-only | New inputs must default to current behavior. |
|
|
128
|
+
| Outputs: `sarif-path`, `error-count` | action.yml:61-67 | stable | |
|
|
129
|
+
| Gem floor `>= 0.8.0` (v2 of the action) | action.yml:45,98-105 | stable | Floor bumps are breaking for the action and need a major action version. |
|
|
130
|
+
|
|
131
|
+
### Registry schema (models-fasttext-onnx)
|
|
132
|
+
|
|
133
|
+
| Surface | Where defined | Verdict | Note |
|
|
134
|
+
|---|---|---|---|
|
|
135
|
+
| `spec: "kotoshu.resources/v1"` (const) | `models-fasttext-onnx/schemas/registry.schema.json:15-17` | stable — frozen | Change requires a new spec string, never an edit. |
|
|
136
|
+
| Resource id pattern `kotoshu://models/(lid/lid-176|{lang}/(full|fluency|mini|buckets))` | registry.schema.json:35 | stable | Ids are never reused; content changes bump `release_tag` (schema description, registry.schema.json:5). |
|
|
137
|
+
| Resource object fields incl. `min_engine_version`, `sha256`, `license` | registry.schema.json:43-165 | stable | `additionalProperties: false` — additive fields need a schema change = coordinated release. |
|
|
138
|
+
|
|
139
|
+
## Deprecation list for the 1.0 cut
|
|
140
|
+
|
|
141
|
+
Concrete, with defining locations. All need small code PRs — deliberately
|
|
142
|
+
NOT implemented by this docs-only change.
|
|
143
|
+
|
|
144
|
+
1. `::DictCommand` — `lib/kotoshu/cli.rb:10`. Global-namespace leak from a
|
|
145
|
+
file that otherwise lives in `Kotoshu::Cli`. Rename to
|
|
146
|
+
`Kotoshu::Cli::DictCommand` (internal; `subcommand "dict"` wiring at
|
|
147
|
+
cli.rb:317-318 is the only consumer). Highest-priority item: it can
|
|
148
|
+
break other gems in the same process, which is exactly the class of
|
|
149
|
+
thing 1.0 exists to prevent.
|
|
150
|
+
Handled in this PR: the class moved to
|
|
151
|
+
`lib/kotoshu/cli/dict_command.rb` as `Kotoshu::Cli::DictCommand`
|
|
152
|
+
(autoloaded like its sibling subcommands); `::DictCommand` is gone.
|
|
153
|
+
2. `Kotoshu::Commands::CheckCommand` — `lib/kotoshu/commands/check_command.rb:4`
|
|
154
|
+
(autoload `lib/kotoshu/commands.rb:9`). Unwired, placeholder-laden, and
|
|
155
|
+
broken at check_command.rb:198 (`SpellChecker` → `NameError`; real class
|
|
156
|
+
is `Kotoshu::Spellchecker`, spellchecker.rb:21). Its flag surface
|
|
157
|
+
(`--model`, `-o/--output`, `csv|yaml` formats) contradicts the real CLI
|
|
158
|
+
and will mislead users who find it via constants. Delete; keep
|
|
159
|
+
`lib/kotoshu/commands.rb` as the namespace or remove both.
|
|
160
|
+
Handled in this PR: deleted, along with the now-empty `Commands`
|
|
161
|
+
namespace (file and autoload). No spec referenced it.
|
|
162
|
+
3. `Kotoshu::CacheCommand` — `lib/kotoshu/commands/cache_command.rb:4`.
|
|
163
|
+
Dead duplicate of `Kotoshu::Cli::CacheCommand`; unreachable by any
|
|
164
|
+
require in the repo; still shipped by the gemspec file list. Delete.
|
|
165
|
+
Handled in this PR: deleted. No spec referenced it.
|
|
166
|
+
4. `Kotoshu::Commands::ModelCommand` — `lib/kotoshu/commands/model_command.rb:4`
|
|
167
|
+
(autoload `lib/kotoshu/commands.rb:10`). No `model` subcommand exists;
|
|
168
|
+
`convert` depends on an untracked Python script not shipped in the gem
|
|
169
|
+
(model_command.rb:33-36 vs gemspec:30-38). Delete.
|
|
170
|
+
Handled in this PR: deleted. No spec referenced it.
|
|
171
|
+
5. Alias constants `Kotoshu::LanguageCache`, `Kotoshu::LanguageIdentifier`,
|
|
172
|
+
`Kotoshu::ModelCache`, `Kotoshu::SemanticAnalyzer` — `lib/kotoshu.rb:80,81,84,85`.
|
|
173
|
+
No usage anywhere in lib/, spec/, or README. Add a deprecation `warn`
|
|
174
|
+
in a 0.x minor and drop the autoloads at 1.0 (canonical names stay).
|
|
175
|
+
Handled in this PR: grep confirmed zero references to the aliases
|
|
176
|
+
anywhere (lib, spec, exe, docs, README, sig) — the short-name hits all
|
|
177
|
+
resolve to the canonical constants lexically — so the autoloads were
|
|
178
|
+
dropped straight away, ahead of the suggested warn cycle. Canonical
|
|
179
|
+
names are untouched.
|
|
180
|
+
6. CLI `fetch` — `lib/kotoshu/cli.rb:280-315`. Already hidden and marked
|
|
181
|
+
deprecated. Remove at 1.0.
|
|
182
|
+
7. `Configuration#dictionaries_url` / `#models_url` —
|
|
183
|
+
`lib/kotoshu/configuration.rb:63-74,318-321`. Marked deprecated in SCHEMA
|
|
184
|
+
since the SourceRegistry pins landed. Emit a warning on use, remove the
|
|
185
|
+
SCHEMA entries at 1.0 (ENV vars `KOTOSHU_DICTIONARIES_URL` /
|
|
186
|
+
`KOTOSHU_MODELS_URL` go with them — call that out in the changelog).
|
|
187
|
+
8. `Kotoshu::Spellchecker::ASCII_WORD_REGEX` — `lib/kotoshu/spellchecker.rb:39`.
|
|
188
|
+
Internal fallback set posing as API. Either document + freeze or make it
|
|
189
|
+
private before 1.0.
|
|
190
|
+
Handled in this PR: zero external references confirmed by grep, so the
|
|
191
|
+
constant was deleted and its literal inlined at the single internal use
|
|
192
|
+
site (constructor fallback). Behavior unchanged.
|
|
193
|
+
9. `Kotoshu::FluentChecker` / `Kotoshu::MultiLanguageChecker` —
|
|
194
|
+
`lib/kotoshu/fluent_checker.rb`, `lib/kotoshu/multi_language_checker.rb`.
|
|
195
|
+
Spec-covered but undocumented in README.adoc. Decide per class: document
|
|
196
|
+
as stable, or deprecate. Do not freeze silently.
|
|
197
|
+
Handled in this PR: both class docstrings now state "Experimental and
|
|
198
|
+
not public API: outside the 1.0 stability freeze, may change or be
|
|
199
|
+
removed without a deprecation cycle." Specs stay.
|
|
200
|
+
10. wasm npm identity — `kotoshu-rs/kotoshu-wasm/pkg/package.json:2` says
|
|
201
|
+
`kotoshu-wasm` while `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:2-3` and
|
|
202
|
+
all docs say `@kotoshu/wasm`. Resolve before first publication; the
|
|
203
|
+
audit treats the documented scoped name as intended.
|
|
204
|
+
Handled in this PR: verified, no code change needed.
|
|
205
|
+
`kotoshu-rs/scripts/wasm_build.sh` rewrites the generated
|
|
206
|
+
`package.json` name to `@kotoshu/wasm` on every build (the source
|
|
207
|
+
manifest name is a wasm-pack crate-name artifact; `pkg/` is a
|
|
208
|
+
gitignored build output). The shipped identity is already
|
|
209
|
+
`@kotoshu/wasm`; the owner decision that remains is publication
|
|
210
|
+
credentials, not the name.
|
|
211
|
+
|
|
212
|
+
## 1.0 checklist
|
|
213
|
+
|
|
214
|
+
Known blockers — none are mechanical code fixes; all are owner decisions
|
|
215
|
+
or ecosystem plumbing, stated honestly:
|
|
216
|
+
|
|
217
|
+
1. Dispositions for the ten deprecation items above (small code PRs; this
|
|
218
|
+
audit only records them).
|
|
219
|
+
2. npm org + publish credentials for `@kotoshu/wasm` (and the name
|
|
220
|
+
decision, item 10). The wasm build exists; publication is blocked.
|
|
221
|
+
3. The JS SDK PR (kotoshu-js #1) still pending merge — the HTTP contract
|
|
222
|
+
it consumes must be the frozen one.
|
|
223
|
+
4. PyPI token for kotoshu-python publication (same train).
|
|
224
|
+
5. kotoshu-server is at 0.1.0 with an unexplained prewarm hang noted in
|
|
225
|
+
ecosystem state — the 1.0 train should not ship a server cut with a
|
|
226
|
+
known hang.
|
|
227
|
+
6. RBS coverage: `sig/kotoshu.rbs` should be checked against the frozen
|
|
228
|
+
surface so the types do not contradict the freeze.
|
|
229
|
+
7. Conformance vectors green on both engines
|
|
230
|
+
(`rake kotoshu:conformance:compare`) at the cut commit.
|
|
231
|
+
|
|
232
|
+
Release-train shape (gem + wasm + server together, per plan):
|
|
233
|
+
|
|
234
|
+
- Cut order: registry release_tag (if models moved) → gem 1.0 → wasm
|
|
235
|
+
package (independent version line, same day) → server (depends on the
|
|
236
|
+
gem floor) → action (no code change; floor stays 0.8.0-compatible, or
|
|
237
|
+
bump with an action major).
|
|
238
|
+
- The conformance vectors must be byte-identical across the train: they
|
|
239
|
+
are re-exported only when engine behavior intentionally changes, and
|
|
240
|
+
any such change is exactly what the deprecation window is for.
|
|
241
|
+
- Docs: this audit + `docs/STABILITY-1.0.md` land first (this PR); the
|
|
242
|
+
policy takes effect the day 1.0 ships.
|
|
243
|
+
|
|
244
|
+
What stays post-1.0 (policy constants, see the policy draft):
|
|
245
|
+
|
|
246
|
+
- The strict two-stage model — `setup` is never implicit; the hot path
|
|
247
|
+
never downloads.
|
|
248
|
+
- Soft-dependency policy for onnxruntime, suika, activemodel, jekyll.
|
|
249
|
+
- XDG paths + `KOTOSHU_*` envs as documented configuration surface.
|
|
250
|
+
- CLI exit codes 0/1/2/3 and the JSON/SARIF output shapes.
|
|
251
|
+
- `kotoshu.resources/v1` and the `/v1` HTTP prefix.
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
# Kotoshu Stability Policy — 1.0 draft
|
|
2
|
+
|
|
3
|
+
Status: DRAFT (plan 109). Takes effect the day 1.0 ships; until then the
|
|
4
|
+
usual 0.x rules apply (anything may change). The surface this policy
|
|
5
|
+
protects is itemized in `docs/STABILITY-1.0-AUDIT.md`.
|
|
6
|
+
|
|
7
|
+
## Principles
|
|
8
|
+
|
|
9
|
+
1. Semver, per delivery vehicle, cut together on one release train.
|
|
10
|
+
2. Additive changes only, within a versioned boundary (`/v1`,
|
|
11
|
+
`kotoshu.resources/v1`, the four-key suggestion row).
|
|
12
|
+
3. Behavior is frozen where the conformance vectors freeze it — the
|
|
13
|
+
vectors ARE the behavioral contract.
|
|
14
|
+
4. No removal without a deprecation window (below).
|
|
15
|
+
5. The strict two-stage resource model is permanent: `setup` is explicit,
|
|
16
|
+
the hot path never downloads. No feature may weaken it.
|
|
17
|
+
|
|
18
|
+
## Per-surface commitments
|
|
19
|
+
|
|
20
|
+
### Ruby gem (`kotoshu`)
|
|
21
|
+
|
|
22
|
+
- **Patch** (1.0.x): bug fixes only. No public API, CLI, ENV, output, or
|
|
23
|
+
conformance-vector change. `conformance/vectors.jsonl` must replay
|
|
24
|
+
byte-identically (`lib/kotoshu/conformance_runner.rb:72`).
|
|
25
|
+
- **Minor** (1.x): additive only — new methods, new config keys (must
|
|
26
|
+
default to current behavior), new CLI flags with defaults, new
|
|
27
|
+
exception subclasses of existing roots, new suggestion strategies.
|
|
28
|
+
Frozen in place: existing method signatures, the exception class list
|
|
29
|
+
at `lib/kotoshu/core/exceptions.rb`, `SCHEMA` key semantics
|
|
30
|
+
(`lib/kotoshu/configuration.rb:38`), exit codes 0/1/2/3
|
|
31
|
+
(`lib/kotoshu/cli.rb:92-97`), and the JSON/SARIF output shapes
|
|
32
|
+
(`lib/kotoshu/cli.rb:624-651,686-715`) — additions to JSON output are
|
|
33
|
+
allowed (consumers must ignore unknown keys), renames and removals are
|
|
34
|
+
not.
|
|
35
|
+
- **Major** (2.0): anything else, including conformance-vector re-exports
|
|
36
|
+
that change expected results.
|
|
37
|
+
- Ruby floor moves only at a major (`kotoshu.gemspec:17`).
|
|
38
|
+
- Requirements and dependency floors are owner decisions, never inferred
|
|
39
|
+
from code findings.
|
|
40
|
+
|
|
41
|
+
### wasm package (`@kotoshu/wasm` / kotoshu-rs)
|
|
42
|
+
|
|
43
|
+
- Independent version line (its `package.json` governs, not the crate
|
|
44
|
+
constant — `kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:149-155`), released
|
|
45
|
+
on the same train as the gem.
|
|
46
|
+
- The JS surface is the module doc contract:
|
|
47
|
+
`KotoshuWasm` (constructor / `correct` / `suggest` / `VERSION`),
|
|
48
|
+
`loadModel`, `rerank`, `semanticSuggest`, `loadLid`, `detectLanguage`,
|
|
49
|
+
and the `KotoshuModel` / `KotoshuLid` handles
|
|
50
|
+
(`kotoshu-rs/kotoshu/src/ffi/wasm/mod.rs:19-46`).
|
|
51
|
+
- The suggestion row is frozen at exactly `word`, `distance`,
|
|
52
|
+
`confidence`, `source` — the conformance `SUGGESTION_KEYS`
|
|
53
|
+
(`lib/kotoshu/conformance_exporter.rb:80`). Additive optional
|
|
54
|
+
parameters are allowed (the `bucketsBytes` third argument of
|
|
55
|
+
`loadModel` is the precedent, mod.rs:215-219); changing existing
|
|
56
|
+
parameters or return shapes is a major.
|
|
57
|
+
- `KotoshuWasm.VERSION` tracks the crate version; it is informational,
|
|
58
|
+
not the package version.
|
|
59
|
+
|
|
60
|
+
### HTTP server (kotoshu-server)
|
|
61
|
+
|
|
62
|
+
- `openapi.yaml` is the source of truth; `lib/kotoshu/server/app.rb`
|
|
63
|
+
must never diverge from it (today they match route for route).
|
|
64
|
+
- The `/v1` prefix IS the contract version. Within `/v1`: new endpoints,
|
|
65
|
+
new optional request fields, and new optional response fields are
|
|
66
|
+
minor. Changing a field type, removing a field, altering status-code
|
|
67
|
+
semantics (400/422/503), or mutating a frozen schema is a major —
|
|
68
|
+
breaking changes ship as `/v2`, with `/v1` kept for the deprecation
|
|
69
|
+
window.
|
|
70
|
+
- Error bodies keep the `{error, message, hint}` shape
|
|
71
|
+
(`kotoshu-server/openapi.yaml:246-251`).
|
|
72
|
+
|
|
73
|
+
### GitHub action (action-kotoshu)
|
|
74
|
+
|
|
75
|
+
- Inputs are additive-only and every new input defaults to the previous
|
|
76
|
+
behavior (`action-kotoshu/action.yml:8-59`). Outputs
|
|
77
|
+
(`sarif-path`, `error-count`) are frozen.
|
|
78
|
+
- Bumping the gem floor (`version` input minimum, action.yml:98-105) or
|
|
79
|
+
dropping an input is a new action major. Exit-code mapping 0/1/2/3 is
|
|
80
|
+
part of the contract (action.yml:178-189).
|
|
81
|
+
|
|
82
|
+
### Resource registry (models-fasttext-onnx)
|
|
83
|
+
|
|
84
|
+
- The spec string `kotoshu.resources/v1` is frozen
|
|
85
|
+
(`schemas/registry.schema.json:15-17`). Any schema change to the
|
|
86
|
+
registry shape means `kotoshu.resources/v2`, consumed side by side.
|
|
87
|
+
- Resource ids are permanent and never reused; artifact content changes
|
|
88
|
+
bump `release_tag` and `registry_version`, never the id (schema
|
|
89
|
+
description, registry.schema.json:5).
|
|
90
|
+
- `min_engine_version` is the compatibility lever for old gem versions
|
|
91
|
+
reading a newer registry — raising it requires a coordinated release
|
|
92
|
+
note.
|
|
93
|
+
|
|
94
|
+
### Conformance vectors (the behavioral contract)
|
|
95
|
+
|
|
96
|
+
- `conformance/vectors.jsonl` (2630 rows) freezes what the engines
|
|
97
|
+
ACTUALLY return over the fixture corpora — Ruby and native must both
|
|
98
|
+
replay it identically (`lib/kotoshu/conformance_runner.rb:97-112`).
|
|
99
|
+
- A patch release MUST NOT change any vector. A minor may add vectors
|
|
100
|
+
(new corpora/words) but not change existing expectations. Changed
|
|
101
|
+
expectations = changed user-visible suggestions = major, with a
|
|
102
|
+
CHANGELOG entry listing affected inputs.
|
|
103
|
+
- Re-export procedure stays as documented: deterministic export
|
|
104
|
+
(`lib/kotoshu/conformance_exporter.rb:44-53`), commit to this repo,
|
|
105
|
+
kotoshu-rs syncs from it (conformance_exporter.rb:54-60).
|
|
106
|
+
|
|
107
|
+
## Deprecation window
|
|
108
|
+
|
|
109
|
+
Proposal: **two minor releases** (the standard the ecosystem already
|
|
110
|
+
drifted toward — `fetch` and `dictionaries_url` both wait indefinitely;
|
|
111
|
+
1.0 is the moment to make it finite).
|
|
112
|
+
|
|
113
|
+
- Ruby API: YARD `@deprecated` tag + a `Kernel#warn` (once per process)
|
|
114
|
+
from the moment of deprecation; removal at the second subsequent
|
|
115
|
+
minor, or at the next major, whichever comes first.
|
|
116
|
+
- Constants/autoloads (e.g. the alias constants at `lib/kotoshu.rb:80-85`):
|
|
117
|
+
same window; `autoload` removal is the deletion.
|
|
118
|
+
- CLI: the command/flag stays functional but hidden from help, printing a
|
|
119
|
+
stderr deprecation notice with the replacement; removal after two
|
|
120
|
+
minors. Exit codes do not change during the window.
|
|
121
|
+
- ENV/config keys (`KOTOSHU_*`): warn on read for two minors, then remove.
|
|
122
|
+
ENV removals are called out in the changelog header — they break CI
|
|
123
|
+
setups silently otherwise.
|
|
124
|
+
- HTTP: a deprecated field/endpoint returns `Deprecation`/`Sunset` style
|
|
125
|
+
headers during the window; `/v1` keeps serving it until `/v2` has been
|
|
126
|
+
stable for one minor.
|
|
127
|
+
- Exception: anything already deprecated throughout late 0.x (the `fetch`
|
|
128
|
+
alias, `dictionaries_url`/`models_url`) may be removed AT 1.0 without a
|
|
129
|
+
new window — the 0.x line was the window.
|
|
130
|
+
|
|
131
|
+
## The 1.0 gate
|
|
132
|
+
|
|
133
|
+
1.0 ships when the audit checklist in `docs/STABILITY-1.0-AUDIT.md` is
|
|
134
|
+
cleared: deprecation dispositions landed, the release train (gem + wasm +
|
|
135
|
+
server) coordinated, conformance compare green on both engines, and this
|
|
136
|
+
policy approved by the owner. Version numbers, floors, and the cut date
|
|
137
|
+
are owner decisions — this document proposes, the owner disposes.
|