kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# 72 — Model2Vec Evaluation (adopt-later C1 decision)
|
|
2
|
+
|
|
3
|
+
From [68-sota-adoption.md](68-sota-adoption.md) item C1. Model2Vec /
|
|
4
|
+
Potion distilled static embeddings: ~30 MB, ~500× faster than fastText,
|
|
5
|
+
beats other statics on MTEB
|
|
6
|
+
([repo](https://github.com/MinishLab/model2vec),
|
|
7
|
+
[fastText comparison](https://minish.ai/blog/2025-07-28-fasttext/)).
|
|
8
|
+
Word-keyed like ours — same OOV gap.
|
|
9
|
+
|
|
10
|
+
## Goal
|
|
11
|
+
|
|
12
|
+
Decide adopt/reject on DATA, using the corpus benchmark plan 69 built:
|
|
13
|
+
can a Potion-class static embedding beat our shipped full/fluency tiers
|
|
14
|
+
on the GitHub Typo Corpus top-1/5/20 hit rates, at comparable or
|
|
15
|
+
smaller size? License per model verified for redistribution (Potion
|
|
16
|
+
series is MIT, models distill from different teachers — record each
|
|
17
|
+
teacher's license).
|
|
18
|
+
|
|
19
|
+
## Tasks
|
|
20
|
+
|
|
21
|
+
1. `eval/model2vec_bench.py` — for each candidate model
|
|
22
|
+
(potion-base-8M, potion-base-32M, potion-mini-8M and one
|
|
23
|
+
fastText-comparable MTEB-strong variant): load via HF hub (network,
|
|
24
|
+
never in CI), map our corpus pair words, compute the same top-k hit
|
|
25
|
+
rates as `corpus_bench.py`, per language.
|
|
26
|
+
2. Report `eval/reports/model2vec.{lang}.json`: candidate vs
|
|
27
|
+
full/fluency/mini hit rates + model size + license chain.
|
|
28
|
+
3. Decision rule (pre-declared): adopt only if a candidate beats
|
|
29
|
+
**fluency** top-5 on ≥ 5 of 9 languages at ≤ 20 MB with a clean
|
|
30
|
+
license chain; otherwise record reject with numbers.
|
|
31
|
+
4. If adopt: a separate plan supersedes this one for integration
|
|
32
|
+
(registry entries, gem resolver) — this plan only decides.
|
|
33
|
+
|
|
34
|
+
## Acceptance
|
|
35
|
+
|
|
36
|
+
- Reports for all 9 languages × candidates, committed; corpora never
|
|
37
|
+
committed; decisions traceable to numbers.
|
|
38
|
+
|
|
39
|
+
## Status
|
|
40
|
+
|
|
41
|
+
**Evaluated (2026-09-03): REJECT on data** (models PR #7). The only
|
|
42
|
+
size-compliant candidate (potion-base-2M, 7.6 MB) wins 0/9 languages
|
|
43
|
+
(en top-5 0.241 vs fluency 0.252); the 129 MB potion-base-32M manages
|
|
44
|
+
one marginal English edge and loses everywhere else. Structural
|
|
45
|
+
blocker: Potion is English-only WordPiece - eight of nine languages
|
|
46
|
+
are not coverable. Licenses were clean; the rejection is on merit.
|
|
47
|
+
Side-result: our fastText tiers beat the current static-embedding SOTA
|
|
48
|
+
on our own task. C1 closed - revisit only if a genuinely multilingual
|
|
49
|
+
static model appears under the size cap.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# 73 — RL-Trained Reranker (adopt-later C2)
|
|
2
|
+
|
|
3
|
+
From [68-sota-adoption.md](68-sota-adoption.md) item C2. The GLM-5.3
|
|
4
|
+
recipe scaled down: RL on **binary verifiable rewards** — spelling
|
|
5
|
+
correction is verifiable by construction (output == reference)
|
|
6
|
+
([GLM tech report](https://arxiv.org/html/2602.15763v1)).
|
|
7
|
+
|
|
8
|
+
## Goal
|
|
9
|
+
|
|
10
|
+
Train tier-sized reranker weights (not LLMs — the existing embedding +
|
|
11
|
+
a small scorer head) with RL where the reward is exact-match on typo
|
|
12
|
+
corrections, replacing heuristic rerank scoring with a learned policy
|
|
13
|
+
that keeps our latency/memory envelope.
|
|
14
|
+
|
|
15
|
+
## Preconditions (all three, no exceptions)
|
|
16
|
+
|
|
17
|
+
1. Conformance vectors exist and are stable (plan 67 M3 output) — the
|
|
18
|
+
reward must be measurable against golden fixtures.
|
|
19
|
+
2. A training-pipeline owner is named (compute, seeds, checkpoints).
|
|
20
|
+
3. The corpus bench (plan 69) provides the train/eval split hygiene:
|
|
21
|
+
train on synthetic keyboard typos, eval on GitHub Typo Corpus.
|
|
22
|
+
|
|
23
|
+
## Tasks (when unblocked)
|
|
24
|
+
|
|
25
|
+
1. `training/` workspace (likely models repo or a new kotoshu-training
|
|
26
|
+
repo — owner decision): policy = current cosine scorer + small MLP
|
|
27
|
+
head over (candidate, context) features; GRPO-style or simple REINFORCE
|
|
28
|
+
with exact-match reward; baselines from the corpus bench.
|
|
29
|
+
2. Gate: beat the shipped rerank on corpus top-1 by ≥ 5 points without
|
|
30
|
+
regressing rank_corr gates; same ONNX exportability; int8-able.
|
|
31
|
+
3. Distribution through the existing registry as new model versions —
|
|
32
|
+
never a silent replacement.
|
|
33
|
+
|
|
34
|
+
## Acceptance
|
|
35
|
+
|
|
36
|
+
- Trained artifact passes all existing eval gates + corpus bench lift;
|
|
37
|
+
training reproducible from committed seeds and data manifests.
|
|
38
|
+
|
|
39
|
+
## Status
|
|
40
|
+
|
|
41
|
+
_Planning — gated_ on plan 67 M3 conformance vectors + named training
|
|
42
|
+
owner. Do not start before both exist.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# 74 — Distilled Reranker (adopt-later C3)
|
|
2
|
+
|
|
3
|
+
From [68-sota-adoption.md](68-sota-adoption.md) item C3. The DeepSeek
|
|
4
|
+
V4 recipe: consolidate specialists into a student via **on-policy
|
|
5
|
+
distillation (reverse KL)** — the exact recipe for distilling a large
|
|
6
|
+
teacher reranker into tier-sized weights
|
|
7
|
+
([report](https://arxiv.org/html/2606.19348v1)).
|
|
8
|
+
|
|
9
|
+
## Goal
|
|
10
|
+
|
|
11
|
+
Where RL (plan 73) learns from binary rewards alone, distillation
|
|
12
|
+
transfers a strong teacher's **soft judgments** — a large embedding or
|
|
13
|
+
LLM scoring the same candidate lists — into the small scorer, then the
|
|
14
|
+
RL stage (73) fine-tunes on top. Teacher runs offline during training
|
|
15
|
+
only; nothing large ships.
|
|
16
|
+
|
|
17
|
+
## Preconditions
|
|
18
|
+
|
|
19
|
+
Same as plan 73, plus a teacher selection (open-weights embedding
|
|
20
|
+
model or API LLM with a cost cap — owner decision; API usage must be
|
|
21
|
+
budgeted and never in CI).
|
|
22
|
+
|
|
23
|
+
## Tasks (when unblocked)
|
|
24
|
+
|
|
25
|
+
1. Teacher scoring pass over the training corpus (offline, one-off,
|
|
26
|
+
committed scores) — candidate lists identical to what the engine
|
|
27
|
+
produces at runtime.
|
|
28
|
+
2. Student = same architecture as plan 73's policy; loss = reverse KL
|
|
29
|
+
to teacher distribution over candidates; then plan 73's RL stage.
|
|
30
|
+
3. Gate: student ≥ teacher-within-1-point on corpus top-1 while fitting
|
|
31
|
+
the tier envelope; all existing gates pass.
|
|
32
|
+
|
|
33
|
+
## Status
|
|
34
|
+
|
|
35
|
+
_Planning — gated_ behind plan 73's preconditions + teacher decision.
|
|
36
|
+
Sequence: 74 distill → 73 RL fine-tune, one pipeline.
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# 75 — LLM-as-Evaluator (adopt-later C4)
|
|
2
|
+
|
|
3
|
+
From [68-sota-adoption.md](68-sota-adoption.md) item C4. LLM-as-GEC-
|
|
4
|
+
evaluator is standard practice in the literature
|
|
5
|
+
([BEA 2024](https://aclanthology.org/2024.bea-1.6/)).
|
|
6
|
+
|
|
7
|
+
## Goal
|
|
8
|
+
|
|
9
|
+
Augment the eval harness with an optional LLM judge that scores tier
|
|
10
|
+
outputs where our proxy metrics are weakest: fluency of the corrected
|
|
11
|
+
sentence and overcorrection risk — dimensions cosine hit rates cannot
|
|
12
|
+
see. Strictly offline tooling for model selection; never a release
|
|
13
|
+
gate (no API dependency in CI), never runtime.
|
|
14
|
+
|
|
15
|
+
## Tasks
|
|
16
|
+
|
|
17
|
+
1. `eval/llm_judge.py` — takes candidate corrections + references,
|
|
18
|
+
prompts a judge model (API, user-provided key via env; never
|
|
19
|
+
committed), produces per-language fluency/overcorrection scores
|
|
20
|
+
appended to a SEPARATE report file (eval/reports/llm-judge.*.json),
|
|
21
|
+
clearly marked non-deterministic.
|
|
22
|
+
2. Prompt + rubric documented in-repo; judge model + version recorded
|
|
23
|
+
in every report.
|
|
24
|
+
3. Cost cap per run (pair count × languages), hard default small
|
|
25
|
+
(e.g. 100 pairs/language) — a full sweep needs explicit flags.
|
|
26
|
+
|
|
27
|
+
## Acceptance
|
|
28
|
+
|
|
29
|
+
- Runs only with an explicit env key + opt-in flag; reports carry
|
|
30
|
+
model/version/cost metadata; zero effect on gates or CI.
|
|
31
|
+
|
|
32
|
+
## Status
|
|
33
|
+
|
|
34
|
+
_Planning — low priority._ Proxy metrics suffice until corpora-driven
|
|
35
|
+
work (72–74) lands.
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# 76 — Optional External LLM Backend (adopt-later C5)
|
|
2
|
+
|
|
3
|
+
From [68-sota-adoption.md](68-sota-adoption.md) item C5 and its
|
|
4
|
+
rejection note: LLMs never become the default engine (violates small/
|
|
5
|
+
offline/CPU), but an explicit opt-in "heavy" backend can exist for
|
|
6
|
+
users who want maximum fluency polishing and accept the cost.
|
|
7
|
+
|
|
8
|
+
## Goal
|
|
9
|
+
|
|
10
|
+
A `Kotoshu::Backends::Llm` adapter behind the same interfaces the
|
|
11
|
+
engine already exposes (checker/reranker seams), delegating
|
|
12
|
+
sentence-level polishing to a local (1–8B via ONNX Runtime GenAI /
|
|
13
|
+
llama.cpp) or API model. Default OFF; absent config = today's behavior
|
|
14
|
+
exactly.
|
|
15
|
+
|
|
16
|
+
## Design constraints
|
|
17
|
+
|
|
18
|
+
1. Adapter pattern, no engine changes: register like any plugin;
|
|
19
|
+
capability flags mark it slow/networked so callers can decline.
|
|
20
|
+
2. Two transports: local GGUF/ORT-GenAI (user-supplied model path) and
|
|
21
|
+
HTTP API (user-supplied endpoint + key via env). No keys in code,
|
|
22
|
+
no telemetry.
|
|
23
|
+
3. Interactive surfaces (LSP, CLI `--interactive`) surface a visible
|
|
24
|
+
"heavy mode" indicator; CI surfaces refuse it by default.
|
|
25
|
+
4. Output contract: same suggestion objects; LLM refinements only
|
|
26
|
+
applied on explicit accept in interactive mode.
|
|
27
|
+
|
|
28
|
+
## Acceptance
|
|
29
|
+
|
|
30
|
+
- Engine untouched with backend disabled (existing suite proves it);
|
|
31
|
+
adapter specs use a stub server (no real API in tests); docs state
|
|
32
|
+
latency/memory expectations honestly.
|
|
33
|
+
|
|
34
|
+
## Status
|
|
35
|
+
|
|
36
|
+
_Planning — gated_ on user demand. Not scheduled until someone asks
|
|
37
|
+
for it.
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# 77 — Model coverage expansion: 9 → 25+ languages
|
|
2
|
+
|
|
3
|
+
> Asked by the owner 2026-09-05: "Are our models fully complete and
|
|
4
|
+
> comprehensive with no user gaps?" They are not — this plan closes the gap.
|
|
5
|
+
|
|
6
|
+
## Context
|
|
7
|
+
|
|
8
|
+
The registry (models-fasttext-onnx `registry.json`, v1.0.1) carries
|
|
9
|
+
semantic models for exactly **9 languages** (de en es fr ja ko pt ru zh)
|
|
10
|
+
× 3 tiers = 27 entries. Meanwhile:
|
|
11
|
+
|
|
12
|
+
- `dictionaries/` repo: **98** languages with Hunspell `.aff`/`.dic`
|
|
13
|
+
- FastText CC vectors exist for **157+** languages
|
|
14
|
+
- The site language matrix and gem language registry advertise the
|
|
15
|
+
dictionary breadth, but semantic reranking silently stops at 9
|
|
16
|
+
|
|
17
|
+
A user who spells in Italian, Polish, Turkish, or Dutch gets the
|
|
18
|
+
traditional path only. That is the largest user gap in the ecosystem.
|
|
19
|
+
|
|
20
|
+
## What is NOT changing
|
|
21
|
+
|
|
22
|
+
- The tier recipe (full 300d / fluency int8 top-50k / mini int8
|
|
23
|
+
top-10k) — settled by plan 67's measured gates; no SVD, no Model2Vec.
|
|
24
|
+
- The eval gates from plan 67's gate table: fluency
|
|
25
|
+
`rank_corr ≥ 0.97 / top1 ≥ 0.95`, mini `rank_corr ≥ 0.90 /
|
|
26
|
+
top1 ≥ 0.85` (keyboard-aware eval from plan 69). A language that
|
|
27
|
+
fails gates ships **no** tiers rather than bad tiers.
|
|
28
|
+
|
|
29
|
+
## Phases
|
|
30
|
+
|
|
31
|
+
### Phase 1 — candidate selection (deterministic)
|
|
32
|
+
|
|
33
|
+
Candidates = dictionaries repo dirs ∩ FastText `cc.{lang}.300.vec`
|
|
34
|
+
availability ∩ eval-gate pass. Rank by speaker base × dictionary
|
|
35
|
+
quality (dictionary repo manifest). Batch 1 target (16):
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
it nl pl uk tr cs sv el hu ro da fi no id vi ca
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Batch 2 (stretch, if compute allows): ar fa he th vi-tok hi bn ms tl
|
|
42
|
+
sk sl lt lv et is hr sr bg.
|
|
43
|
+
|
|
44
|
+
### Phase 2 — pipeline run per language
|
|
45
|
+
|
|
46
|
+
Use the existing scripts (see models repo `TODO.impl/01`, `04`, `06`):
|
|
47
|
+
download `cc.{lang}.300.vec.gz` → convert to ONNX full → int8 +
|
|
48
|
+
vocab-cut to fluency/mini → keyboard-aware eval → gates. Record
|
|
49
|
+
per-language eval JSON into `eval/` like the existing nine.
|
|
50
|
+
|
|
51
|
+
### Phase 3 — registry + release
|
|
52
|
+
|
|
53
|
+
Regenerate `registry.json` (+ manifest, schemas untouched), bump
|
|
54
|
+
registry_version, tag `v1.1.0` (policy from models plan 05: pre-declared
|
|
55
|
+
minor for coverage expansion). Release assets: 3 tiers × new languages,
|
|
56
|
+
mirrored to the media host per plan 07's distribution map.
|
|
57
|
+
|
|
58
|
+
### Phase 4 — downstream truth updates
|
|
59
|
+
|
|
60
|
+
- Gem `Language` registry: full-feature languages list gains the
|
|
61
|
+
new entries (spelling + model available).
|
|
62
|
+
- Site language matrix + per-language pages: flip `hasModel` for
|
|
63
|
+
the new set; copy updated by the site plan (79/80 reuse).
|
|
64
|
+
- kotoshu-rs / SDKs: no code change (registry-driven).
|
|
65
|
+
|
|
66
|
+
## Owner gates
|
|
67
|
+
|
|
68
|
+
- None for batch 1 execution (standing directive 2026-09-05).
|
|
69
|
+
- Version numbers: v1.1.0 follows plan 05's pre-declared policy —
|
|
70
|
+
registry major/minor tracks coverage, patch tracks regeneration.
|
|
71
|
+
If the owner objects, yank per policy is NOT possible for release
|
|
72
|
+
assets — ask first if in doubt.
|
|
73
|
+
|
|
74
|
+
## Success criteria
|
|
75
|
+
|
|
76
|
+
- ≥ 8 new languages pass gates and appear in registry + release.
|
|
77
|
+
- Every shipped tier has eval JSON proving the gates.
|
|
78
|
+
- Registry schema version unchanged; gem + site consume it without
|
|
79
|
+
code changes.
|
|
80
|
+
|
|
81
|
+
## Status
|
|
82
|
+
|
|
83
|
+
**Implemented (2026-09-05, models PR #13 + tag v1.1.0).** 13 of 16
|
|
84
|
+
batch-one candidates shipped: ca cs da el hu it nl pl ro sv tr uk vi
|
|
85
|
+
— 22 languages x 3 tiers, 66 registry resources, 134 release assets.
|
|
86
|
+
All tiers passed the gates unweakened (fluency rank_corr 0.9999,
|
|
87
|
+
top1 0.9583-1.0000). noise.py grew QWERTY/QWERTZ supplements plus
|
|
88
|
+
curated Turkish-Q / Ukrainian JCUKEN / Greek-phonetic grids; a
|
|
89
|
+
pre-existing release_notes.py crash on null tier mirrors was fixed
|
|
90
|
+
in passing. Released registry byte-identical to committed (local
|
|
91
|
+
rebuild reproduces CI sha256).
|
|
92
|
+
|
|
93
|
+
Dropped from batch 1 (no dictionaries-repo aff/dic): fi, no, id —
|
|
94
|
+
`no` exists only as nb/nn; an owner mapping decision for a v1.2.
|
|
95
|
+
Site truth updated (MODEL_LANGUAGES + news entry, site PR #4); the
|
|
96
|
+
gem needed no change (registry-driven).
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# 78 — Zero-install wasm playground on the site
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
The site playground (`/playground`) currently requires a locally
|
|
6
|
+
running `kotoshu-server` — a real barrier for the curious visitor.
|
|
7
|
+
Since `@kotoshu/wasm` 0.1.0 is live on npm (291 KiB gz engine), a
|
|
8
|
+
zero-install in-browser demo is now possible: load the engine from a
|
|
9
|
+
CDN, fetch `.aff`/`.dic` + a mini-tier model, check text live.
|
|
10
|
+
|
|
11
|
+
This is the single highest-leverage marketing surface: every visitor
|
|
12
|
+
can feel the product in five seconds with no install.
|
|
13
|
+
|
|
14
|
+
## Design
|
|
15
|
+
|
|
16
|
+
- **Route**: keep `/playground` as the home of the in-browser demo;
|
|
17
|
+
the server-based playground moves to `/playground/server` (linked
|
|
18
|
+
as "run it against your own instance").
|
|
19
|
+
- **Engine**: `import { init } from "@kotoshu/wasm"` via a pinned
|
|
20
|
+
CDN URL (esm.run / esm.sh, version-pinned to 0.1.x). Never
|
|
21
|
+
`@latest`.
|
|
22
|
+
- **Dictionaries**: `kotoshu/dictionaries` repo files via jsDelivr
|
|
23
|
+
`/gh/kotoshu/dictionaries@<tag>/<lang>.aff|.dic` — verify the
|
|
24
|
+
tag layout first (dictionaries repo manifest). Files are < 20 MiB
|
|
25
|
+
so jsDelivr gh CDN is legal.
|
|
26
|
+
- **Model**: mini tier from the models registry. Registry URLs point
|
|
27
|
+
at release assets — verify browser CORS on
|
|
28
|
+
`objects.githubusercontent.com` before committing to that source;
|
|
29
|
+
fallback = jsDelivr gh for tag-tree files, else document the
|
|
30
|
+
limitation and ship dictionary-only mode with a note that
|
|
31
|
+
reranking needs the full app.
|
|
32
|
+
- **UX**: language select (defaults to browser language if one of
|
|
33
|
+
the supported set), textarea with misspelling underlines,
|
|
34
|
+
suggestion popover on click, tier badge (mini), first-load size
|
|
35
|
+
readout. Reduced-motion respected; no layout shift.
|
|
36
|
+
- **Aesthetic**: follows the site's dictionary-catalog design
|
|
37
|
+
system (site plans 05/09) — the playground is a "specimen" page,
|
|
38
|
+
not a generic SaaS demo.
|
|
39
|
+
|
|
40
|
+
## Phases
|
|
41
|
+
|
|
42
|
+
1. **Fetch spike** (an hour): prove CORS + CDN loads for engine,
|
|
43
|
+
one dictionary, mini model. Decide model source. If models CORS
|
|
44
|
+
fails → dictionary-only v1, models behind the server playground.
|
|
45
|
+
2. **Component**: `PlaygroundLab.astro` + island
|
|
46
|
+
(`client:load`), module-scope caching of loaded languages,
|
|
47
|
+
Web Worker for the engine to keep typing at 60fps (the engine
|
|
48
|
+
API is sync; worker keeps the main thread free).
|
|
49
|
+
3. **Wire routes**, update copy on `/` hero to link the demo
|
|
50
|
+
("try it now — nothing installs").
|
|
51
|
+
4. **Verify in a browser** before merge: golden path (load en,
|
|
52
|
+
type a typo, get suggestions), CJK (ja), offline (graceful
|
|
53
|
+
error), reduced-motion, mobile width.
|
|
54
|
+
|
|
55
|
+
## Owner gates
|
|
56
|
+
|
|
57
|
+
None. CDN pinning note: record the chosen CDN + version in this
|
|
58
|
+
file's Status when shipped.
|
|
59
|
+
|
|
60
|
+
## Status
|
|
61
|
+
|
|
62
|
+
**Implemented (2026-09-05, site PRs #1-#3, live at kotoshu.org/playground).**
|
|
63
|
+
Zero-install wasm demo: @kotoshu/wasm 0.1.0 in a worker, underlines +
|
|
64
|
+
click-to-mend popover, six languages; server playground moved to
|
|
65
|
+
/playground/server. CDN pin: jsDelivr npm raw files @0.1.0 with manual
|
|
66
|
+
instantiate (esm.sh and esm.run both broken for the bundler-target
|
|
67
|
+
package); dictionaries via jsDelivr gh @commit 1829a3e. Models verdict:
|
|
68
|
+
GitHub release assets send no CORS headers, so the demo is honestly
|
|
69
|
+
dictionary-only; the model story lives on the server playground page.
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# 79 — News + releases section on the site
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
The site has docs, languages, audiences, install — but no news.
|
|
6
|
+
Releases across seven registries shipped in 2026-09 and a visitor
|
|
7
|
+
cannot tell the project is alive. The owner asked for this
|
|
8
|
+
explicitly ("think also about documentation and our website
|
|
9
|
+
(e.g. docs, news)").
|
|
10
|
+
|
|
11
|
+
## Design
|
|
12
|
+
|
|
13
|
+
- `/news` — index, reverse-chronological, one entry per release or
|
|
14
|
+
event, each rendered as a dictionary entry (date as the headword's
|
|
15
|
+
etymology line; the entry body is the note).
|
|
16
|
+
- Entry pages `/news/<slug>` for anything longer than a paragraph.
|
|
17
|
+
- **RSS + Atom** at `/news/rss.xml` + `/news/atom.xml` (static).
|
|
18
|
+
- Design follows the site system (eyebrows, `辞書` vertical accent,
|
|
19
|
+
numbered senses for changelog bullets). No blog-engine machinery —
|
|
20
|
+
entries are typed content collections.
|
|
21
|
+
|
|
22
|
+
## Seed content (real history, verified 2026-09-05)
|
|
23
|
+
|
|
24
|
+
1. **2026-09-05 — Trusted publishing everywhere** — npm, RubyGems,
|
|
25
|
+
crates.io now publish keyless via OIDC (RFC 3691 on crates.io,
|
|
26
|
+
provenance on npm).
|
|
27
|
+
2. **2026-09-05 — First releases on all channels** — PyPI
|
|
28
|
+
`kotoshu` + `kotoshu-native` 0.1.0; npm `@kotoshu/wasm` +
|
|
29
|
+
`@kotoshu/client` 0.1.0; crates.io `kotoshu` 0.1.0.
|
|
30
|
+
3. **2026-09-05 — Ruby gem 0.7.0** — the universal-kotoshu cut:
|
|
31
|
+
model tiers + registry, native extension (dual-backend parity,
|
|
32
|
+
2630 conformance vectors, 0 divergences), legacy-cache bridge,
|
|
33
|
+
tier default = fluency.
|
|
34
|
+
4. **2026-09-04 — Engine correctness wave** — compound/suggest
|
|
35
|
+
parity with hunspell semantics (T2), 1315+1315 conformance green
|
|
36
|
+
in Rust and Ruby.
|
|
37
|
+
5. **2026-09-03 — Models v1.0.x registry + tiers** — 9 languages ×
|
|
38
|
+
3 tiers, keyboard-aware eval, media-host distribution.
|
|
39
|
+
6. **Earlier** — kotoshu-lsp 0.1.0, kotoshu-server 0.1.0,
|
|
40
|
+
kotoshu-go v0.1.0, action-kotoshu v1, site launch.
|
|
41
|
+
|
|
42
|
+
Pull dates from the registries/gem CHANGELOG while writing — verify,
|
|
43
|
+
don't guess.
|
|
44
|
+
|
|
45
|
+
## Phases
|
|
46
|
+
|
|
47
|
+
1. Content collection schema + seed entries above.
|
|
48
|
+
2. `/news` index + slugs + feeds; nav + footer links; sitemap.
|
|
49
|
+
3. Cross-link: gem docs page → news; news → install.
|
|
50
|
+
|
|
51
|
+
## Owner gates
|
|
52
|
+
|
|
53
|
+
None.
|
|
54
|
+
|
|
55
|
+
## Status
|
|
56
|
+
|
|
57
|
+
**Implemented (2026-09-05, site PR #1).** /news with 8 entries as
|
|
58
|
+
dictionary entries, RSS 2.0 + Atom 1.0 static feeds with autodiscovery,
|
|
59
|
+
nav/footer/sitemap wired. Dates verified against registry timestamps
|
|
60
|
+
(two plan guesses corrected: PyPI shipped 09-04; channels entry split).
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# 80 — Ecosystem docs: every channel documented
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
The site documents the gem well (cli, caching, configuration, api,
|
|
6
|
+
comparison, migration, plugins). But the ecosystem now ships on
|
|
7
|
+
eight channels, and a Python/JS/Go/Rust user arriving at
|
|
8
|
+
kotoshu.org finds no path in. Audience: non-Ruby developers —
|
|
9
|
+
exactly who the access libraries (plan 67) were built for.
|
|
10
|
+
|
|
11
|
+
## Channels to document (verified live 2026-09-05)
|
|
12
|
+
|
|
13
|
+
| Channel | Package | What a user needs |
|
|
14
|
+
|---|---|---|
|
|
15
|
+
| PyPI | `kotoshu`, `kotoshu-native` | pip install, Python API, maturin wheel note |
|
|
16
|
+
| npm | `@kotoshu/client`, `@kotoshu/wasm` | npm install, JS API, browser/Node |
|
|
17
|
+
| crates.io | `kotoshu` | cargo add, Rust API, KOSH-v1 batch format |
|
|
18
|
+
| Go | `kotoshu-go` | go get, HTTP-mode usage |
|
|
19
|
+
| RubyGems | `kotoshu` | already documented (link) |
|
|
20
|
+
| HTTP | `kotoshu-server` | /v1/check contract, docker run |
|
|
21
|
+
| LSP | `kotoshu-lsp` | gem install + editor config (Neovim lspconfig, generic LSP) |
|
|
22
|
+
| CI | `action-kotoshu` | workflow yaml, SARIF upload |
|
|
23
|
+
|
|
24
|
+
## Structure
|
|
25
|
+
|
|
26
|
+
- `/install` becomes a channel-picker (tabs or a table: language →
|
|
27
|
+
command), each linking to a per-channel page under
|
|
28
|
+
`/docs/clients/…`.
|
|
29
|
+
- `/docs/clients/python|javascript|rust|go|http|lsp|action` pages:
|
|
30
|
+
install, minimal example, capability notes (which engine tier,
|
|
31
|
+
what is native vs client), link to package registry page.
|
|
32
|
+
- Per-channel quick starts **verified by running them** (pip install
|
|
33
|
+
+ 5-line script; npm install + node -e; cargo add + cargo run;
|
|
34
|
+
docker run + curl) — the docs/quickstart-verification precedent
|
|
35
|
+
(gem plan 28) applies: no example ships unexecuted.
|
|
36
|
+
- The audiences page cross-links each persona to their channel.
|
|
37
|
+
|
|
38
|
+
## Phases
|
|
39
|
+
|
|
40
|
+
1. `/install` restructure (channel picker).
|
|
41
|
+
2. Seven per-channel pages + examples verified.
|
|
42
|
+
3. Nav/sitemap/search-index updates.
|
|
43
|
+
|
|
44
|
+
## Owner gates
|
|
45
|
+
|
|
46
|
+
None.
|
|
47
|
+
|
|
48
|
+
## Status
|
|
49
|
+
|
|
50
|
+
**Implemented (2026-09-05, site PR #2).** /install is a channel catalog
|
|
51
|
+
(8 channels); seven guides under /docs/clients/ (python javascript rust
|
|
52
|
+
go http lsp action). Every quick start executed for real (pip, npm http
|
|
53
|
+
+ wasm, cargo, go, server-from-source + curl, LSP stdio handshake);
|
|
54
|
+
the action YAML parse-validated only. Each page carries a verified-date
|
|
55
|
+
annotation. kotoshu-server docs intentionally show run-from-source
|
|
56
|
+
until the empty 0.1.0 gem is republished.
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# 81 — Editor reach: VS Code extension + pre-commit
|
|
2
|
+
|
|
3
|
+
> Supersedes the unshipped parts of
|
|
4
|
+
> [61-editor-ecosystem.md](61-editor-ecosystem.md). Status
|
|
5
|
+
> correction: kotoshu-lsp 0.1.0 (plan 60), kotoshu-server 0.1.0 +
|
|
6
|
+
> SDKs (plan 64, partial), and action-kotoshu v1 shipped during the
|
|
7
|
+
> 2026-09 campaign; what remains is the editor package and the
|
|
8
|
+
> pre-commit hook.
|
|
9
|
+
|
|
10
|
+
## Context
|
|
11
|
+
|
|
12
|
+
The LSP server exists (`kotoshu-lsp` gem) but nothing delivers it
|
|
13
|
+
into editors. The generic-LSP path requires manual config most
|
|
14
|
+
users never do. VS Code via a marketplace extension is the single
|
|
15
|
+
highest-leverage editor surface. pre-commit is where CI-minded
|
|
16
|
+
Python/Go shops hook spellchecking.
|
|
17
|
+
|
|
18
|
+
## Track A — `kotoshu-vscode` (new repo)
|
|
19
|
+
|
|
20
|
+
- Scaffold: TypeScript extension, `vscode-languageclient`, no
|
|
21
|
+
bundling surprises (esbuild), `package.json` contributes
|
|
22
|
+
`kotoshu.languageTool` (document selectors: markdown, asciidoc,
|
|
23
|
+
plaintext, code comments opt-in later).
|
|
24
|
+
- Server discovery order: `kotoshu-lsp.serverPath` setting →
|
|
25
|
+
`bundle exec kotoshu-lsp` in workspace → global `kotoshu-lsp` →
|
|
26
|
+
actionable error message with install command (no silent
|
|
27
|
+
failure).
|
|
28
|
+
- Features v0.1: publish diagnostics on open/change/save;
|
|
29
|
+
suggestions as code actions; personal-dictionary add action
|
|
30
|
+
(server command if supported, else documented).
|
|
31
|
+
- Repo: `kotoshu/kotoshu-vscode`, CI (build + `vscode-test` smoke
|
|
32
|
+
against a fixture workspace with a deliberate misspelling),
|
|
33
|
+
LICENSE BSD-2-Clause, README with GIF slot.
|
|
34
|
+
- Packaging: `vsce package` produces a .vsix in CI artifacts.
|
|
35
|
+
**Marketplace publish is owner-gated** (needs publisher PAT);
|
|
36
|
+
Open VSX after that.
|
|
37
|
+
|
|
38
|
+
## Track B — pre-commit hook
|
|
39
|
+
|
|
40
|
+
- `.pre-commit-hooks.yaml` living where the hook can `gem install`
|
|
41
|
+
or reuse a bundled environment. Evaluate: hook id
|
|
42
|
+
`kotoshu-spell` running `kotoshu check` (requires Ruby+gem —
|
|
43
|
+
document honestly), vs a container hook (`language: docker_ruby`
|
|
44
|
+
via the published image). Ship the honest one with clear docs;
|
|
45
|
+
note the wasm client as the future zero-Ruby path.
|
|
46
|
+
- Placement: `action-kotoshu` repo (CI-adjacent) or the gem repo
|
|
47
|
+
(ships with the tool). Prefer the **gem repo** — the hook file
|
|
48
|
+
documents itself beside the CLI it calls.
|
|
49
|
+
|
|
50
|
+
## Phases
|
|
51
|
+
|
|
52
|
+
A1 scaffold → A2 discovery + diagnostics → A3 code actions + smoke
|
|
53
|
+
CI → B1 hook + docs → B2 (owner) marketplace publish.
|
|
54
|
+
|
|
55
|
+
## Owner gates
|
|
56
|
+
|
|
57
|
+
- Marketplace publish (Track A final step).
|
|
58
|
+
- Open VSX mirror.
|
|
59
|
+
|
|
60
|
+
## Status
|
|
61
|
+
|
|
62
|
+
**Track A done (2026-09-05): kotoshu/kotoshu-vscode exists** - TS
|
|
63
|
+
extension wrapping kotoshu-lsp over stdio, diagnostics + quickfix code
|
|
64
|
+
actions + hover, server discovery (setting -> bundler -> PATH) with
|
|
65
|
+
actionable install error, CI green including a vscode-test smoke
|
|
66
|
+
against a misspelling fixture, vsce package artifact gated by a manifest
|
|
67
|
+
check. Marketplace publish remains owner-gated (publisher PAT).
|
|
68
|
+
**Track B done: .pre-commit-hooks.yaml shipped in gem PR #118.**
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# 82 — Document ignores + CI baselines
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
Two long-standing user gaps in the gem (verified 2026-09-05 — no
|
|
6
|
+
`disable` handling exists anywhere in `lib/kotoshu/documents/`):
|
|
7
|
+
|
|
8
|
+
1. **Inline ignores** — a word that is intentionally "wrong"
|
|
9
|
+
(product names, code identifiers in prose, dialect) cannot be
|
|
10
|
+
suppressed per-line; users fall back to polluting the personal
|
|
11
|
+
dictionary.
|
|
12
|
+
2. **CI baselines** — `kotoshu check` in CI fails on ALL known
|
|
13
|
+
errors forever; there is no way to freeze existing debt and fail
|
|
14
|
+
only on new errors. Every serious linter has this (ESLint
|
|
15
|
+
`--baseline`, RuboCop `--exclude-limit` culture, cspell
|
|
16
|
+
`--no-cache` + wordlists).
|
|
17
|
+
|
|
18
|
+
## Track A — ignore directives
|
|
19
|
+
|
|
20
|
+
Syntax (comments recognized by each document format's parser):
|
|
21
|
+
|
|
22
|
+
```
|
|
23
|
+
kotoshu:disable-line — suppress everything on this line
|
|
24
|
+
kotoshu:disable-next-line — suppress on the following line
|
|
25
|
+
kotoshu:disable-next-line foo bar — suppress only these words
|
|
26
|
+
kotoshu:disable-file / kotoshu:enable-file — block suppression
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
- Markdown: HTML comments `<!-- kotoshu:disable-next-line -->`;
|
|
30
|
+
AsciiDoc: `// kotoshu:disable-next-line`; plain text: a line whose
|
|
31
|
+
trimmed content is the directive. Trailing directives
|
|
32
|
+
(`text <!-- kotoshu:disable-line -->`) supported in MD/ADOC.
|
|
33
|
+
- Implementation: document layer (`documents/`) attaches
|
|
34
|
+
`suppressions` (source ranges) during parse; the checker filters
|
|
35
|
+
results whose range/word matches. Suppression data flows through
|
|
36
|
+
the public result objects (`results/*`) so SARIF/JSON consumers
|
|
37
|
+
see suppressed entries either omitted or marked (omitted by
|
|
38
|
+
default; `--show-suppressed` lists them).
|
|
39
|
+
- Also recognized: the spellchecker facade (`.check`) so the gem
|
|
40
|
+
API gets it for free.
|
|
41
|
+
|
|
42
|
+
## Track B — baseline files
|
|
43
|
+
|
|
44
|
+
- `kotoshu check --baseline .kotoshu-baseline.json`:
|
|
45
|
+
- Baseline format: canonical JSON `{file, line, word, count}` —
|
|
46
|
+
stable across reformatting (count-based, not position-based).
|
|
47
|
+
- `kotoshu baseline init` generates it from current findings.
|
|
48
|
+
- Semantics: an error passes if it matches the baseline
|
|
49
|
+
(same file+word, count not exceeded); new errors fail; a
|
|
50
|
+
summary line reports how many baseline entries are now stale
|
|
51
|
+
(file no longer has the error) so debt shrinks visibly.
|
|
52
|
+
- SARIF output marks suppressed entries with the baseline tag.
|
|
53
|
+
|
|
54
|
+
## Specs
|
|
55
|
+
|
|
56
|
+
Behavior specs throughout (no doubles; real documents). Cover:
|
|
57
|
+
directive forms per format, word-scoped suppression, block
|
|
58
|
+
disable/enable nesting, baseline count semantics, stale-entry
|
|
59
|
+
reporting, SARIF marking.
|
|
60
|
+
|
|
61
|
+
## Phases
|
|
62
|
+
|
|
63
|
+
A1 directive parsing + filtering → A2 per-format specs + docs page
|
|
64
|
+
section → B1 baseline format + commands → B2 CI docs example.
|
|
65
|
+
|
|
66
|
+
## Owner gates
|
|
67
|
+
|
|
68
|
+
None. CHANGELOG entry under Unreleased (version = owner).
|
|
69
|
+
|
|
70
|
+
## Status
|
|
71
|
+
|
|
72
|
+
**Implemented (2026-09-05, gem PR #118).** Shared
|
|
73
|
+
Documents::Suppressions scanner (disable-line, disable-next-line with
|
|
74
|
+
optional word list, nestable disable-file/enable-file blocks; per-format
|
|
75
|
+
comment detection; directive lines never spellchecked); check filters
|
|
76
|
+
via the :auto profile; suppressed errors ride DocumentResult with
|
|
77
|
+
suppressed/suppressed_by flags; --show-suppressed. Baselines: baseline
|
|
78
|
+
init + check --baseline with count-based budgets, stale-entry
|
|
79
|
+
reporting, SARIF suppression marking; serialized via lutaml-model.
|
|
80
|
+
55 new examples; default suite 3410/0/26. Release cut pending owner
|
|
81
|
+
version decision.
|