kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
# 63 — Web playground (WASM)
|
|
2
|
+
|
|
3
|
+
Promotes T6.2 in `TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md`
|
|
4
|
+
into a promotable plan.
|
|
5
|
+
|
|
6
|
+
## Goal
|
|
7
|
+
|
|
8
|
+
A browser-only Kotoshu playground at `play.kotoshu.dev` (or under
|
|
9
|
+
`kotoshu.github.io/playground/`) that:
|
|
10
|
+
|
|
11
|
+
- Loads instantly, no signup, no backend.
|
|
12
|
+
- Spell-checks user-pasted text in real time.
|
|
13
|
+
- Demonstrates the library's capabilities (suggestions, language
|
|
14
|
+
detection, document-format awareness) as a marketing surface for the
|
|
15
|
+
gem.
|
|
16
|
+
|
|
17
|
+
Built on [ruby.wasm](https://github.com/ruby/ruby.wasm), so the actual
|
|
18
|
+
Kotoshu Ruby code runs in the browser — not a JS port, not a server
|
|
19
|
+
round-trip.
|
|
20
|
+
|
|
21
|
+
## Why
|
|
22
|
+
|
|
23
|
+
A playground is the canonical "try before you install" for developer
|
|
24
|
+
tools. Without one:
|
|
25
|
+
|
|
26
|
+
- Curious users have to `gem install`, set up a language, and write a
|
|
27
|
+
Ruby script just to see what Kotoshu does. Most bounce.
|
|
28
|
+
- The semantic path (the gem's standout feature) is invisible until
|
|
29
|
+
you've installed onnxruntime and a 4 GB model — a non-starter for
|
|
30
|
+
casual evaluation.
|
|
31
|
+
|
|
32
|
+
A WASM playground flips that: paste, see, decide. It also doubles as:
|
|
33
|
+
|
|
34
|
+
- **A bug-report reproducer** — link a playground URL with the failing
|
|
35
|
+
text in issues.
|
|
36
|
+
- **A feature showcase** for the marketing site (plan in
|
|
37
|
+
`kotoshu.github.io/TODO.impl/`).
|
|
38
|
+
- **A regression test surface** — snapshot outputs across releases.
|
|
39
|
+
|
|
40
|
+
## Constraints (these shape every task)
|
|
41
|
+
|
|
42
|
+
- **No native extensions in the browser.** WASM Ruby cannot load C
|
|
43
|
+
extensions. This means:
|
|
44
|
+
- No `onnxruntime` — the semantic path is not available in-browser.
|
|
45
|
+
- No `suika` — Japanese tokenization (the dartsclone native ext) is
|
|
46
|
+
unavailable. Japanese either ships with a pre-tokenized dictionary
|
|
47
|
+
or is omitted from the playground.
|
|
48
|
+
- No `rubyzip` — bundle dictionaries unpacked.
|
|
49
|
+
- **Size budget.** ruby.wasm is ~3 MB. A full Hunspell dictionary is
|
|
50
|
+
~5 MB compressed. A full FastText `.vec` is 2 GB. The playground
|
|
51
|
+
cannot ship the full corpus.
|
|
52
|
+
- **Cold start.** WASM Ruby instantiation takes ~1–2 s on a fast
|
|
53
|
+
machine, more on mobile. The UI must hide that behind a loading
|
|
54
|
+
state.
|
|
55
|
+
|
|
56
|
+
## Tasks
|
|
57
|
+
|
|
58
|
+
### Phase 1 — Minimum viable playground
|
|
59
|
+
|
|
60
|
+
1. **Pick the dictionary subset.** Ship a top-50K-words-per-language
|
|
61
|
+
subset of the Hunspell `.dic` plus the full `.aff` (affixes are
|
|
62
|
+
small). 50K covers >99% of typical prose. Subset lives in the
|
|
63
|
+
`kotoshu.github.io` repo under `playground/dictionaries/<code>/`.
|
|
64
|
+
2. **Language support.** Start with the six full-feature languages
|
|
65
|
+
(de, en, es, fr, pt, ru). Document that Japanese is omitted (no
|
|
66
|
+
suika in WASM); other languages follow as language modules land
|
|
67
|
+
(plan 04).
|
|
68
|
+
3. **ruby.wasm packaging.** Build pipeline (Rakefile in the
|
|
69
|
+
`kotoshu.github.io` repo) that:
|
|
70
|
+
- Compiles a `kotoshu.wasm` with the gem baked in.
|
|
71
|
+
- Strips the gem's `onnxruntime`, `suika`, `rubyzip` require paths
|
|
72
|
+
via conditional loading (the library already soft-requires them —
|
|
73
|
+
verify WASM mode is treated like the "neither soft-dep installed"
|
|
74
|
+
case).
|
|
75
|
+
- Bundles the 50K-word dictionaries as gzipped JSON.
|
|
76
|
+
4. **UI shell.** Single page:
|
|
77
|
+
- A textarea (or Monaco editor for syntax-highlighting) on the left.
|
|
78
|
+
- A diagnostics panel on the right, mirroring what the LSP server
|
|
79
|
+
emits (plan 60).
|
|
80
|
+
- A language selector (defaults to auto-detect).
|
|
81
|
+
- A "Format" toggle (plain / markdown / asciidoc) to demo document
|
|
82
|
+
parsing.
|
|
83
|
+
5. **In-browser caching.** IndexedDB stores the WASM blob, the
|
|
84
|
+
dictionaries, and the user's personal dictionary. Subsequent visits
|
|
85
|
+
load from cache.
|
|
86
|
+
|
|
87
|
+
### Phase 2 — Showcase features
|
|
88
|
+
|
|
89
|
+
6. **Suggestion panel.** Clicking a flagged word shows ranked
|
|
90
|
+
suggestions with frequency-rank badges (from the Kelly lists, if
|
|
91
|
+
the playground ships them).
|
|
92
|
+
7. **Language detection demo.** A visible "Detected: fr (95%
|
|
93
|
+
confidence)" badge using the pure-Ruby heuristic path
|
|
94
|
+
(`Language::Identifier` falls back to n-gram detection when the
|
|
95
|
+
FastText LID model is unavailable — verify this is what happens in
|
|
96
|
+
WASM, or ship the LID model if feasible).
|
|
97
|
+
8. **Permalinks.** The textarea content is encoded in the URL hash
|
|
98
|
+
(compressed with `lz-string`) so users can share a reproduction
|
|
99
|
+
case.
|
|
100
|
+
9. **"Get this in your editor" CTA.** A footer banner linking to the
|
|
101
|
+
install instructions (plan 61) for users who want it locally.
|
|
102
|
+
|
|
103
|
+
### Phase 3 — Performance and reliability
|
|
104
|
+
|
|
105
|
+
10. **Lazy dictionary load.** Don't ship all six dictionaries upfront;
|
|
106
|
+
fetch the relevant one on first language detection. Cache for
|
|
107
|
+
subsequent visits.
|
|
108
|
+
11. **Web Worker.** Run Kotoshu in a Web Worker so spell-checking
|
|
109
|
+
doesn't block the UI thread. Debounce by 500 ms.
|
|
110
|
+
12. **Offline support.** Service Worker caches the WASM + dictionaries
|
|
111
|
+
so the playground works without network after first load.
|
|
112
|
+
13. **Memory ceiling.** Browsers tab-dismiss pages that exceed memory
|
|
113
|
+
budgets. Cap resident memory at ~100 MB; refuse to load a second
|
|
114
|
+
dictionary until the first is released.
|
|
115
|
+
|
|
116
|
+
### Phase 4 — Operations
|
|
117
|
+
|
|
118
|
+
14. **Hosting.** GitHub Pages on `kotoshu.github.io`. Subpath
|
|
119
|
+
`/playground/` or a custom subdomain via CNAME.
|
|
120
|
+
15. **CI build.** On release of the library gem, a workflow rebuilds
|
|
121
|
+
the WASM and deploys. Version the playground to match the gem so
|
|
122
|
+
bug reports reference a known state.
|
|
123
|
+
16. **Telemetry.** Privacy-respecting: count unique visits, top
|
|
124
|
+
languages used, top error patterns. No user content logged. Use
|
|
125
|
+
Plausible or self-hosted; never Google Analytics.
|
|
126
|
+
|
|
127
|
+
## Acceptance criteria
|
|
128
|
+
|
|
129
|
+
- `https://kotoshu.github.io/playground/` loads in under 5 s on a
|
|
130
|
+
cold cache (3G simulation), under 1 s on a warm cache.
|
|
131
|
+
- Pasting English prose flags misspellings within 500 ms of typing
|
|
132
|
+
pause.
|
|
133
|
+
- Selecting a suggestion replaces the word in the editor.
|
|
134
|
+
- Sharing a permalink reproduces the exact state.
|
|
135
|
+
- The page works offline after first visit.
|
|
136
|
+
- Memory stays under 100 MB with one language loaded; under 200 MB
|
|
137
|
+
with three.
|
|
138
|
+
|
|
139
|
+
## Dependencies
|
|
140
|
+
|
|
141
|
+
- **Blocked by:**
|
|
142
|
+
- `02-cli-unification` — the playground wraps the library API; a
|
|
143
|
+
clean surface is required
|
|
144
|
+
- `04-language-modules` — for the languages available in the
|
|
145
|
+
playground (only languages whose tokenizer + normalizer is pure
|
|
146
|
+
Ruby can ship in WASM)
|
|
147
|
+
- **Blocks:** nothing — this is a marketing/UX surface, not a
|
|
148
|
+
prerequisite for other features.
|
|
149
|
+
- **Cross-repo:** ships in `kotoshu/kotoshu.github.io`; consumes the
|
|
150
|
+
library gem's WASM build (which the library repo must produce — see
|
|
151
|
+
task 3).
|
|
152
|
+
- **Promotes from:** T6.2 in `56-t4-t5-t6-quality-architecture-ecosystem.md`
|
|
153
|
+
|
|
154
|
+
## Out of scope
|
|
155
|
+
|
|
156
|
+
- The semantic path in-browser. No `onnxruntime` in WASM; revisit if
|
|
157
|
+
ONNX-runtime-on-WASM matures (it's nascent in 2026).
|
|
158
|
+
- A full FastText model (2 GB per language is not browser-feasible).
|
|
159
|
+
- User accounts or saved documents. The playground is for trying, not
|
|
160
|
+
for storing.
|
|
161
|
+
- A full dictionary (every word, every language). The 50K subset is a
|
|
162
|
+
deliberate tradeoff for size; users who need full coverage install
|
|
163
|
+
the gem.
|
|
164
|
+
|
|
165
|
+
## Risks
|
|
166
|
+
|
|
167
|
+
- **ruby.wasm maturity.** ruby.wasm is actively developed; some stdlib
|
|
168
|
+
pieces behave differently (notably `IO`, `File`, `Thread`). Audit
|
|
169
|
+
the Kotoshu codepath for stdlib calls that don't translate and add
|
|
170
|
+
WASM-specific stubs.
|
|
171
|
+
- **Bundle size creep.** Easy to accidentally ship the full gem
|
|
172
|
+
(with onnxruntime glue code) — the build pipeline must tree-shake
|
|
173
|
+
aggressively. Set a CI gate on the WASM blob size.
|
|
174
|
+
- **Browser memory limits.** Mobile Safari is aggressive. Test on a
|
|
175
|
+
real iPhone before claiming "works on mobile."
|
|
176
|
+
- **Dictionary licensing.** Each Hunspell dictionary has its own
|
|
177
|
+
license. Verify the subset bundle is redist-compatible. The
|
|
178
|
+
`dictionaries/` repo already tracks per-language licenses — reuse
|
|
179
|
+
that metadata.
|
|
180
|
+
|
|
181
|
+
## Status
|
|
182
|
+
|
|
183
|
+
_Pending._ Phase 1 is a coherent MVP — could ship before v1.0 of the
|
|
184
|
+
library as part of marketing buildup. The single biggest risk is
|
|
185
|
+
ruby.wasm compatibility; spike task (verify Kotoshu loads in ruby.wasm
|
|
186
|
+
at all) before committing to a deadline.
|
|
@@ -0,0 +1,191 @@
|
|
|
1
|
+
# 64 — HTTP API server & multi-language SDKs
|
|
2
|
+
|
|
3
|
+
Promotes the cross-ecosystem reach idea into a promotable plan. This is
|
|
4
|
+
the "make it big" path for users who don't run Ruby.
|
|
5
|
+
|
|
6
|
+
## Goal
|
|
7
|
+
|
|
8
|
+
A self-hostable HTTP API that wraps the Kotoshu library, plus thin SDK
|
|
9
|
+
clients in Python, JavaScript/TypeScript, and Go so non-Ruby projects
|
|
10
|
+
can use Kotoshu without shelling out to the CLI.
|
|
11
|
+
|
|
12
|
+
```
|
|
13
|
+
┌────────────────┐ ┌─────────────────┐
|
|
14
|
+
│ Python app │ HTTPS │ │
|
|
15
|
+
│ JS app ├────────►│ kotoshu server │──► Kotoshu.check / .suggest
|
|
16
|
+
│ Go app │ │ (Rack/Sinatra) │
|
|
17
|
+
│ Rust app │ │ │
|
|
18
|
+
└────────────────┘ └─────────────────┘
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
The server is a **separate gem** (`kotoshu-server`); SDKs are separate
|
|
22
|
+
repos under `kotoshu/`. The library gem stays focused on
|
|
23
|
+
spell-checking; the server gem holds the HTTP surface.
|
|
24
|
+
|
|
25
|
+
## Why
|
|
26
|
+
|
|
27
|
+
Kotoshu is a Ruby library. The Ruby ecosystem is one of many. Without
|
|
28
|
+
an HTTP entry point:
|
|
29
|
+
|
|
30
|
+
- Python shops (Django, Flask) can't use Kotoshu without a Ruby
|
|
31
|
+
subprocess.
|
|
32
|
+
- JavaScript shops (Node, Deno, Bun) reach for `spellchecker` (node)
|
|
33
|
+
or `cspell` and never hear about Kotoshu.
|
|
34
|
+
- Go shops shell out and parse SARIF — works, but no one's first
|
|
35
|
+
choice.
|
|
36
|
+
- Enterprise stacks (Java/Kotlin, .NET) are simply unreachable from a
|
|
37
|
+
Ruby library.
|
|
38
|
+
|
|
39
|
+
An HTTP API + thin SDKs puts Kotoshu in the same shape as LanguageTool
|
|
40
|
+
(which ships a public HTTP API and is the de-facto standard for this)
|
|
41
|
+
without giving up the "self-hostable" guarantee that the
|
|
42
|
+
privacy-conscious require.
|
|
43
|
+
|
|
44
|
+
This is also the answer to "more use cases": with the API in place,
|
|
45
|
+
the use cases multiply without further Kotoshu work — a browser
|
|
46
|
+
extension, a Slack bot, a WordPress plugin, an Obsidian plugin all
|
|
47
|
+
become one HTTP call away.
|
|
48
|
+
|
|
49
|
+
## Tasks
|
|
50
|
+
|
|
51
|
+
### Phase 1 — HTTP server (`kotoshu-server` gem)
|
|
52
|
+
|
|
53
|
+
1. **New repo `kotoshu/kotoshu-server`.** Rack-based (Sinatra for the
|
|
54
|
+
routing, or pure Rack for minimalism). Depends on `kotoshu`.
|
|
55
|
+
2. **Endpoints.**
|
|
56
|
+
- `POST /v1/check` — body: `{ text, language?, format? }`. Returns
|
|
57
|
+
a `DocumentResult`-shaped JSON (mirrors the CLI's `--format json`).
|
|
58
|
+
- `POST /v1/suggest` — body: `{ word, language?, max? }`. Returns
|
|
59
|
+
a `SuggestionSet`-shaped JSON.
|
|
60
|
+
- `POST /v1/detect` — body: `{ text }`. Returns detected language
|
|
61
|
+
+ confidence.
|
|
62
|
+
- `GET /v1/languages` — lists supported + cached languages.
|
|
63
|
+
- `GET /v1/health` — liveness probe; returns cache state.
|
|
64
|
+
- `GET /v1/version` — library + server versions.
|
|
65
|
+
3. **Resource management.** On startup, the server runs
|
|
66
|
+
`Kotoshu.setup(*configured_languages, want: %i[spelling frequency])`
|
|
67
|
+
so the cache is warm before serving requests. New languages can be
|
|
68
|
+
added at runtime via `POST /v1/admin/setup` (gated behind an admin
|
|
69
|
+
token).
|
|
70
|
+
4. **Semantic mode.** Optional: when ONNX is enabled (plan 05), a
|
|
71
|
+
`model: "hybrid"` query param turns on reranking for that request.
|
|
72
|
+
Default `model: "hunspell"` to bound latency.
|
|
73
|
+
5. **Concurrency model.** Puma or Falcon (Falcon for async I/O —
|
|
74
|
+
better fit for the long-running suggestion pipeline). Multiple
|
|
75
|
+
workers; each worker owns its own `Spellchecker` instance (or
|
|
76
|
+
shares a thread-safe one — verify in `02-cli-unification`).
|
|
77
|
+
6. **Rate limiting + auth.** Optional API key middleware (operator's
|
|
78
|
+
choice). Ship disabled by default; document how to enable.
|
|
79
|
+
7. **Observability.** Structured JSON logs (one line per request);
|
|
80
|
+
Prometheus metrics endpoint at `/v1/metrics`. Latency histograms
|
|
81
|
+
for `check` and `suggest`.
|
|
82
|
+
|
|
83
|
+
### Phase 2 — Docker & ops
|
|
84
|
+
|
|
85
|
+
8. **`kotoshu/server` Docker image.** Multi-arch (amd64, arm64).
|
|
86
|
+
Pre-warmed with the six full-feature languages. Configurable via
|
|
87
|
+
`KOTOSHU_*` env vars.
|
|
88
|
+
9. **Helm chart.** For Kubernetes deployments. Configurable replicas,
|
|
89
|
+
resource limits, language pre-warm list.
|
|
90
|
+
10. **`docker-compose.yml`.** For local dev: one command (`docker
|
|
91
|
+
compose up`) gives a running server on `:9292`.
|
|
92
|
+
11. **Benchmark suite.** `spec/performance/server_bench.rb`:
|
|
93
|
+
- Throughput: requests/sec on a 4-core machine (target: > 200 rps
|
|
94
|
+
for `check` with short text).
|
|
95
|
+
- Latency p50/p95/p99 for `check` and `suggest`.
|
|
96
|
+
- Memory under sustained load (target: < 2 GB resident with
|
|
97
|
+
`hunspell` mode, < 5 GB with `hybrid`).
|
|
98
|
+
|
|
99
|
+
### Phase 3 — SDKs
|
|
100
|
+
|
|
101
|
+
12. **`kotoshu/kotoshu-python`.** PyPI package `kotoshu`.
|
|
102
|
+
- Sync client (`requests`) and async client (`httpx`).
|
|
103
|
+
- Type stubs for IDE autocomplete.
|
|
104
|
+
- Methods: `check(text, language=None)`, `suggest(word, language=None)`,
|
|
105
|
+
`detect(text)`.
|
|
106
|
+
- Streaming client for `POST /v1/check` if/when streaming lands.
|
|
107
|
+
13. **`kotoshu/kotoshu-js`.** npm package `@kotoshu/client`.
|
|
108
|
+
- Works in Node, Deno, Bun, and modern browsers (via `fetch`).
|
|
109
|
+
- TypeScript types.
|
|
110
|
+
- Same method surface as the Python SDK.
|
|
111
|
+
14. **`kotoshu/kotoshu-go`.** Go module.
|
|
112
|
+
- `kotoshu.NewClient(url)` → `.Check(ctx, text, opts)` etc.
|
|
113
|
+
- Idiomatic Go (context.Context everywhere, typed errors).
|
|
114
|
+
15. **SDK contract tests.** A shared OpenAPI spec drives generated
|
|
115
|
+
fixtures; each SDK's test suite runs against the fixtures so the
|
|
116
|
+
SDKs stay in sync with the server.
|
|
117
|
+
|
|
118
|
+
### Phase 4 — Documentation & reach
|
|
119
|
+
|
|
120
|
+
16. **OpenAPI spec** in the server repo, published to
|
|
121
|
+
[`openapis.org`](https://openapis.org) cataloging. Auto-generates
|
|
122
|
+
SDK clients in other languages (Rust, .NET, Java) via
|
|
123
|
+
openapi-generator — best-effort, community-supported.
|
|
124
|
+
17. **Self-hosting guide.** Step-by-step for Docker, Kubernetes, bare
|
|
125
|
+
Ruby. Covers sizing, language pre-warm, and monitoring.
|
|
126
|
+
18. **Privacy guarantee.** Document that self-hosted = your data stays
|
|
127
|
+
in your network. Contrast with LanguageTool's public API.
|
|
128
|
+
19. **Migrating from LanguageTool API.** Side-by-side: same input,
|
|
129
|
+
different response shape; provide a shim if there's demand.
|
|
130
|
+
|
|
131
|
+
## Acceptance criteria
|
|
132
|
+
|
|
133
|
+
- `docker run -p 9292:9292 kotoshu/server` serves `POST /v1/check`
|
|
134
|
+
within 5 s of container start.
|
|
135
|
+
- Throughput > 200 rps for `check` (short text) on a 4-core machine.
|
|
136
|
+
- Each SDK has a one-page quickstart that works against a running
|
|
137
|
+
server.
|
|
138
|
+
- All SDKs pass contract tests against the latest server release.
|
|
139
|
+
- OpenAPI spec validates; auto-generated Rust client compiles.
|
|
140
|
+
- Helm chart deploys to a test cluster and serves requests.
|
|
141
|
+
|
|
142
|
+
## Dependencies
|
|
143
|
+
|
|
144
|
+
- **Blocked by:**
|
|
145
|
+
- `02-cli-unification` — clean library API to wrap
|
|
146
|
+
- `03-dynamic-download` — resource management for the server
|
|
147
|
+
- `05-semantic-path` — for the `hybrid` mode option
|
|
148
|
+
- `09-integrity-security` — for safe download + verification in
|
|
149
|
+
server-managed cache
|
|
150
|
+
- **Blocks:** `11-release-v1` (the server + SDKs are part of the v1
|
|
151
|
+
reach story).
|
|
152
|
+
- **Cross-repo:** new repos `kotoshu/kotoshu-server`,
|
|
153
|
+
`kotoshu/kotoshu-python`, `kotoshu/kotoshu-js`, `kotoshu/kotoshu-go`.
|
|
154
|
+
- **Promotes from:** implied by "every project, every language" in
|
|
155
|
+
`00-vision.md`.
|
|
156
|
+
|
|
157
|
+
## Out of scope
|
|
158
|
+
|
|
159
|
+
- A managed / hosted API (operator concern; document self-hosting
|
|
160
|
+
instead — privacy is the differentiator).
|
|
161
|
+
- Multi-tenant features (per-tenant dictionaries, per-tenant rate
|
|
162
|
+
limits). Operators build that on top if needed.
|
|
163
|
+
- Real-time streaming of partial check results (defer to a v2 if
|
|
164
|
+
there's demand).
|
|
165
|
+
- Auth provider integrations (OAuth, OIDC). Ship an API-key
|
|
166
|
+
middleware; operators wire their own auth in front.
|
|
167
|
+
- A Rust SDK in-repo (the OpenAPI-generated client is best-effort;
|
|
168
|
+
accept community contributions if a maintainer steps up).
|
|
169
|
+
|
|
170
|
+
## Risks
|
|
171
|
+
|
|
172
|
+
- **Operational burden.** A Docker image + Helm chart is a maintenance
|
|
173
|
+
surface. Decide upfront whether the project commits to LTS-style
|
|
174
|
+
releases of the server, or treats it as community-maintained.
|
|
175
|
+
- **API stability.** v1 endpoints must be stable. Freeze the JSON
|
|
176
|
+
shapes before tagging `kotoshu-server` v1.0; thereafter breaking
|
|
177
|
+
changes go to `/v2/*`.
|
|
178
|
+
- **Cold-start cost.** Pre-warming six languages on container start
|
|
179
|
+
means slow boot. Tradeoff: eager warm (fast requests, slow boot) vs
|
|
180
|
+
lazy (fast boot, slow first request). Default eager; document
|
|
181
|
+
`KOTOSHU_SERVER_LAZY=1` for the other mode.
|
|
182
|
+
- **SDK drift.** Three SDKs × ongoing server changes is real work.
|
|
183
|
+
The OpenAPI-generated contract tests (task 15) are the only sane
|
|
184
|
+
way to keep them aligned; budget for maintaining the spec.
|
|
185
|
+
|
|
186
|
+
## Status
|
|
187
|
+
|
|
188
|
+
_Pending._ Phase 1 + 2 is a coherent v0.1; phase 3 SDKs are v0.2; v1.0
|
|
189
|
+
is the API-freeze milestone. The single biggest risk is the operational
|
|
190
|
+
commitment — confirm with maintainers that the project is willing to
|
|
191
|
+
ship and support a server image before starting phase 2.
|
|
@@ -0,0 +1,251 @@
|
|
|
1
|
+
# 65 — Universal Kotoshu: master plan & roadmap
|
|
2
|
+
|
|
3
|
+
Polyglot engine · tiered micro-models · distribution everywhere.
|
|
4
|
+
This is the master plan for "full success": a spell-checking system
|
|
5
|
+
**every user can install in under two commands, on every platform,
|
|
6
|
+
for every language they write** — developers, writers, CI, enterprise,
|
|
7
|
+
and eventually office-suite users.
|
|
8
|
+
|
|
9
|
+
Supersedes the scale of `00-vision.md` (which remains the statement of
|
|
10
|
+
philosophy); this file is the execution roadmap that reaches it.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## 1. Grounding — what we know today
|
|
15
|
+
|
|
16
|
+
| Fact | Source | Implication |
|
|
17
|
+
|---|---|---|
|
|
18
|
+
| Microsoft Word ships **14 MB** of fluency models: one `word_fluency_v2.onnx` + a 168-byte runtime-config JSON, code-signed, on onnxruntime | `/Applications/Microsoft Word.app/Contents/Resources/WordModels.bundle` | Proof that a world-class fluency model is a **single-digit-MB ONNX artifact**. We already run onnxruntime. Same distribution pattern applies 1:1. |
|
|
19
|
+
| Our converted models are **~120 MB/lang** (`.onnx` + vocab JSON), 9 languages converted, sha256 manifest exists | `models-fasttext-onnx/manifest.json` | 10× too heavy for bundling. Needs vocab pruning + dim reduction + int8 quantization → tiers. |
|
|
20
|
+
| ~98 Hunspell dictionaries staged, each dir already has `license` + `package.json` | `dictionaries/` | License metadata foundation exists; must become machine-readable + redistribution-classified. |
|
|
21
|
+
| Gem already downloads → sha256-verifies → caches with TTL, cache-only hot path | `LanguageCache` / `FrequencyCache` / `ModelCache` | The resource *client* is done. What's missing is the resource *contract* (tiers, licenses, lockfiles) and the *small artifacts*. |
|
|
22
|
+
| 12 repos live: gem 0.6.0, lsp/server 0.1.0, go v0.1.0, action v1, 3 SDKs, docker, 3 content repos, site | RubyGems / Go proxy / GitHub | The surfaces exist. The gap is engine portability + model tiers + packaging depth. |
|
|
23
|
+
|
|
24
|
+
**North star.** *Any user, any platform, any language: install in ≤2
|
|
25
|
+
commands, work fully offline, and get tiered accuracy they can choose —
|
|
26
|
+
from a 1 MB dictionary to a 15 MB fluency model to a research-grade
|
|
27
|
+
full model.*
|
|
28
|
+
|
|
29
|
+
### Success metrics (measurable, per cut)
|
|
30
|
+
|
|
31
|
+
- **Languages**: 6 full-feature today → 30 wired at v1 → 98 at v2;
|
|
32
|
+
mini+fluency tiers published for ≥30 languages.
|
|
33
|
+
- **Accuracy**: fluency tier within 2 % top-1 of the full tier on the
|
|
34
|
+
benchmark harness; scores published per language × tier.
|
|
35
|
+
- **Size/time**: editor bundle < 25 MB; cold start < 5 s; CI cache hit
|
|
36
|
+
= zero downloads; air-gapped install = one tarball + one command.
|
|
37
|
+
- **Portability**: conformance suite green on every engine
|
|
38
|
+
implementation, on linux/amd64, linux/arm64, macOS universal,
|
|
39
|
+
windows x64/arm64, and wasm32.
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## 2. Five pillars
|
|
44
|
+
|
|
45
|
+
### Pillar 1 — Resource Spec v1 + registry (the contract)
|
|
46
|
+
|
|
47
|
+
One versioned format that every repo, cache, and SDK speaks:
|
|
48
|
+
|
|
49
|
+
```yaml
|
|
50
|
+
# kotoshu-resource-spec v1 (per language package)
|
|
51
|
+
language: en
|
|
52
|
+
version: 2026.09.1
|
|
53
|
+
license: { name: BSD-3-Clause, redistribution: allowed } # or download-only
|
|
54
|
+
resources:
|
|
55
|
+
spelling: { path: spelling/index.dic, sha256: …, size: 551_762 }
|
|
56
|
+
frequency: { path: frequency/tiers.json, sha256: … }
|
|
57
|
+
model:
|
|
58
|
+
mini: { path: models/mini.onnx, sha256: …, size: ~2_000_000,
|
|
59
|
+
metrics: { top1: 0.91, recall5: 0.98 } }
|
|
60
|
+
fluency: { path: models/fluency.onnx, sha256: …, size: ~14_000_000,
|
|
61
|
+
metrics: { top1: 0.95, recall5: 0.99 } }
|
|
62
|
+
full: { path: models/full.onnx, sha256: …, size: ~120_000_000 }
|
|
63
|
+
signature: cosign …
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
- **New repo `kotoshu/registry`**: one signed index aggregating
|
|
67
|
+
dictionaries + frequency + models manifests; lockfile support
|
|
68
|
+
(`kotoshu.lock`) for reproducible offline installs.
|
|
69
|
+
- License field is load-bearing: `redistribution: allowed` gates what
|
|
70
|
+
may be **bundled** into images/extensions vs download-only.
|
|
71
|
+
- Gem change: `ResourceManager` resolves **tiers** (`want: [:spelling,
|
|
72
|
+
model: :mini]`); `kotoshu export en --tiers mini --out en.tar.zst`
|
|
73
|
+
and `kotoshu install ./en.tar.zst` close the air-gap loop.
|
|
74
|
+
|
|
75
|
+
### Pillar 2 — Tiered micro-models + model factory
|
|
76
|
+
|
|
77
|
+
| Tier | Size target | Contents | Purpose |
|
|
78
|
+
|---|---|---|---|
|
|
79
|
+
| **0 · dict** | 0.5–5 MB | Hunspell dic/aff + frequency tiers | Correctness; all 98 languages; no runtime |
|
|
80
|
+
| **1 · mini** | 1–3 MB | char-ngram-only embeddings, 32–64 d, int8 | OOV vectors + reranking where no model ships today |
|
|
81
|
+
| **2 · fluency** | 5–15 MB | pruned-vocab (top ~200 k) + dim-reduced + int8 embeddings, optional tiny context head | The Word-class tier: context-aware rerank, desert/dessert disambiguation |
|
|
82
|
+
| **3 · full** | 100–120 MB+ | today's `models-fasttext-onnx` artifacts | Research-grade; opt-in |
|
|
83
|
+
|
|
84
|
+
- **New repo `kotoshu/model-factory`**: pipeline `sources → prune →
|
|
85
|
+
reduce → quantize → eval → sign → publish`. Sources: cc/wiki fastText
|
|
86
|
+
vectors, Kelly + OPUS frequencies, dictionaries wordlists.
|
|
87
|
+
Reuses `scripts/convert_fasttext_to_onnx.py` from
|
|
88
|
+
models-fasttext-onnx as the `full` tier path.
|
|
89
|
+
- **Eval harness inside model-factory** (eval/): per-language benchmark
|
|
90
|
+
suites seeded from the Spylls/Hunspell fixtures already ported in the
|
|
91
|
+
gem, plus Wikipedia-typo corpora where available. Metrics: top-1,
|
|
92
|
+
recall@5, MRR, latency, size. **Gate**: a tier only publishes with
|
|
93
|
+
metrics attached; fluency must be within 2 % top-1 of full.
|
|
94
|
+
- Site gains an **accuracy page**: language × tier matrix with sizes
|
|
95
|
+
and scores — honesty as a feature.
|
|
96
|
+
- `auto` tier selection: engine picks the best tier fitting a
|
|
97
|
+
caller-supplied size budget (editor default 15 MB; server default full).
|
|
98
|
+
|
|
99
|
+
### Pillar 3 — Polyglot engine: one Rust core, feature-gated bindings (decision made)
|
|
100
|
+
|
|
101
|
+
Two meanings of "language" — both are goals:
|
|
102
|
+
- **Checked natural languages**: 98 staged (Pillar 1/2 supply them).
|
|
103
|
+
- **Implementation languages**: TS, Ruby, Python, Rust — decided; this is
|
|
104
|
+
the **parsanol blueprint**, copied wholesale because it works in
|
|
105
|
+
production at Ribose today (`parsanol-rs` + `parsanol-ruby`).
|
|
106
|
+
|
|
107
|
+
**Architecture (parsanol model):**
|
|
108
|
+
|
|
109
|
+
- **New repo `kotoshu/kotoshu-rs`** — Cargo workspace; core crate is a
|
|
110
|
+
pure `rlib` (dic/aff parsing, affix expansion, trie/DAFSA lookup,
|
|
111
|
+
suggestion strategies, rerank math, tokenization, resource verify).
|
|
112
|
+
All FFI lives **inside the core** under `ffi/`, behind features:
|
|
113
|
+
`ruby` (magnus), `wasm` (wasm-bindgen + js-sys + panic hook), and a
|
|
114
|
+
**C ABI (`ffi/c.rs`) always available** — Python and any C-calling
|
|
115
|
+
host bind through it. `ffi/shared.rs` defines the batch serialization
|
|
116
|
+
every binding shares (parsanol's 3–5× FFI speedup pattern).
|
|
117
|
+
- **Language packages are thin cdylib shims in their own repos**:
|
|
118
|
+
`kotoshu` gem gains `ext/kotoshu_native` (≈10 lines:
|
|
119
|
+
`#[magnus::init]` calling `kotoshu::ffi::ruby::init`), depending on
|
|
120
|
+
the core via git `features=["ruby"]`. Python adds a PyO3 shim wheel
|
|
121
|
+
(maturin) over the same core; TS gets `@kotoshu/wasm` built from the
|
|
122
|
+
core's `wasm` feature (Node *and* browser); Rust consumes the core
|
|
123
|
+
directly. Each package versions independently (parsanol ships gem
|
|
124
|
+
1.3.0 over core 0.5.1 — same policy).
|
|
125
|
+
- **Pure Ruby stays as the fallback and the semantic reference**: the
|
|
126
|
+
gem works without the ext (`Kotoshu::Native.available?` guards), and
|
|
127
|
+
the existing rspec suite runs against **both backends**
|
|
128
|
+
(`rake compat:ruby` / `compat:native`, parsanol's compat-namespace
|
|
129
|
+
pattern) — the gem's suite *is* the conformance suite.
|
|
130
|
+
- **ONNX: `ort` (pyke/ort v2), not the abandoned `onnxruntime` crate.**
|
|
131
|
+
Policy: inference stays host-injected via an `EmbeddingProvider`
|
|
132
|
+
trait; the core's own optional provider (`feature = "onnx"`) uses
|
|
133
|
+
**`ort` with `load-dynamic`** so it dlopens the libonnxruntime the
|
|
134
|
+
host already ships (Ruby's `onnxruntime` gem, pip's onnxruntime,
|
|
135
|
+
onnxruntime-node) — one runtime per process, no double linking, tiny
|
|
136
|
+
artifacts. Standalone Rust builds (CLI/LSP binaries) use ort's
|
|
137
|
+
bundled binaries.
|
|
138
|
+
- **Distribution (parsanol's low-maintenance path)**: source gem that
|
|
139
|
+
compiles on `gem install` via rb-sys (`create_rust_makefile`,
|
|
140
|
+
stable-API fallback for ruby-head, optional auto toolchain install);
|
|
141
|
+
prebuilt platform gems via rake-compiler-dock only if source builds
|
|
142
|
+
hurt adoption. Core releases automated with release-plz.
|
|
143
|
+
- Copy list from parsanol, as policy: git-rev `[patch.crates-io]` for
|
|
144
|
+
magnus/rb-sys when crates.io lags; `[net] git-fetch-with-cli`; wasm
|
|
145
|
+
rustflags for getrandom; cargo-deny + typos + pre-commit; core
|
|
146
|
+
profile `lto="fat", codegen-units=1`, ext profile `lto="off"`; core
|
|
147
|
+
CI includes `ruby-ffi.yml` and `wasm.yml` so every binding is tested
|
|
148
|
+
from the core repo.
|
|
149
|
+
|
|
150
|
+
### Pillar 4 — Distribution matrix (install in ≤2 commands, everywhere)
|
|
151
|
+
|
|
152
|
+
| Channel | Package | Phase |
|
|
153
|
+
|---|---|---|
|
|
154
|
+
| RubyGems | `kotoshu`, `-lsp`, `-server` | live |
|
|
155
|
+
| Go proxy | `kotoshu-go` | live |
|
|
156
|
+
| GitHub Action / Docker | `action-kotoshu`, CI image → GHCR | live → GHCR at 0.7 |
|
|
157
|
+
| PyPI / npm (SDKs) | `kotoshu`, `@kotoshu/client` | 0.7 (tokens pending) |
|
|
158
|
+
| npm `@kotoshu/wasm` + jsDelivr CDN | browser engine, plays into playground & embeds | 0.8 |
|
|
159
|
+
| VS Code Marketplace + Open VSX, JetBrains | editor extensions bundling fluency tiers (self-contained, <25 MB, no gem install) | 0.9–1.0 |
|
|
160
|
+
| Homebrew tap / winget / choco / scoop / apt via OBS | `kotoshu` CLI + standalone LSP binaries | 1.0–1.1 |
|
|
161
|
+
| OCI (GHCR) | server image + resource bundles as artifacts | 1.0 |
|
|
162
|
+
| Offline bundles | `kotoshu export/install` tarballs, cosign-signed, sbom (CycloneDX), SLSA provenance | 0.7 → 1.0 |
|
|
163
|
+
|
|
164
|
+
Standalone LSP at 0.9 via traveling-ruby/ruby-packer (single binary,
|
|
165
|
+
3 platforms) so editor users never need Ruby; native core later makes
|
|
166
|
+
this trivially static.
|
|
167
|
+
|
|
168
|
+
### Pillar 5 — Audiences & end-to-end journeys
|
|
169
|
+
|
|
170
|
+
| Audience | Need | Surface (→ phase) |
|
|
171
|
+
|---|---|---|
|
|
172
|
+
| Ruby devs | library API | gem (live) |
|
|
173
|
+
| Python/JS/Go devs | typed client | SDKs + server (live → 1.0 at 0.9) |
|
|
174
|
+
| Any-stack devs | no SDK needed | HTTP API + OpenAPI (live) |
|
|
175
|
+
| Editor writers | instant offline squiggles | LSP (live) → **bundled-fluency extension** (0.9–1.0) |
|
|
176
|
+
| CI maintainers | fail builds nicely | Action + Docker (live, mini-tier bake at 0.7) |
|
|
177
|
+
| Enterprise / air-gapped | offline, signed, auditable | export/install bundles + SBOM (0.7 → 1.0) |
|
|
178
|
+
| Multi-lingual writers | every language on demand | 30 wired (1.0) → 98 (2.0) |
|
|
179
|
+
| Office-suite users | proofing like Word | Word add-in / LibreOffice ext exploration (2.0 gate) |
|
|
180
|
+
| Low-resource language communities | contribute a language | dictionaries + model-factory recipes (1.0+) |
|
|
181
|
+
|
|
182
|
+
**Golden journeys** (each must be testable e2e): A) VS Code user
|
|
183
|
+
installs extension → squiggles offline in <25 MB. B) `gem install
|
|
184
|
+
kotoshu && kotoshu setup en` → check. C) CI run with cache hit → SARIF
|
|
185
|
+
annotations, zero downloads. D) air-gapped host: `kotoshu install
|
|
186
|
+
en.tar.zst` after `cosign verify`. E) browser playground running the
|
|
187
|
+
WASM engine on mini/fluency tiers. F) polyglot team: `docker run
|
|
188
|
+
kotoshu/server` + `pip install kotoshu`.
|
|
189
|
+
|
|
190
|
+
**Platform/arch matrix**: linux amd64/arm64, macOS universal2,
|
|
191
|
+
windows x64/arm64, wasm32-wasi (browser); CI builds all; native core
|
|
192
|
+
adds riscv64/mobile for free.
|
|
193
|
+
|
|
194
|
+
---
|
|
195
|
+
|
|
196
|
+
## 3. Repo-by-repo deltas
|
|
197
|
+
|
|
198
|
+
| Repo | Change |
|
|
199
|
+
|---|---|
|
|
200
|
+
| **new** `kotoshu/kotoshu-rs` | the Rust core workspace (see plan 66): pure-rlib engine, `ffi/{c,ruby,wasm}` feature-gated, conformance CI (`ruby-ffi.yml`, `wasm.yml`), release-plz, builds `@kotoshu/wasm` |
|
|
201
|
+
| `kotoshu` (gem) | gains `ext/kotoshu_native` cdylib shim + rb-sys extconf; pure-Ruby stays fallback; `rake compat:{ruby,native}` runs the suite on both backends; plus tier-aware ResourceManager, export/install bundles, lockfile, `auto` tier |
|
|
202
|
+
| `dictionaries` | machine-readable license manifest (name, redistribution flag) — already has per-lang files; aggregate to registry |
|
|
203
|
+
| `frequency-list-kelly` | no change; feed model-factory |
|
|
204
|
+
| `models-fasttext-onnx` | becomes the **full tier** source feeding model-factory; keep publishing |
|
|
205
|
+
| **new** `kotoshu/registry` | signed index, lockfiles, tier metadata, accuracy scores |
|
|
206
|
+
| **new** `kotoshu/model-factory` | prune/reduce/quantize/eval/sign/publish pipeline + benchmark suites |
|
|
207
|
+
| **new** `kotoshu/conformance` | shared golden-vector pack (extracted from the gem's Spylls fixtures); consumed by kotoshu-rs CI and any future implementation |
|
|
208
|
+
| `kotoshu-python` | adds native wheel (PyO3 shim over the core, maturin) alongside the HTTP client |
|
|
209
|
+
| `kotoshu-js` | adds `@kotoshu/wasm` re-export + types (built in kotoshu-rs); HTTP client stays |
|
|
210
|
+
| `kotoshu-server` | tier param on endpoints; resource listing; auth/metrics/Helm at GA |
|
|
211
|
+
| `kotoshu-lsp` | tier selection config; standalone single-binary distribution from the Rust core at 0.9 |
|
|
212
|
+
| `docker-kotoshu-ci` | publish to GHCR; bake mini tier (image stays <300 MB) |
|
|
213
|
+
| `action-kotoshu` | prewarm-mini default |
|
|
214
|
+
| `kotoshu.github.io` | accuracy page, downloads page, journey docs |
|
|
215
|
+
|
|
216
|
+
---
|
|
217
|
+
|
|
218
|
+
## 4. Roadmap (cuts)
|
|
219
|
+
|
|
220
|
+
| Cut | Theme | Key deliverables | Exit criteria |
|
|
221
|
+
|---|---|---|---|
|
|
222
|
+
| **0.7** | Resource Spec + mini tier + core scaffold | spec v1; `registry` repo; license manifest; model-factory alpha producing **mini** for 6 langs; gem tier-aware cache + export/install; GHCR image; SDK registry pushes; **`kotoshu-rs` workspace scaffolded with C ABI + conformance-vector pack v0** | mini tier installable for 6 langs; bundle verified offline; Docker <300 MB; core passes conformance v0 vectors |
|
|
223
|
+
| **0.8** | Fluency tier + core engine | **fluency** (≈14 MB) for 6 langs with eval gates; accuracy page; **kotoshu-rs ports dic/aff + lookup + suggest strategies, conformance green**; `@kotoshu/wasm` published from the core (resolves plan 63 — playground goes client-side) | fluency within 2 % top-1 of full; core matches Ruby backend on the whole suite; playground runs on wasm |
|
|
224
|
+
| **0.9** | Polyglot GA + editors | server 1.0 (auth/metrics/Helm); SDKs 1.0 with contract tests; **gem ships `ext/kotoshu_native` (both backends green); Python native wheel**; **VS Code extension** bundling fluency; **standalone single-binary LSP/CLI from the Rust core** (3 platforms) | dual-backend suite green; extension works offline <25 MB; single binary passes conformance |
|
|
225
|
+
| **1.0** | Kotoshu for developers | 30 languages wired; Neovim/JetBrains distributions; Action GA; cosign+SBOM on all artifacts; docs complete; Homebrew tap + winget | every journey A–F green; 5-signature quality bar met |
|
|
226
|
+
| **1.1–2.0** | Everyone | prebuilt platform gems (rake-compiler-dock) if source installs hurt; 98 languages; **gate:** office-suite add-ins exploration; mobile via wasm/native; auto-tier under size budgets | any user, any platform, any language, ≤2 commands |
|
|
227
|
+
|
|
228
|
+
**Decision gates** (explicit, not defaults): prebuilt-vs-source gems
|
|
229
|
+
(post-1.0, adoption-driven); office add-ins (2.0, only after a design
|
|
230
|
+
spike proves the API story). The native-core question is **decided**
|
|
231
|
+
(plan 66) — WASM and Ruby ext both come from the same `kotoshu-rs`.
|
|
232
|
+
|
|
233
|
+
---
|
|
234
|
+
|
|
235
|
+
## 5. Risks
|
|
236
|
+
|
|
237
|
+
| Risk | Mitigation |
|
|
238
|
+
|---|---|
|
|
239
|
+
| License heterogeneity blocks bundling | redistribution flag in spec; bundle only `allowed`; download-only path stays |
|
|
240
|
+
| Small models lose accuracy | eval gates with published metrics; tiers never silently replace full |
|
|
241
|
+
| Two engines (Ruby + Rust) drift | the gem's suite runs on **both backends** (`rake compat:*`); shared conformance-vector pack in kotoshu-rs CI; pure Ruby remains the reference |
|
|
242
|
+
| Source gem needs a Rust toolchain to install | rb-sys optional auto-install; pure-Ruby fallback always works; prebuilt platform gems gated at post-1.0 |
|
|
243
|
+
| magnus/rb-sys rev churn (Ruby 4.0 prep) | parsanol's patch policy copied: pin git revs in `[patch.crates-io]`, relax when crates.io catches up |
|
|
244
|
+
| ort dynamic-lib version skew with host onnxruntime | `load-dynamic` against onnxruntime 1.x C API (stable); document minimum; standalone Rust uses bundled ort |
|
|
245
|
+
| Registry uptime | GitHub Releases + CDN + lockfiles; caches already tolerate outage |
|
|
246
|
+
| Scope explosion | cuts with exit criteria; gates before big bets; 30-before-98 languages |
|
|
247
|
+
|
|
248
|
+
## Status
|
|
249
|
+
|
|
250
|
+
_Planning._ This document is the master index; per-cut plans get their
|
|
251
|
+
own `TODO.impl/{n}-*.md` when promoted, per house convention.
|