kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# 96 — kotoshu-server: semantic models over the HTTP API
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
The server is dictionary-only: `/v1/check`, `/v1/suggest`,
|
|
6
|
+
`/v1/detect` never load a model — `app.rb` has zero model surface.
|
|
7
|
+
The gem has supported semantic reranking since 0.7.0 (tiers,
|
|
8
|
+
registry, cascade). Every non-Ruby SDK user (python/js/go clients)
|
|
9
|
+
gets NO semantic quality through HTTP even though the engine under
|
|
10
|
+
the server can do it. Found in the wave-4 audit (2026-09-06).
|
|
11
|
+
|
|
12
|
+
## Design
|
|
13
|
+
|
|
14
|
+
- **Boot-time opt-in**: `KOTOSHU_SERVER_MODEL_LANGS` (space-separated,
|
|
15
|
+
e.g. "en de it") + `KOTOSHU_SERVER_MODEL_TIER` (default `fluency`,
|
|
16
|
+
the ecosystem default). On boot (the existing lazy-prewarm pattern —
|
|
17
|
+
PR #1 detached warmup thread), set up spelling+model for the listed
|
|
18
|
+
languages. No implicit downloads ever (the two-stage promise).
|
|
19
|
+
- **`/v1/check`** gains an optional `"model": true|false` request flag
|
|
20
|
+
overriding the server default for that call; reranking rides the
|
|
21
|
+
gem's existing check path (cascade decides when ONNX actually runs).
|
|
22
|
+
- **`/v1/languages`** reports `model: true|false` per language
|
|
23
|
+
(from what is set up in the server's cache).
|
|
24
|
+
- **OpenAPI** (`openapi.yaml`) updated for the new field + response
|
|
25
|
+
shape; README documents the env vars, memory expectations
|
|
26
|
+
(~15 MB/language at fluency), and that the published 0.1.0 gem is
|
|
27
|
+
broken (run from source until the 0.1.1 republish — owner gate).
|
|
28
|
+
- Specs against the real app (existing spec style — the repo has 9
|
|
29
|
+
examples on real setup caches).
|
|
30
|
+
|
|
31
|
+
## Owner gates
|
|
32
|
+
|
|
33
|
+
None for source; the release itself remains the standing 0.1.1 gate.
|
|
34
|
+
|
|
35
|
+
## Status
|
|
36
|
+
|
|
37
|
+
**Implemented (2026-09-06, kotoshu-server PR #3).** /v1/check gains
|
|
38
|
+
"model": true|false (default = language has a model set up; explicit
|
|
39
|
+
true on a model-less language is 422, non-boolean 400); rerank rides
|
|
40
|
+
the gem cascade, memoized per language under a Mutex; per-word
|
|
41
|
+
analyzer failure falls back to traditional suggestions. /v1/languages
|
|
42
|
+
reports model availability. Boot opt-in via KOTOSHU_SERVER_MODEL_LANGS
|
|
43
|
+
+ KOTOSHU_SERVER_MODEL_TIER (default fluency) through the prewarm
|
|
44
|
+
thread; unset = byte-for-byte 0.1.0 behavior; never implicit
|
|
45
|
+
downloads. Gemspec constraint untouched (owner decision): the server
|
|
46
|
+
fail-fasts at boot on kotoshu < 0.7.0 with a clear message; "model":
|
|
47
|
+
true returns 503 there. Specs 15 green against 0.6.0 + 19 against
|
|
48
|
+
local 0.7.0 source. Gem quirk recorded: SemanticAnalyzer raises
|
|
49
|
+
in-vocab ArgumentError Similarity must be 0-1 (worked around
|
|
50
|
+
server-side; gem-side fix pending).
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
# 97 — Executes plan 59: YARD API documentation published
|
|
2
|
+
|
|
3
|
+
## Context
|
|
4
|
+
|
|
5
|
+
Plan 59 has been Pending since before the campaign: the gem is the
|
|
6
|
+
primary audience surface for Ruby developers, its public API is
|
|
7
|
+
large (facade, Spellchecker, two-stage ResourceManager, tiers,
|
|
8
|
+
baselines, integrations), and there is NO published API reference —
|
|
9
|
+
no .yardopts, no docs CI, coverage gaps everywhere.
|
|
10
|
+
|
|
11
|
+
## Job
|
|
12
|
+
|
|
13
|
+
1. `.yardopts` (markup markdown, the right --title, exclude
|
|
14
|
+
spec/tasks; default rule set).
|
|
15
|
+
2. **Coverage pass** on the PUBLIC surface only (no internals
|
|
16
|
+
archaeology): the `Kotoshu` module facade (kotoshu.rb), Cli,
|
|
17
|
+
Spellchecker, ResourceManager/Bundle/SetupResult, Configuration,
|
|
18
|
+
Paths, Documents + Suppressions, PersonalDictionary, Baseline,
|
|
19
|
+
Suggestions::Generator strategies' public constructors, the
|
|
20
|
+
integrations entry points (validator/matchers/tasks/jekyll —
|
|
21
|
+
these are what users paste into their apps). Every public method:
|
|
22
|
+
@param/@return/@example where non-obvious; examples must be ones
|
|
23
|
+
that would run. `bundle exec yard stats` driven to a stated
|
|
24
|
+
coverage number on the public list; undocumented ratio printed.
|
|
25
|
+
3. **CI**: a `docs` workflow building `yard doc` as an artifact on
|
|
26
|
+
main (cheap job). HOSTING decision (gh-pages vs a subpath on
|
|
27
|
+
kotoshu.github.io) is an owner choice — do not deploy anywhere;
|
|
28
|
+
record the two options in the plan Status for the owner.
|
|
29
|
+
4. Cross-link: README.adoc "API documentation" section pointing at
|
|
30
|
+
wherever the owner decides to host (placeholder link text for now).
|
|
31
|
+
|
|
32
|
+
## Constraints
|
|
33
|
+
|
|
34
|
+
Docstring work must not change runtime behavior — comments and
|
|
35
|
+
declarations only. Suite stays 3701/0/27.
|
|
36
|
+
|
|
37
|
+
## Status
|
|
38
|
+
|
|
39
|
+
**Merged via PR #134 (docs/yard-api).**
|
|
40
|
+
|
|
41
|
+
What shipped:
|
|
42
|
+
|
|
43
|
+
- `.yardopts`: title "Kotoshu API Reference", markup markdown,
|
|
44
|
+
README.adoc as the readme, excludes for `^spec/`, `^tasks/`,
|
|
45
|
+
`^ext/`, `^conformance/`, `--hide-void-return` (pragmatic default:
|
|
46
|
+
no hard undocumented-count rule).
|
|
47
|
+
- Coverage pass on the full public list (facade, Cli, Spellchecker,
|
|
48
|
+
ResourceManager/Bundle/SetupResult, Configuration, Paths, Documents
|
|
49
|
+
+ Suppressions, PersonalDictionary, Baseline, Suggestions::Generator,
|
|
50
|
+
integration entry points). Undocumented objects in the covered
|
|
51
|
+
files: 28 before, 0 after; repo-wide 84.72% -> 85.80%. Broken tag
|
|
52
|
+
warnings (`@param` name mismatches etc.) fixed repo-wide so the CI
|
|
53
|
+
gate can start green — includes small docstring-only fixes in
|
|
54
|
+
internals (cspell, metrics_collector, language_cache, strategies,
|
|
55
|
+
pattern matchers, embedding pipeline).
|
|
56
|
+
- CI: `.github/workflows/docs.yml` — yard doc on main + PRs touching
|
|
57
|
+
`lib/**`/`.yardopts`/README, uploads `doc/` as the `yard-docs`
|
|
58
|
+
artifact, fails on warnings indicating broken tags (unknown/duplicate
|
|
59
|
+
parameter names, unknown or malformed tags). Unresolved prose links
|
|
60
|
+
in internal namespaces are known debt (~30 sites) and do not gate.
|
|
61
|
+
- README.adoc gained an "API Documentation" section.
|
|
62
|
+
|
|
63
|
+
HOSTING — owner choice, nothing deployed yet:
|
|
64
|
+
|
|
65
|
+
1. **gh-pages off this repo** — the docs workflow gets a deploy step
|
|
66
|
+
publishing `doc/` to the `gh-pages` branch. Zero new moving parts,
|
|
67
|
+
everything lives in this repo; URL becomes
|
|
68
|
+
`kotoshu.github.io/kotoshu`.
|
|
69
|
+
2. **Subpath on kotoshu.github.io** — the site repo owns the single
|
|
70
|
+
public host and links `/api/` to the reference. YARD's output uses
|
|
71
|
+
relative links so it works under a subpath unchanged; costs a small
|
|
72
|
+
sync job (artifact download or subtree copy) plus a site-repo PR
|
|
73
|
+
per doc rebuild.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# 98 — Campaign cleanup: inventory, executed items, and approvals
|
|
2
|
+
|
|
3
|
+
> Asked by the owner 2026-09-06: "What is the clean up like?" after
|
|
4
|
+
> five waves (plans 77–97) and the release day. This file is the
|
|
5
|
+
> record: what was cleaned, what needs an explicit yes, and what is
|
|
6
|
+
> deliberately kept.
|
|
7
|
+
|
|
8
|
+
## Executed (non-destructive)
|
|
9
|
+
|
|
10
|
+
- The 34 untracked TODO.impl plan files (01–11, 20–76 era) committed
|
|
11
|
+
to the gem repo — the plan corpus is finally fully tracked.
|
|
12
|
+
- This plan file itself.
|
|
13
|
+
|
|
14
|
+
## Keep (actively in use)
|
|
15
|
+
|
|
16
|
+
| Item | Why |
|
|
17
|
+
|---|---|
|
|
18
|
+
| `kotoshu/.claude/worktrees/pr93` | PR #139 is OPEN awaiting owner review |
|
|
19
|
+
| `kotoshu` main checkout untracked files (examples/, docs/, scripts/, test_*.rb — ~90 files) | The owner's pre-campaign material; not mine to commit or delete |
|
|
20
|
+
| `fork` remote in the gem main checkout | Points at the #93 contributor's fork; harmless, useful if follow-ups |
|
|
21
|
+
|
|
22
|
+
## Proposals awaiting explicit approval (destructive)
|
|
23
|
+
|
|
24
|
+
1. **~214 GB of models-repo worktrees** — the big one.
|
|
25
|
+
`batch2` 121 GB + `expand` 73 GB + `nb` 13 GB + `mirrors` 7.2 GB +
|
|
26
|
+
`agent-71` 280 MB. Contents: fastText `.vec.gz`/`.vec` source
|
|
27
|
+
archives (re-downloadable from the CDN; every shipped artifact
|
|
28
|
+
lives in LFS + release assets, sha-verified at each release) plus
|
|
29
|
+
merged working copies. Disposition: `git worktree remove` +
|
|
30
|
+
delete. Nothing irreplaceable; ~200 GB reclaimed.
|
|
31
|
+
2. **16 merged/stale worktrees** across gem (2b, two agent-*, correct3,
|
|
32
|
+
features, wave2, yard, an empty kotoshu-rs dir), kotoshu-rs
|
|
33
|
+
(wasm-models, wheels), site (adoption, flip, integrations, news6,
|
|
34
|
+
rerank, truth2) — ~2 GB, mostly node_modules. Verification scratch
|
|
35
|
+
inside three of them (correct3/scratch, adoption/spike = CDP
|
|
36
|
+
scripts, truth2/.verify) gets relocated to
|
|
37
|
+
`docs/verification-evidence/` in each repo BEFORE removal.
|
|
38
|
+
3. **The `2b` reference branch** (fix/t2-phase2b-compounding — the
|
|
39
|
+
alternate T2-2B implementation kept for comparison). Options:
|
|
40
|
+
keep as-is, or preserve its tip as a local tag
|
|
41
|
+
(`t2-phase2b-reference`) and then remove the worktree.
|
|
42
|
+
4. **Local merged branches** in each repo (rebase-merge leaves the
|
|
43
|
+
local copies behind): `git branch -d` list per repo — merged-only,
|
|
44
|
+
reflog-recoverable.
|
|
45
|
+
|
|
46
|
+
## Deliberately NOT proposed
|
|
47
|
+
|
|
48
|
+
- Yanking the two empty 0.1.0 gems — happens right after their 0.1.1s
|
|
49
|
+
land (owner registrations pending).
|
|
50
|
+
- The ~184 GB figure quoted earlier understated: the true worktree
|
|
51
|
+
footprint is ~216 GB (214 models + ~2 elsewhere).
|
|
52
|
+
- `~/.gem/credentials`, `~/.cargo/credentials.toml`, `~/.pypirc` —
|
|
53
|
+
never touched.
|
|
54
|
+
|
|
55
|
+
## Status
|
|
56
|
+
|
|
57
|
+
**Executed (2026-09-06, owner approved all + rebase-merge).**
|
|
58
|
+
|
|
59
|
+
- PR #139 (the rescued #93 work) rebase-merged: remove_word corruption,
|
|
60
|
+
PlainText length-index sync, threshold fix, cache-only hot path,
|
|
61
|
+
KOTOSHU_OFFLINE env fix — suite 3751/0.
|
|
62
|
+
- Models worktrees (batch2/expand/nb/mirrors/agent-71, ~214 GB
|
|
63
|
+
re-downloadable source archives) removed; worktree table pruned.
|
|
64
|
+
- All campaign worktrees removed across gem/rs/site (two were locked
|
|
65
|
+
by long-dead agent pids — unlocked first); verification scratch from
|
|
66
|
+
correct3 preserved in docs/verification-evidence (PR #142; cached
|
|
67
|
+
dictionaries excluded). The truth2 `.verify` turned out to be a
|
|
68
|
+
Chrome profile dir, not scripts — not preserved, deleted with its
|
|
69
|
+
worktree; adoption spike scripts were one-shot probes, findings live
|
|
70
|
+
in plan 78's status.
|
|
71
|
+
- `2b` reference tip preserved as LOCAL tag `t2-phase2b-reference`
|
|
72
|
+
(not pushed — tags are owner actions); worktree + merged branches
|
|
73
|
+
deleted.
|
|
74
|
+
- Merged local branches deleted per repo via `git branch -d` only.
|
|
75
|
+
The SHA-diverged rebase-merged campaign branches (content verified
|
|
76
|
+
on main via --cherry-pick) and pre-campaign user branches are LEFT
|
|
77
|
+
IN PLACE — force-deleting them was not what was approved.
|
|
78
|
+
- Disk: models .claude/worktrees now 0 B.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Plan 99 — Site truth pass + measured performance documentation
|
|
2
|
+
|
|
3
|
+
## Why
|
|
4
|
+
The docs were written against the versions live at the time; three releases
|
|
5
|
+
later some pages misstate reality. Users deserve the measured numbers the
|
|
6
|
+
perf work produced — no spellchecker site publishes per-language sweep
|
|
7
|
+
latency; we have it, verified, for eight languages.
|
|
8
|
+
|
|
9
|
+
## Work (site repo)
|
|
10
|
+
1. `/docs/performance` (new page): the measured tables — whole-text check
|
|
11
|
+
latency (1-33 ms per 100 words), per-sweep averages post-0.3.2
|
|
12
|
+
(en 45 ms ... pt 423 ms, fr worst-case note), tier sizes (full/fluency/mini),
|
|
13
|
+
and tuning: personal dictionaries, baselines, tier choice guidance
|
|
14
|
+
(`--tier`), backend choice (ruby vs native vs wasm). Link from the
|
|
15
|
+
playground footnote, /languages, and /install.
|
|
16
|
+
2. Truth pass: /docs/clients/action still says the action "installs the gem
|
|
17
|
+
from RubyGems where 0.7.0 gates six languages" — stale; the action now
|
|
18
|
+
rides 0.9.2 (20 full-feature languages, baselines, directory mode).
|
|
19
|
+
Audit every page for version references (0.7.0/0.8.0/0.1.1/wasm 0.2.0)
|
|
20
|
+
and flip to current-with-dates. /playground/server: verify the hosted
|
|
21
|
+
demo state and the run-your-own instructions.
|
|
22
|
+
3. Footer/readout: confirm every registry badge resolves (RubyGems 0.9.2,
|
|
23
|
+
npm 0.3.2, PyPI, crates.io).
|
|
24
|
+
|
|
25
|
+
## Verification
|
|
26
|
+
Rendered text extraction on the built site for stale version strings;
|
|
27
|
+
every measured number in /docs/performance traceable to the 0.3.2 bench
|
|
28
|
+
harness (kotoshu-rs PR #23 table) — no inferred numbers.
|
|
29
|
+
|
|
30
|
+
## Status
|
|
31
|
+
Executed 2026-09-07 — site PR #26: /docs/performance + truth pass. Action gap found and closed via action-kotoshu PR #2 + site PR #27 docs flip.
|