kotoshu 1.0.2-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
data/README.adoc
ADDED
|
@@ -0,0 +1,1292 @@
|
|
|
1
|
+
= Kotoshu: Semantic Spell Checking for Ruby
|
|
2
|
+
|
|
3
|
+
image:https://img.shields.io/gem/v/kotoshu.svg[RubyGems Version]
|
|
4
|
+
image:https://img.shields.io/github/license/kotoshu/kotoshu.svg[License]
|
|
5
|
+
image:https://github.com/kotoshu/kotoshu/actions/workflows/main.yml/badge.svg["CI", link="https://github.com/kotoshu/kotoshu/actions/workflows/main.yml"]
|
|
6
|
+
|
|
7
|
+
== Status
|
|
8
|
+
|
|
9
|
+
Kotoshu is **v0.3.0** — building on the 0.2 cut, this release adds a strict
|
|
10
|
+
two-stage resource model (explicit `setup`, cache-only hot path), XDG base
|
|
11
|
+
directory layout, SHA-256 integrity verification, SARIF output, an
|
|
12
|
+
`--interactive` review loop, and ONNX model pipeline wiring.
|
|
13
|
+
|
|
14
|
+
=== What works in 0.3
|
|
15
|
+
|
|
16
|
+
* Two-stage resource model — `Kotoshu.setup(:en)` then `Kotoshu.correct?("hello")`. The hot path is cache-only and raises `ResourceNotSetupError` on miss; downloads are never implicit.
|
|
17
|
+
* `Kotoshu.check(text, language: "en")` / `Kotoshu.suggest("helo")` — full document check and suggestions
|
|
18
|
+
* `Kotoshu.spellchecker_for(lang)` — per-language checker resolved from the cache (`strict:` is a `Kotoshu.setup` option)
|
|
19
|
+
* `kotoshu check FILE` CLI with these flags:
|
|
20
|
+
** `--language LANG|auto` — any set-up language code, script variants included (`sr-Latn`) (default: `auto`)
|
|
21
|
+
** `--format text|json|sarif`
|
|
22
|
+
** `--offline` — use only cached resources, never download
|
|
23
|
+
** `--strict` — exit 3 if any optional resource (frequency, model) can't load
|
|
24
|
+
** `--interactive` — review each error after the check
|
|
25
|
+
** `--verbose`
|
|
26
|
+
* `kotoshu setup LANGUAGE [LANGUAGE ...]` — pre-warm spelling + frequency + ONNX caches for offline use (`fetch` is kept as a hidden deprecated alias)
|
|
27
|
+
* Local-source setup: `kotoshu setup en --aff path/to.en.aff --dic path/to.en.dic` or `kotoshu setup en --from /path/to/dict/dir/`
|
|
28
|
+
* Exit codes: `0` clean, `1` errors found, `2` usage error, `3` resource setup failed
|
|
29
|
+
* SHA-256 integrity verification (manifest-based, with graceful degradation when manifest is absent)
|
|
30
|
+
* Offline mode via `KOTOSHU_OFFLINE=1` or `--offline`
|
|
31
|
+
* XDG base directory layout — caches in `$XDG_CACHE_HOME/kotoshu/`, config in `$XDG_CONFIG_HOME/kotoshu/`, data in `$XDG_LOCAL_HOME/kotoshu/` (overridable via `KOTOSHU_CACHE_PATH`, `KOTOSHU_CONFIG_PATH`, `KOTOSHU_DATA_PATH`)
|
|
32
|
+
|
|
33
|
+
=== Planned for 0.4+
|
|
34
|
+
|
|
35
|
+
* `--output` (file output redirection)
|
|
36
|
+
* ONNX semantic reranking as default path
|
|
37
|
+
* ≥30 language modules wired
|
|
38
|
+
* Grammar rule packs
|
|
39
|
+
* CJK and RTL language support
|
|
40
|
+
|
|
41
|
+
See link:TODO.impl/00-cut-0.2.md[the 0.2 cut plan], the 0.3 tasks under
|
|
42
|
+
`TODO.impl/`, and link:TODO.impl/00-vision.md[the vision] for the path to 1.0.
|
|
43
|
+
|
|
44
|
+
== Purpose
|
|
45
|
+
|
|
46
|
+
Kotoshu 「言修」 is a pure-Ruby spell checker that aims to work for every
|
|
47
|
+
language by dynamically downloading the right combination of dictionary,
|
|
48
|
+
frequency data, and embedding model on demand.
|
|
49
|
+
|
|
50
|
+
The current release pairs a Ruby port of the Hunspell algorithm
|
|
51
|
+
(traditional morphological lookup + affix rules) with optional FastText
|
|
52
|
+
word embeddings converted to ONNX for context-aware reranking.
|
|
53
|
+
|
|
54
|
+
NOTE: The semantic (ONNX) path is an optional feature. `gem install kotoshu`
|
|
55
|
+
works without `onnxruntime`; install it separately (`gem install onnxruntime`)
|
|
56
|
+
to enable context-aware reranking. Set `KOTOSHU_NO_ONNX=1` to opt back out.
|
|
57
|
+
|
|
58
|
+
== Features
|
|
59
|
+
|
|
60
|
+
NOTE: The list below describes the design vision. See <<status>> for
|
|
61
|
+
exactly what works in 0.2 and what is planned for 0.3+.
|
|
62
|
+
|
|
63
|
+
* <<multi-language, Multi-language support with automatic detection>>
|
|
64
|
+
* <<semantic-analysis, Semantic error detection using word embeddings>> (opt-in via Ruby API in 0.2)
|
|
65
|
+
* <<interactive-mode, Interactive review mode with full navigation>> (planned for 0.3)
|
|
66
|
+
* <<batch-processing, Batch processing for CI/CD>> (JSON in 0.2; SARIF planned for 0.3)
|
|
67
|
+
* <<onnx-models, Fast ONNX inference via ONNX Runtime>>
|
|
68
|
+
* <<document-formats, Support for Markdown, AsciiDoc, and plain text>>
|
|
69
|
+
* <<multiple-models, Multiple analysis models (Hunspell, FastText, Hybrid)>> (Hunspell path only in 0.2)
|
|
70
|
+
|
|
71
|
+
== Architecture
|
|
72
|
+
|
|
73
|
+
Kotoshu is built on a modern, semantic architecture:
|
|
74
|
+
|
|
75
|
+
.Architecture overview
|
|
76
|
+
[source]
|
|
77
|
+
----
|
|
78
|
+
╔═══════════════════════════════════════════════════════════════════╗
|
|
79
|
+
║ Kotoshu Semantic Architecture ║
|
|
80
|
+
╠═══════════════════════════════════════════════════════════════════╣
|
|
81
|
+
║ ║
|
|
82
|
+
║ ┌─────────────────────────────────────────────────────────────┐ ║
|
|
83
|
+
║ │ Interface Layer │ ║
|
|
84
|
+
║ │ ┌─────────────────────┐ ┌─────────────────────────────┐ │ ║
|
|
85
|
+
║ │ │ CLI (Thor) │ │ Ruby API │ │ ║
|
|
86
|
+
║ │ │ lib/kotoshu/cli/ │ │ Kotoshu module methods │ │ ║
|
|
87
|
+
║ │ └──────────┬──────────┘ └───────────┬─────────────────┘ │ ║
|
|
88
|
+
║ │ │ Auto Language Detect │ │ ║
|
|
89
|
+
║ └─────────────┼──────────────────────────┼─────────────────────┘ ║
|
|
90
|
+
║ │ │ ║
|
|
91
|
+
║ ▼ ▼ ║
|
|
92
|
+
║ ┌─────────────────────────────────────────────────────────────┐ ║
|
|
93
|
+
║ │ Analysis Layer │ ║
|
|
94
|
+
║ │ ┌──────────────┐ ┌─────────────┐ ┌───────────────────┐ │ ║
|
|
95
|
+
║ │ │ Hunspell │ │ FastText │ │ Hybrid (Best!) │ │ ║
|
|
96
|
+
║ │ │ Dictionary │ │ Embeddings │ │ Combined │ │ ║
|
|
97
|
+
║ │ │ (Traditional)│ │ (ONNX) │ │ Approach │ │ ║
|
|
98
|
+
║ │ └──────────────┘ └─────────────┘ └───────────────────┘ │ ║
|
|
99
|
+
║ └───────────────────────────┬─────────────────────────────────┘ ║
|
|
100
|
+
║ │ ║
|
|
101
|
+
║ ┌───────────────────────────▼─────────────────────────────────┐ ║
|
|
102
|
+
║ │ Model Layer (ONNX) │ ║
|
|
103
|
+
║ │ ┌──────────────────────────────────────────────────────┐ │ ║
|
|
104
|
+
║ │ │ ONNX Runtime → Fast Embedding Lookup │ │ ║
|
|
105
|
+
║ │ │ Semantic Similarity → Context-Aware Suggestions │ │ ║
|
|
106
|
+
║ │ │ Nearest Neighbor Search → Smart Corrections │ │ ║
|
|
107
|
+
║ │ └──────────────────────────────────────────────────────┘ │ ║
|
|
108
|
+
║ └─────────────────────────────────────────────────────────────┘ ║
|
|
109
|
+
║ ║
|
|
110
|
+
╚═══════════════════════════════════════════════════════════════════╝
|
|
111
|
+
----
|
|
112
|
+
|
|
113
|
+
=== Key Components
|
|
114
|
+
|
|
115
|
+
* `Kotoshu::Models::OnnxModel`: ONNX-based word embedding model for fast
|
|
116
|
+
semantic similarity and nearest neighbor search.
|
|
117
|
+
|
|
118
|
+
* `Kotoshu::Analyzers::SemanticAnalyzer`: Unified semantic error detection
|
|
119
|
+
using word embeddings (no artificial spelling/grammar split).
|
|
120
|
+
|
|
121
|
+
* `Kotoshu::Language::LanguageIdentifier`: Automatic language detection
|
|
122
|
+
using FastText LID model (127 languages).
|
|
123
|
+
|
|
124
|
+
* `Kotoshu::Cli::InteractiveReviewer`: Interactive CLI for error review
|
|
125
|
+
with full navigation (forward, backward, jump, skip, accept).
|
|
126
|
+
|
|
127
|
+
* `Kotoshu::Dictionary::Hunspell`: Traditional Hunspell dictionary backend
|
|
128
|
+
for morphological analysis and affix rules.
|
|
129
|
+
|
|
130
|
+
== Why ONNX?
|
|
131
|
+
|
|
132
|
+
ONNX Runtime provides:
|
|
133
|
+
|
|
134
|
+
* **Performance**: C++ implementation, 10-100x faster than pure Ruby
|
|
135
|
+
* **Portability**: Works on CPU, GPU, TPU, mobile devices
|
|
136
|
+
* **Optimization**: Automatic graph optimization and quantization
|
|
137
|
+
* **Interoperability**: Models can be trained in Python, deployed in Ruby
|
|
138
|
+
|
|
139
|
+
Kotoshu uses FastText models converted to ONNX format for semantic spell checking.
|
|
140
|
+
|
|
141
|
+
[[semantic-analysis]]
|
|
142
|
+
== Semantic Analysis
|
|
143
|
+
|
|
144
|
+
Unlike traditional spell checkers that only check dictionary membership and
|
|
145
|
+
edit distance, Kotoshu uses semantic similarity to:
|
|
146
|
+
|
|
147
|
+
* Detect contextually appropriate corrections ("desert" vs "dessert")
|
|
148
|
+
* Handle out-of-vocabulary words via subword embeddings
|
|
149
|
+
* Provide ranked suggestions based on semantic similarity
|
|
150
|
+
* Support compound words and morphological variations
|
|
151
|
+
|
|
152
|
+
.Usage example
|
|
153
|
+
[example]
|
|
154
|
+
====
|
|
155
|
+
[source,ruby]
|
|
156
|
+
----
|
|
157
|
+
Kotoshu.setup(:en, want: %i[spelling model]) # one-time per language
|
|
158
|
+
|
|
159
|
+
# Traditional: knows "helo" is wrong and lists edit-distance candidates
|
|
160
|
+
Kotoshu.suggest("helo").to_words
|
|
161
|
+
# => ["hello", "help", "held", "hell", "hole"]
|
|
162
|
+
|
|
163
|
+
# Semantic: reranks candidates by context similarity
|
|
164
|
+
model = Kotoshu::Models::OnnxModel.from_github("en")
|
|
165
|
+
analyzer = Kotoshu::Analyzers::SemanticAnalyzer.new(model)
|
|
166
|
+
analyzer.suggest_corrections("helo", context: "I said helo to the world").map(&:word)
|
|
167
|
+
# => ["hello"] # "hello" makes more sense in greeting context
|
|
168
|
+
----
|
|
169
|
+
====
|
|
170
|
+
|
|
171
|
+
NOTE: The semantic path requires the optional `onnxruntime` gem. See
|
|
172
|
+
<<_requirements,Requirements>>.
|
|
173
|
+
|
|
174
|
+
[[confidence-cascade]]
|
|
175
|
+
=== Confidence cascade
|
|
176
|
+
|
|
177
|
+
When the traditional (dictionary + edit distance) strategies are already
|
|
178
|
+
confident about a word, the semantic rerank adds latency and model load
|
|
179
|
+
without changing the answer. The confidence cascade formalizes the
|
|
180
|
+
hybrid split: dictionary verdict first, ONNX rerank only for uncertain
|
|
181
|
+
candidates.
|
|
182
|
+
|
|
183
|
+
The `semantic_cascade_threshold` configuration option controls it:
|
|
184
|
+
|
|
185
|
+
[cols="1,3"]
|
|
186
|
+
|===
|
|
187
|
+
|Threshold |Behavior
|
|
188
|
+
|
|
189
|
+
|`1.0` (default)
|
|
190
|
+
|Always rerank — identical to the classic pipeline. This is the sentinel:
|
|
191
|
+
the composite can legitimately score exactly `1.0`, so `1.0` never skips.
|
|
192
|
+
|
|
193
|
+
|`0.9`
|
|
194
|
+
|Rerank only when the top traditional candidate scores below `0.9`;
|
|
195
|
+
otherwise skip the ONNX rerank for that word.
|
|
196
|
+
|
|
197
|
+
|`0.0`
|
|
198
|
+
|Never rerank — the semantic path is bypassed entirely.
|
|
199
|
+
|===
|
|
200
|
+
|
|
201
|
+
[source,bash]
|
|
202
|
+
----
|
|
203
|
+
# Tune via environment variable
|
|
204
|
+
KOTOSHU_SEMANTIC_CASCADE_THRESHOLD=0.9 kotoshu check document.txt
|
|
205
|
+
|
|
206
|
+
# Or via the Ruby API
|
|
207
|
+
Kotoshu.configuration.semantic_cascade_threshold = 0.9
|
|
208
|
+
----
|
|
209
|
+
|
|
210
|
+
Each skip is counted in the metrics registry
|
|
211
|
+
(`Kotoshu::Metrics.stats[:semantic_cascade_skips]`, also exported in
|
|
212
|
+
StatsD and Prometheus formats) and logged under debug/verbose output.
|
|
213
|
+
|
|
214
|
+
Tuning guidance will come from the eval harness once the corpus
|
|
215
|
+
benchmark lands; until then, `0.8`–`0.9` is a reasonable range for
|
|
216
|
+
typo-heavy text.
|
|
217
|
+
|
|
218
|
+
[[multi-language]]
|
|
219
|
+
== Multi-Language Support
|
|
220
|
+
|
|
221
|
+
Kotoshu serves every language staged in the dictionaries manifest —
|
|
222
|
+
36 full-feature codes with per-language modules (script-aware
|
|
223
|
+
tokenizer, national keyboard grid, engine-verified specimens) and the
|
|
224
|
+
script-fallback basic tier for the rest; see the
|
|
225
|
+
<<language-matrix, Language Support Matrix>> for the breakdown.
|
|
226
|
+
Semantic reranking adds an ONNX model where the models registry pins
|
|
227
|
+
one (50+ languages), for example:
|
|
228
|
+
|
|
229
|
+
* *de* - German (Deutsch)
|
|
230
|
+
* *en* - English
|
|
231
|
+
* *es* - Spanish (Español)
|
|
232
|
+
* *fr* - French (Français)
|
|
233
|
+
* *pt* - Portuguese (Português)
|
|
234
|
+
* *ru* - Russian (Русский)
|
|
235
|
+
|
|
236
|
+
Automatic language detection is enabled by default:
|
|
237
|
+
|
|
238
|
+
.Usage example
|
|
239
|
+
[example]
|
|
240
|
+
====
|
|
241
|
+
[source,bash]
|
|
242
|
+
----
|
|
243
|
+
# Language auto-detected from document content
|
|
244
|
+
kotoshu check document.txt
|
|
245
|
+
# Detected: en (95% confidence)
|
|
246
|
+
# Analyzing document.txt (language: en)...
|
|
247
|
+
|
|
248
|
+
# Explicit language specification
|
|
249
|
+
kotoshu check document.txt --language de
|
|
250
|
+
----
|
|
251
|
+
====
|
|
252
|
+
|
|
253
|
+
[[onnx-models]]
|
|
254
|
+
== ONNX Models
|
|
255
|
+
|
|
256
|
+
Kotoshu uses FastText crawl vectors converted to ONNX format:
|
|
257
|
+
|
|
258
|
+
* Source: https://fasttext.cc/docs/en/crawl-vectors.html[FastText Crawl Vectors]
|
|
259
|
+
* Format: ONNX with optimized runtime
|
|
260
|
+
* Vocabulary: 2 million words per language (full coverage)
|
|
261
|
+
* Dimension: 300-dimensional word vectors
|
|
262
|
+
* Size: ~2.4GB per language
|
|
263
|
+
|
|
264
|
+
=== FastText File Formats
|
|
265
|
+
|
|
266
|
+
FastText provides two file formats. Kotoshu uses the `.vec` format for ONNX conversion.
|
|
267
|
+
|
|
268
|
+
[cols="1,1,2"]
|
|
269
|
+
|===
|
|
270
|
+
|Aspect |`.vec` (Text) |`.bin` (Binary)
|
|
271
|
+
|
|
272
|
+
|Content
|
|
273
|
+
|Word vectors only (pre-computed embeddings)
|
|
274
|
+
|Full FastText model (trained model)
|
|
275
|
+
|
|
276
|
+
|Structure
|
|
277
|
+
|Text: one word + 300 floats per line
|
|
278
|
+
|Binary: complete model with matrices
|
|
279
|
+
|
|
280
|
+
|File Size
|
|
281
|
+
|~1.3GB compressed (~2.4GB uncompressed)
|
|
282
|
+
|~1.8GB compressed (~4.8GB uncompressed)
|
|
283
|
+
|
|
284
|
+
|Train New Words
|
|
285
|
+
|✗ No (static lookup only)
|
|
286
|
+
|✓ Yes (can train/OOV with subword info)
|
|
287
|
+
|
|
288
|
+
|Subword Embeddings
|
|
289
|
+
|✗ No
|
|
290
|
+
|✓ Yes (n-gram character embeddings)
|
|
291
|
+
|
|
292
|
+
|ONNX Converter
|
|
293
|
+
|✓ Supported (what we use)
|
|
294
|
+
|✗ Not supported
|
|
295
|
+
|
|
296
|
+
|Use Case
|
|
297
|
+
|Simple word vector lookup for spell checking
|
|
298
|
+
|Full FastText functionality (training, OOV)
|
|
299
|
+
|===
|
|
300
|
+
|
|
301
|
+
Kotoshu uses `.vec` files because:
|
|
302
|
+
|
|
303
|
+
* Simpler extraction: Just word → vector mapping
|
|
304
|
+
* No subword complexity needed: Dictionary-based spell checking doesn't require OOV generation
|
|
305
|
+
* Smaller ONNX models: ~2.4GB vs ~4.8GB
|
|
306
|
+
* Faster conversion: Direct serialization to ONNX
|
|
307
|
+
|
|
308
|
+
.Model management
|
|
309
|
+
[example]
|
|
310
|
+
====
|
|
311
|
+
[source,bash]
|
|
312
|
+
----
|
|
313
|
+
# Set up a language with spelling + ONNX semantic model
|
|
314
|
+
kotoshu setup en --want spelling,model
|
|
315
|
+
|
|
316
|
+
# List what's set up in the cache
|
|
317
|
+
kotoshu setup --list
|
|
318
|
+
|
|
319
|
+
# Re-validate cached resources
|
|
320
|
+
kotoshu cache validate
|
|
321
|
+
----
|
|
322
|
+
====
|
|
323
|
+
|
|
324
|
+
NOTE: FastText `.vec` → ONNX conversion is done upstream in the
|
|
325
|
+
https://github.com/kotoshu/models-fasttext-onnx[`kotoshu/models-fasttext-onnx']
|
|
326
|
+
repo. The CLI downloads pre-converted artifacts; users do not run
|
|
327
|
+
conversion locally.
|
|
328
|
+
|
|
329
|
+
[[model-tiers]]
|
|
330
|
+
=== Model tiers
|
|
331
|
+
|
|
332
|
+
ONNX models come in three tiers, published by
|
|
333
|
+
https://github.com/kotoshu/models-fasttext-onnx[models-fasttext-onnx]
|
|
334
|
+
in a resource registry under stable ids (`kotoshu://models/{lang}/{tier}`):
|
|
335
|
+
|
|
336
|
+
[cols="1,2,3"]
|
|
337
|
+
|===
|
|
338
|
+
|Tier |Approx. size |What it is
|
|
339
|
+
|
|
340
|
+
|`full`
|
|
341
|
+
|~120 MB
|
|
342
|
+
|Unquantized, ~100k-word vocabulary. The same artifact and cache layout
|
|
343
|
+
as before tiers existed.
|
|
344
|
+
|
|
345
|
+
|`fluency`
|
|
346
|
+
|~15-18 MB
|
|
347
|
+
|int8-quantized, reduced vocabulary (~50k-60k words).
|
|
348
|
+
|
|
349
|
+
|`mini`
|
|
350
|
+
|~3 MB
|
|
351
|
+
|int8-quantized, 10k-word vocabulary.
|
|
352
|
+
|===
|
|
353
|
+
|
|
354
|
+
Tiered downloads are verified: the model bytes are fetched from the
|
|
355
|
+
registry entry's primary URL, falling back to its mirror, with the vocab
|
|
356
|
+
sibling downloaded alongside; the SHA-256 is checked against the
|
|
357
|
+
registry entry before the model is accepted into the cache.
|
|
358
|
+
|
|
359
|
+
.Picking a tier
|
|
360
|
+
[example]
|
|
361
|
+
====
|
|
362
|
+
[source,ruby]
|
|
363
|
+
----
|
|
364
|
+
# One-time setup at a specific tier
|
|
365
|
+
Kotoshu.setup(:en, want: %i[model], tier: :fluency)
|
|
366
|
+
|
|
367
|
+
# Resolve stays cache-only: the requested tier must already be cached,
|
|
368
|
+
# otherwise ResourceNotSetupError is raised (no download, no fallback
|
|
369
|
+
# to another tier)
|
|
370
|
+
Kotoshu.resolve(language: :en, want: %i[spelling model], tier: :fluency)
|
|
371
|
+
|
|
372
|
+
# "Is this tier cached?" (tier: :any asks whether any tier is cached)
|
|
373
|
+
Kotoshu.setup?(:en, resource: :model, tier: :mini)
|
|
374
|
+
----
|
|
375
|
+
|
|
376
|
+
[source,bash]
|
|
377
|
+
----
|
|
378
|
+
kotoshu setup en --model --tier mini
|
|
379
|
+
----
|
|
380
|
+
====
|
|
381
|
+
|
|
382
|
+
`tier: :any` on resolve maps to the *single* cached tier. It resolves
|
|
383
|
+
when exactly one tier is cached and raises otherwise: with no tier
|
|
384
|
+
cached it raises `ResourceNotSetupError`, and with several it raises
|
|
385
|
+
`Kotoshu::ResourceResolutionError` listing them. That listing uses the
|
|
386
|
+
fixed order `mini`, `fluency`, `full` — a reporting order for
|
|
387
|
+
deterministic error messages, not a fallback chain; tiers are never
|
|
388
|
+
silently substituted for one another.
|
|
389
|
+
|
|
390
|
+
The default tier is `fluency` (~15 MB, near-lossless; owner decision
|
|
391
|
+
2026-09-04) — caches written under the old `full` default keep working
|
|
392
|
+
via the tier-less legacy bridge. Override it per call with `tier:`, or set
|
|
393
|
+
it globally via the `model_tier` configuration option (environment
|
|
394
|
+
variable `KOTOSHU_MODEL_TIER`; one of `full`, `fluency`, `mini`):
|
|
395
|
+
|
|
396
|
+
[source,bash]
|
|
397
|
+
----
|
|
398
|
+
KOTOSHU_MODEL_TIER=mini kotoshu setup en --model
|
|
399
|
+
----
|
|
400
|
+
|
|
401
|
+
[[interactive-mode]]
|
|
402
|
+
[[interactive-mode]]
|
|
403
|
+
== Interactive Mode
|
|
404
|
+
|
|
405
|
+
NOTE: Interactive mode shipped in 0.3.0. It is navigation-only — the
|
|
406
|
+
session records which suggestions the user accepted but does not rewrite
|
|
407
|
+
the source file yet.
|
|
408
|
+
|
|
409
|
+
[source,bash]
|
|
410
|
+
----
|
|
411
|
+
kotoshu check README.md --interactive
|
|
412
|
+
----
|
|
413
|
+
|
|
414
|
+
Features in 0.3:
|
|
415
|
+
|
|
416
|
+
* **Navigate**: [n] / Enter next, [p] previous, [l] list
|
|
417
|
+
* **Accept**: [1-9] record suggestion N for the current error
|
|
418
|
+
* **Skip**: [s] skip the current error
|
|
419
|
+
* **Quit**: [q] exit the review loop
|
|
420
|
+
|
|
421
|
+
[[batch-processing]]
|
|
422
|
+
== Batch Processing
|
|
423
|
+
|
|
424
|
+
For CI/CD and automation, Kotoshu supports JSON and SARIF output in 0.3;
|
|
425
|
+
`--output` file redirection is planned for 0.4+.
|
|
426
|
+
|
|
427
|
+
.JSON output for CI/CD
|
|
428
|
+
[example]
|
|
429
|
+
====
|
|
430
|
+
[source,bash]
|
|
431
|
+
----
|
|
432
|
+
# JSON output to stdout (supported in 0.3)
|
|
433
|
+
kotoshu check README.md --format json
|
|
434
|
+
|
|
435
|
+
# SARIF 2.1.0 output (supported in 0.3)
|
|
436
|
+
kotoshu check README.md --format sarif
|
|
437
|
+
|
|
438
|
+
# Exit code for CI
|
|
439
|
+
kotoshu check README.md
|
|
440
|
+
echo $? # 0 if no errors, 1 if errors found
|
|
441
|
+
----
|
|
442
|
+
====
|
|
443
|
+
|
|
444
|
+
[[ignoring-words]]
|
|
445
|
+
== Ignoring Words Inline
|
|
446
|
+
Words that are intentionally "wrong" - product names, code identifiers
|
|
447
|
+
quoted in prose, dialect - can be suppressed in place instead of being
|
|
448
|
+
added to the personal dictionary:
|
|
449
|
+
----
|
|
450
|
+
kotoshu:disable-line suppress everything on this line
|
|
451
|
+
kotoshu:disable-next-line suppress the following line
|
|
452
|
+
kotoshu:disable-next-line foo bar suppress only these words
|
|
453
|
+
kotoshu:disable-file / enable-file block suppression (nestable)
|
|
454
|
+
----
|
|
455
|
+
Each format recognizes the directive in its own comment syntax, and the
|
|
456
|
+
directive line itself is never spellchecked:
|
|
457
|
+
[options="header"]
|
|
458
|
+
|===
|
|
459
|
+
|Format |Form |Trailing directive
|
|
460
|
+
|
|
461
|
+
|Plain text
|
|
462
|
+
|a bare line: `kotoshu:disable-line`
|
|
463
|
+
|not supported
|
|
464
|
+
|
|
465
|
+
|Markdown
|
|
466
|
+
|`<!-- kotoshu:disable-next-line -->`
|
|
467
|
+
|supported
|
|
468
|
+
|
|
469
|
+
|AsciiDoc
|
|
470
|
+
|`// kotoshu:disable-next-line`
|
|
471
|
+
|supported
|
|
472
|
+
|
|
473
|
+
|===
|
|
474
|
+
|
|
475
|
+
The Ruby API picks this up automatically:
|
|
476
|
+
`Kotoshu.spellchecker_for("en").check(text)` moves suppressed words to
|
|
477
|
+
`result.suppressed_errors` (each marked `suppressed: true` and
|
|
478
|
+
`suppressed_by: "inline"`) so only real errors stay in `result.errors`.
|
|
479
|
+
Pass `--show-suppressed` to `kotoshu check` to list them.
|
|
480
|
+
|
|
481
|
+
[[ci-baselines]]
|
|
482
|
+
== CI Baselines
|
|
483
|
+
Freeze existing spelling debt so CI fails only on NEW errors:
|
|
484
|
+
[source,bash]
|
|
485
|
+
----
|
|
486
|
+
# Record current findings: one entry per file+word with a count,
|
|
487
|
+
# stable across reformatting
|
|
488
|
+
kotoshu baseline init README.md docs/*.adoc
|
|
489
|
+
|
|
490
|
+
# CI: errors covered by the baseline pass, new errors fail
|
|
491
|
+
kotoshu check README.md --baseline .kotoshu-baseline.json
|
|
492
|
+
----
|
|
493
|
+
A summary line reports baseline entries that have gone stale (the error
|
|
494
|
+
no longer exists), so the debt shrinks visibly; refresh by re-running
|
|
495
|
+
`kotoshu baseline init`. JSON output adds `suppressedCount` /
|
|
496
|
+
`suppressedErrors` plus a `baseline` block, and SARIF marks
|
|
497
|
+
baseline-suppressed results with a `suppressions` entry
|
|
498
|
+
(`justification: "baseline"`).
|
|
499
|
+
|
|
500
|
+
[[pre-commit]]
|
|
501
|
+
== Git pre-commit Hook
|
|
502
|
+
The gem repository ships a https://pre-commit.com[pre-commit] hook
|
|
503
|
+
definition (`.pre-commit-hooks.yaml`):
|
|
504
|
+
[source,yaml]
|
|
505
|
+
----
|
|
506
|
+
repos:
|
|
507
|
+
- repo: https://github.com/kotoshu/kotoshu
|
|
508
|
+
rev: v0.7.0 # pin to a released tag
|
|
509
|
+
hooks:
|
|
510
|
+
- id: kotoshu
|
|
511
|
+
----
|
|
512
|
+
NOTE: The hook uses `language: system`: it runs the `kotoshu` executable
|
|
513
|
+
from PATH, so it honestly requires a Ruby environment with the gem
|
|
514
|
+
installed (`gem install kotoshu`, then `kotoshu setup en`). A
|
|
515
|
+
WebAssembly client (in development) is the future zero-Ruby path.
|
|
516
|
+
|
|
517
|
+
[[document-formats]]
|
|
518
|
+
== Document Formats
|
|
519
|
+
|
|
520
|
+
Kotoshu supports structured documents with AST parsing:
|
|
521
|
+
|
|
522
|
+
* *Plain text*: Line-based error detection
|
|
523
|
+
* *Markdown*: AST-based using Kramdown parser
|
|
524
|
+
* *AsciiDoc*: AST-based using Asciidoctor parser
|
|
525
|
+
|
|
526
|
+
Structured documents preserve node paths for precise error location.
|
|
527
|
+
|
|
528
|
+
[[multiple-models]]
|
|
529
|
+
== Analysis Models
|
|
530
|
+
|
|
531
|
+
NOTE: In 0.2, the CLI runs the Hunspell traditional path only.
|
|
532
|
+
The `--model` flag and FastText/Hybrid paths are planned for 0.3+.
|
|
533
|
+
The Ruby API can opt into the semantic path today via
|
|
534
|
+
`Kotoshu::Models::OnnxModel` (auto-available when `onnxruntime` is installed).
|
|
535
|
+
|
|
536
|
+
Kotoshu is designed to support three analysis models:
|
|
537
|
+
|
|
538
|
+
.Dictionary backend comparison
|
|
539
|
+
[cols="2,2,2"]
|
|
540
|
+
|===
|
|
541
|
+
| Model | Description | Best For
|
|
542
|
+
|
|
543
|
+
| *hunspell* | Traditional dictionary-based with morphological rules | Fast checking, compound words, languages with complex morphology
|
|
544
|
+
|
|
545
|
+
| *fasttext* | Pure semantic embeddings via ONNX | Context awareness, out-of-vocabulary words, semantic similarity
|
|
546
|
+
|
|
547
|
+
| *hybrid* | Hunspell candidates + FastText reranking (recommended) | Maximum accuracy, best of both worlds
|
|
548
|
+
|===
|
|
549
|
+
|
|
550
|
+
.Intended usage (0.3+)
|
|
551
|
+
[example]
|
|
552
|
+
====
|
|
553
|
+
[source,bash]
|
|
554
|
+
----
|
|
555
|
+
# Fast dictionary-based checking (default in 0.2)
|
|
556
|
+
kotoshu check document.txt # 0.2: Hunspell path
|
|
557
|
+
|
|
558
|
+
# Semantic / hybrid paths: planned for 0.3
|
|
559
|
+
# kotoshu check document.txt --model fasttext
|
|
560
|
+
# kotoshu check document.txt --model hybrid
|
|
561
|
+
----
|
|
562
|
+
====
|
|
563
|
+
|
|
564
|
+
== Installation
|
|
565
|
+
|
|
566
|
+
Add this line to your application's Gemfile:
|
|
567
|
+
|
|
568
|
+
[source,ruby]
|
|
569
|
+
----
|
|
570
|
+
gem 'kotoshu'
|
|
571
|
+
----
|
|
572
|
+
|
|
573
|
+
And then execute:
|
|
574
|
+
|
|
575
|
+
[source,sh]
|
|
576
|
+
----
|
|
577
|
+
bundle install
|
|
578
|
+
----
|
|
579
|
+
|
|
580
|
+
Or install it yourself as:
|
|
581
|
+
|
|
582
|
+
[source,sh]
|
|
583
|
+
----
|
|
584
|
+
gem install kotoshu
|
|
585
|
+
----
|
|
586
|
+
|
|
587
|
+
NOTE: `onnxruntime` is an optional dependency. Install it separately
|
|
588
|
+
(`gem install onnxruntime`) to enable semantic analysis; the
|
|
589
|
+
traditional Hunspell path works without it.
|
|
590
|
+
|
|
591
|
+
== Quick Start
|
|
592
|
+
|
|
593
|
+
[source,sh]
|
|
594
|
+
----
|
|
595
|
+
# One-time per language: download spelling dictionary from
|
|
596
|
+
# github.com/kotoshu/dictionaries (idempotent, ~5 MB)
|
|
597
|
+
kotoshu setup en
|
|
598
|
+
|
|
599
|
+
# Then check files instantly, cache-only
|
|
600
|
+
kotoshu check README.md
|
|
601
|
+
----
|
|
602
|
+
|
|
603
|
+
Or skip the explicit setup — the CLI will prompt interactively the
|
|
604
|
+
first time you check a file in a non-cached language (TTY only; in
|
|
605
|
+
non-TTY or `KOTOSHU_OFFLINE=1` mode it exits with code 3).
|
|
606
|
+
|
|
607
|
+
.Command-line usage
|
|
608
|
+
[source,bash]
|
|
609
|
+
----
|
|
610
|
+
# Check a file (uses --language, or auto-detects from content)
|
|
611
|
+
kotoshu check README.md
|
|
612
|
+
|
|
613
|
+
# Explicit language
|
|
614
|
+
kotoshu check README.md --language en
|
|
615
|
+
|
|
616
|
+
# JSON output for programmatic use
|
|
617
|
+
kotoshu check README.md --format json
|
|
618
|
+
|
|
619
|
+
# Offline mode — use only cached dictionaries, never download
|
|
620
|
+
kotoshu check README.md --offline
|
|
621
|
+
|
|
622
|
+
# Check stdin
|
|
623
|
+
echo "helo wrld" | kotoshu check
|
|
624
|
+
----
|
|
625
|
+
|
|
626
|
+
Exit codes: `0` (no errors), `1` (errors found), `2` (usage error),
|
|
627
|
+
`3` (language not set up — run `kotoshu setup LANG`, or run `kotoshu check`
|
|
628
|
+
in a TTY to be prompted).
|
|
629
|
+
|
|
630
|
+
.Ruby API usage
|
|
631
|
+
[source,ruby]
|
|
632
|
+
----
|
|
633
|
+
require 'kotoshu'
|
|
634
|
+
|
|
635
|
+
# Stage 1: set up the language once (downloads from github.com/kotoshu/dictionaries)
|
|
636
|
+
Kotoshu.setup(:en)
|
|
637
|
+
|
|
638
|
+
# Stage 2: hot-path checks are cache-only and never touch the network
|
|
639
|
+
Kotoshu.correct?("hello") # => true
|
|
640
|
+
Kotoshu.correct?("helo") # => false
|
|
641
|
+
|
|
642
|
+
# Suggestions return a SuggestionSet; call #to_words for an Array
|
|
643
|
+
Kotoshu.suggest("helo").to_words # => ["hello", "help", "held", ...]
|
|
644
|
+
|
|
645
|
+
# Check a document
|
|
646
|
+
result = Kotoshu.check("Hello wrold")
|
|
647
|
+
result.errors.map(&:word) # => ["wrold"]
|
|
648
|
+
|
|
649
|
+
# Each error carries position + suggestions
|
|
650
|
+
result = Kotoshu.check_file("README.md")
|
|
651
|
+
result.errors.each do |error|
|
|
652
|
+
puts "#{error.word} at offset #{error.position}: #{error.top_suggestions(3).join(', ')}"
|
|
653
|
+
end
|
|
654
|
+
|
|
655
|
+
# Semantic analysis is optional — requires the onnxruntime gem
|
|
656
|
+
# (gem install onnxruntime). Skip this block if you only want Hunspell.
|
|
657
|
+
if Kotoshu::Models::OnnxModel::ONNX_LOADED
|
|
658
|
+
Kotoshu.setup(:en, want: %i[spelling model])
|
|
659
|
+
model = Kotoshu::Models::OnnxModel.from_github('en')
|
|
660
|
+
analyzer = Kotoshu::Analyzers::SemanticAnalyzer.new(model)
|
|
661
|
+
analyzer.analyze(Kotoshu.check("Hello wrold"))
|
|
662
|
+
end
|
|
663
|
+
----
|
|
664
|
+
|
|
665
|
+
NOTE: The library API is strict: calls like `Kotoshu.correct?` raise
|
|
666
|
+
`Kotoshu::ResourceNotSetupError` until you've run `Kotoshu.setup`. This
|
|
667
|
+
prevents surprise downloads on metered networks. The CLI (`kotoshu check`)
|
|
668
|
+
intercepts the error and prompts to download interactively.
|
|
669
|
+
|
|
670
|
+
== Requirements
|
|
671
|
+
|
|
672
|
+
* Ruby 3.1+
|
|
673
|
+
* `onnxruntime` gem (optional — enables semantic spell checking; install separately with `gem install onnxruntime`)
|
|
674
|
+
* Python 3 + fasttext (optional, only if you want to convert `.vec` → `.onnx` upstream)
|
|
675
|
+
|
|
676
|
+
== Resource Caching and Language Support
|
|
677
|
+
|
|
678
|
+
Kotoshu uses a sophisticated multi-layer caching system to manage dictionaries,
|
|
679
|
+
frequency lists, and embedding models. Resources are downloaded explicitly via
|
|
680
|
+
`Kotoshu.setup` (or `kotoshu setup`) and cached under the XDG base directory
|
|
681
|
+
layout (`~/.cache/kotoshu/` by default; override via `KOTOSHU_CACHE_PATH`,
|
|
682
|
+
`KOTOSHU_CONFIG_PATH`, `KOTOSHU_DATA_PATH`, or the `XDG_*_HOME` vars).
|
|
683
|
+
|
|
684
|
+
=== Cache Architecture
|
|
685
|
+
|
|
686
|
+
.Cache System Class Diagram
|
|
687
|
+
[source]
|
|
688
|
+
----
|
|
689
|
+
┌────────────────────────────────────────────────────────────────────────────┐
|
|
690
|
+
│ BaseCache (Abstract) │
|
|
691
|
+
│ ┌────────────────────────────────────────────────────────────────────┐ │
|
|
692
|
+
│ │ Common: download, metadata, validation, stats, TTL management │ │
|
|
693
|
+
│ └────────────────────────────────────────────────────────────────────┘ │
|
|
694
|
+
└────────────────────┬───────────────────┬────────────────────┬──────────────┘
|
|
695
|
+
│ │ │
|
|
696
|
+
┌────────────▼────────┐ ┌──────▼──────┐ ┌───────▼─────────┐
|
|
697
|
+
│ LanguageCache │ │ModelCache │ │ FrequencyCache │
|
|
698
|
+
│ (Dictionaries) │ │ (Embeddings)│ │ (Kelly Lists) │
|
|
699
|
+
└─────────────────────┘ └─────────────┘ └─────────────────┘
|
|
700
|
+
│ │ │
|
|
701
|
+
┌────────────▼────────┐ ┌──────▼──────┐ ┌───────▼─────────┐
|
|
702
|
+
│ ~/.cache/kotoshu/ │ │~/.cache/ │ │ ~/.cache/kotoshu/│
|
|
703
|
+
│ languages/ │ │ kotoshu/ │ │frequency-lists/ │
|
|
704
|
+
│ │ │ models/ │ │ │
|
|
705
|
+
└─────────────────────┘ └─────────────┘ └─────────────────┘
|
|
706
|
+
----
|
|
707
|
+
|
|
708
|
+
=== Cache Types
|
|
709
|
+
|
|
710
|
+
==== LanguageCache (Dictionaries)
|
|
711
|
+
|
|
712
|
+
Manages Hunspell dictionaries and grammar rules for spell checking.
|
|
713
|
+
|
|
714
|
+
* **Cache Path**: `~/.cache/kotoshu/languages/{code}/`
|
|
715
|
+
* **TTL**: 7 days (604,800 seconds)
|
|
716
|
+
* **Source**: https://github.com/kotoshu/dictionaries[kotoshu/dictionaries]
|
|
717
|
+
* **Resources per language**:
|
|
718
|
+
** `spelling/`**: Hunspell dictionary (`index.dic`, `index.aff`)
|
|
719
|
+
** `grammar/`**: Grammar rules (`rules.yaml`) - *future*
|
|
720
|
+
** `frequency/`**: Frequency data - *deprecated, use FrequencyCache*
|
|
721
|
+
|
|
722
|
+
.Usage
|
|
723
|
+
[source,ruby]
|
|
724
|
+
----
|
|
725
|
+
# Access via cache
|
|
726
|
+
cache = Kotoshu::Cache::LanguageCache.new
|
|
727
|
+
dict = cache.get_spelling('en')
|
|
728
|
+
|
|
729
|
+
# Result:
|
|
730
|
+
# {
|
|
731
|
+
# aff_path: "~/.cache/kotoshu/languages/en/spelling/index.aff",
|
|
732
|
+
# dic_path: "~/.cache/kotoshu/languages/en/spelling/index.dic",
|
|
733
|
+
# cached: true,
|
|
734
|
+
# metadata: { ... }
|
|
735
|
+
# }
|
|
736
|
+
----
|
|
737
|
+
|
|
738
|
+
==== FrequencyCache (Kelly Project)
|
|
739
|
+
|
|
740
|
+
Manages Kelly Project frequency lists for intelligent suggestion ranking.
|
|
741
|
+
|
|
742
|
+
* **Cache Path**: `~/.cache/kotoshu/frequency-lists/{code}/`
|
|
743
|
+
* **TTL**: 7 days (604,800 seconds)
|
|
744
|
+
* **Source**: https://github.com/kotoshu/frequency-list-kelly[kotoshu/frequency-list-kelly]
|
|
745
|
+
* **Format**: JSON with tiered word frequency data
|
|
746
|
+
|
|
747
|
+
.Kelly Frequency Data Structure
|
|
748
|
+
[source,json]
|
|
749
|
+
----
|
|
750
|
+
{
|
|
751
|
+
"metadata": {
|
|
752
|
+
"language": "en",
|
|
753
|
+
"source": "Kelly Project (University of Leeds)",
|
|
754
|
+
"total_words_analyzed": 1500000
|
|
755
|
+
},
|
|
756
|
+
"tiers": {
|
|
757
|
+
"top_50": {
|
|
758
|
+
"words": ["the", "be", "to", "of", "and", ...],
|
|
759
|
+
"info": "Most common 50 words"
|
|
760
|
+
},
|
|
761
|
+
"top_200": {
|
|
762
|
+
"words": ["will", "my", "one", "all", ...],
|
|
763
|
+
"info": "Most common 200 words"
|
|
764
|
+
},
|
|
765
|
+
"top_1000": {
|
|
766
|
+
"words": ["however", "although", ...],
|
|
767
|
+
"info": "Most common 1000 words"
|
|
768
|
+
}
|
|
769
|
+
}
|
|
770
|
+
}
|
|
771
|
+
----
|
|
772
|
+
|
|
773
|
+
.Usage
|
|
774
|
+
[source,ruby]
|
|
775
|
+
----
|
|
776
|
+
# Access via cache
|
|
777
|
+
cache = Kotoshu::Cache::FrequencyCache.new
|
|
778
|
+
freq_data = cache.get('en', force_download: true)
|
|
779
|
+
|
|
780
|
+
# Result:
|
|
781
|
+
# {
|
|
782
|
+
# frequency_path: "~/.cache/kotoshu/frequency-lists/en/frequency.json",
|
|
783
|
+
# tiers: {
|
|
784
|
+
# top_50: Set<...>,
|
|
785
|
+
# top_200: Set<...>,
|
|
786
|
+
# top_1000: Set<...>
|
|
787
|
+
# },
|
|
788
|
+
# metadata: { ... }
|
|
789
|
+
# }
|
|
790
|
+
|
|
791
|
+
# Integrated into EditDistanceStrategy
|
|
792
|
+
strategy = Kotoshu::Suggestions::Strategies::EditDistanceStrategy.new(
|
|
793
|
+
language_code: 'en'
|
|
794
|
+
)
|
|
795
|
+
strategy.frequency_bonus('the') # => 200 (top 50)
|
|
796
|
+
strategy.frequency_bonus('hello') # => 100 (top 200)
|
|
797
|
+
strategy.frequency_bonus('xyz') # => 0 (not in lists)
|
|
798
|
+
----
|
|
799
|
+
|
|
800
|
+
==== ModelCache (Embedding Models)
|
|
801
|
+
|
|
802
|
+
Manages FastText and ONNX embedding models for semantic spell checking.
|
|
803
|
+
|
|
804
|
+
* **Cache Path**: `~/.cache/kotoshu/models/{code}/models/{type}/`
|
|
805
|
+
* **TTL**: 30 days (2,592,000 seconds)
|
|
806
|
+
* **Sources**:
|
|
807
|
+
** FastText (.vec): Facebook CDN (https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/[dl.fbaipublicfiles.com])
|
|
808
|
+
** ONNX (.onnx): Converted locally from FastText models
|
|
809
|
+
* **Supported Types**:
|
|
810
|
+
** `fasttext`: FastText word vectors (.vec files, 300D) - Downloaded from Facebook CDN
|
|
811
|
+
** `onnx`: ONNX-converted models (.onnx files) - Auto-converted from FastText
|
|
812
|
+
|
|
813
|
+
NOTE: ONNX models are automatically converted from FastText models on first use.
|
|
814
|
+
The conversion uses `lib/kotoshu/scripts/fasttext_to_onnx.py` and requires Python 3 with
|
|
815
|
+
`numpy` and `onnx` packages installed.
|
|
816
|
+
|
|
817
|
+
.Model Files by Language
|
|
818
|
+
[width="100%",cols="3,m"]
|
|
819
|
+
|=========================================
|
|
820
|
+
| Language | FastText File | ONNX File |
|
|
821
|
+
|=========================================
|
|
822
|
+
| de (German) | cc.de.300.vec | fasttext.de.onnx |
|
|
823
|
+
| en (English) | cc.en.300.vec | fasttext.en.onnx |
|
|
824
|
+
| es (Spanish) | cc.es.300.vec | fasttext.es.onnx |
|
|
825
|
+
| fr (French) | cc.fr.300.vec | fasttext.fr.onnx |
|
|
826
|
+
| pt (Portuguese) | cc.pt.300.vec | fasttext.pt.onnx |
|
|
827
|
+
| ru (Russian) | cc.ru.300.vec | fasttext.ru.onnx |
|
|
828
|
+
|=========================================|
|
|
829
|
+
|
|
830
|
+
=== CLI Cache Management
|
|
831
|
+
|
|
832
|
+
Kotoshu provides CLI commands for managing cached resources:
|
|
833
|
+
|
|
834
|
+
[source,bash]
|
|
835
|
+
----
|
|
836
|
+
# List all cached resources
|
|
837
|
+
kotoshu cache list
|
|
838
|
+
|
|
839
|
+
# List specific cache type
|
|
840
|
+
kotoshu cache list language
|
|
841
|
+
kotoshu cache list model
|
|
842
|
+
kotoshu cache list frequency
|
|
843
|
+
|
|
844
|
+
# Show cache statistics
|
|
845
|
+
kotoshu cache status
|
|
846
|
+
|
|
847
|
+
# Show detailed status (verbose)
|
|
848
|
+
kotoshu cache status --verbose
|
|
849
|
+
|
|
850
|
+
# Download a resource
|
|
851
|
+
kotoshu cache download language en
|
|
852
|
+
kotoshu cache download model en:fasttext
|
|
853
|
+
kotoshu cache download frequency en
|
|
854
|
+
|
|
855
|
+
# Get information about a resource
|
|
856
|
+
kotoshu cache info language en
|
|
857
|
+
kotoshu cache info model en:fasttext
|
|
858
|
+
kotoshu cache info frequency en
|
|
859
|
+
|
|
860
|
+
# Purge cached data
|
|
861
|
+
kotoshu cache purge all
|
|
862
|
+
kotoshu cache purge language en
|
|
863
|
+
kotoshu cache purge frequency
|
|
864
|
+
|
|
865
|
+
# Clean expired entries
|
|
866
|
+
kotoshu cache clean
|
|
867
|
+
----
|
|
868
|
+
|
|
869
|
+
=== Cache Statistics
|
|
870
|
+
|
|
871
|
+
Each cache type tracks statistics:
|
|
872
|
+
|
|
873
|
+
* **Hits**: Number of cache hits (resource found locally)
|
|
874
|
+
* **Misses**: Number of cache misses (had to download)
|
|
875
|
+
* **Hit Rate**: Percentage of cache hits
|
|
876
|
+
* **Size**: Total disk space used
|
|
877
|
+
* **Cached Resources**: Number of resources cached
|
|
878
|
+
|
|
879
|
+
[source,bash]
|
|
880
|
+
----
|
|
881
|
+
$ kotoshu cache status
|
|
882
|
+
======================================================================
|
|
883
|
+
Kotoshu Cache Status
|
|
884
|
+
======================================================================
|
|
885
|
+
|
|
886
|
+
Language Cache:
|
|
887
|
+
Directory: /Users/username/.cache/kotoshu/languages
|
|
888
|
+
Resources cached: 2
|
|
889
|
+
Size: 2.45 MB
|
|
890
|
+
Hits: 15, Misses: 2
|
|
891
|
+
Hit rate: 88.2%
|
|
892
|
+
|
|
893
|
+
Frequency Cache:
|
|
894
|
+
Directory: /Users/username/.cache/kotoshu/frequency-lists
|
|
895
|
+
Resources cached: 1
|
|
896
|
+
Size: 815.84 KB
|
|
897
|
+
Hits: 42, Misses: 1
|
|
898
|
+
Hit rate: 97.7%
|
|
899
|
+
|
|
900
|
+
Model Cache:
|
|
901
|
+
Directory: /Users/username/.cache/kotoshu/models
|
|
902
|
+
Resources cached: 0
|
|
903
|
+
Size: 0 B
|
|
904
|
+
Hits: 0, Misses: 0
|
|
905
|
+
Hit rate: 0.0%
|
|
906
|
+
|
|
907
|
+
Total:
|
|
908
|
+
Total size: 3.26 MB
|
|
909
|
+
Overall hit rate: 93.5%
|
|
910
|
+
======================================================================
|
|
911
|
+
----
|
|
912
|
+
|
|
913
|
+
=== Shell Completion
|
|
914
|
+
|
|
915
|
+
Kotoshu ships tab-completion scripts for bash, zsh, and fish. The scripts
|
|
916
|
+
complete subcommands for `kotoshu <TAB>` and language codes for
|
|
917
|
+
`kotoshu setup <TAB>` (the language list is read dynamically via
|
|
918
|
+
`kotoshu completions languages`, so newly registered languages appear
|
|
919
|
+
without re-running the install step).
|
|
920
|
+
|
|
921
|
+
Install for your shell:
|
|
922
|
+
|
|
923
|
+
[source,bash]
|
|
924
|
+
----
|
|
925
|
+
# bash (system-wide)
|
|
926
|
+
kotoshu completions bash > /etc/bash_completion.d/kotoshu
|
|
927
|
+
|
|
928
|
+
# bash (single user)
|
|
929
|
+
kotoshu completions bash > ~/.local/share/bash-completion/completions/kotoshu
|
|
930
|
+
|
|
931
|
+
# zsh — drop into any directory on your $fpath
|
|
932
|
+
kotoshu completions zsh > "${fpath[1]}/_kotoshu"
|
|
933
|
+
|
|
934
|
+
# fish
|
|
935
|
+
kotoshu completions fish > ~/.config/fish/completions/kotoshu.fish
|
|
936
|
+
----
|
|
937
|
+
|
|
938
|
+
To see the supported language codes without installing a completion
|
|
939
|
+
script:
|
|
940
|
+
|
|
941
|
+
[source,bash]
|
|
942
|
+
----
|
|
943
|
+
kotoshu completions languages
|
|
944
|
+
----
|
|
945
|
+
|
|
946
|
+
[[language-matrix]]
|
|
947
|
+
=== Language Support Matrix
|
|
948
|
+
|
|
949
|
+
Kotoshu serves languages in tiers. Every language staged in the
|
|
950
|
+
kotoshu/dictionaries manifest downloads through `kotoshu setup` and
|
|
951
|
+
checks through the Hunspell engine. Region variants fold onto their
|
|
952
|
+
base dictionary (`en-GB` resolves `en`, `es-MX` resolves `es`);
|
|
953
|
+
script variants keep a dictionary of their own (`sr-Latn` stages the
|
|
954
|
+
Latin Serbian dictionary alongside the Cyrillic `sr`).
|
|
955
|
+
|
|
956
|
+
.Language Support Matrix
|
|
957
|
+
[width="100%",cols="^2,^3,^3",options="header"]
|
|
958
|
+
|=========================================
|
|
959
|
+
| Tier | Languages | What it carries |
|
|
960
|
+
|=========================================
|
|
961
|
+
| Full feature (35 languages, 36 codes) | ar bg ca cs da de el en es et fa fr he hr hu id it ko lt lv nb ne nl nn pl pt ro ru sk sl sr sr-Latn sv tr uk vi | Per-language module: staged dictionary + Hunspell affix rules, script-aware tokenizer, national keyboard grid for typo-proximity suggestions, engine-verified specimens |
|
|
962
|
+
| Basic (26 languages) | br cy eo eu fo fur fy ga gd gl hy hyw ia ie is ka la lb ltg mk mn nds oc rw tk tlh | Staged dictionary + script-fallback word extraction and the generic keyboard |
|
|
963
|
+
| Kelly frequency (8 languages) | ar el en it no ru sv zh | Top-1000 frequency list wired into suggestion ranking, on top of either tier |
|
|
964
|
+
|=========================================+
|
|
965
|
+
|
|
966
|
+
ONNX semantic models (opt-in, `--model`) resolve per the registry the
|
|
967
|
+
link:https://github.com/kotoshu/models-fasttext-onnx[models repo]
|
|
968
|
+
pins — 50+ languages across the tiers, in full/fluency/mini sizes.
|
|
969
|
+
|
|
970
|
+
.Dictionary Sources
|
|
971
|
+
[width="100%",cols="^1,^1,^1,^1",options="header"]
|
|
972
|
+
|=========================================
|
|
973
|
+
| Language | Word Count | License | Source |
|
|
974
|
+
|=========================================
|
|
975
|
+
| de (German) | 75,873 | GPL | igerman98 |
|
|
976
|
+
| en (English) | 49,568 | LGPL/MPL/GPL | SCOWL |
|
|
977
|
+
| es (Spanish) | 57,344 | GPL | LibreOffice |
|
|
978
|
+
| fr (French) | 84,310 | MPL 2.0 | Grammalecte |
|
|
979
|
+
| pt (Portuguese) | 312,368 | LGPLv3 + MPL | VERO |
|
|
980
|
+
| ru (Russian) | 146,269 | BSD-style | Alexander Lebedev |
|
|
981
|
+
|=========================================+
|
|
982
|
+
|
|
983
|
+
Staged dictionaries for the other tiers carry their provenance in the
|
|
984
|
+
dictionaries repo manifest (`LICENSE` and source per file; LibreOffice
|
|
985
|
+
/ wooorm-dictionaries derivations dominate).
|
|
986
|
+
|
|
987
|
+
.Kelly Frequency Lists
|
|
988
|
+
[width="100%",cols="^1,^1,^1",options="header"]
|
|
989
|
+
|=========================================
|
|
990
|
+
| Language | Size | Coverage |
|
|
991
|
+
|=========================================
|
|
992
|
+
| ar (Arabic) | ~750 KB | Top 1000 words |
|
|
993
|
+
| zh (Chinese) | ~800 KB | Top 1000 words |
|
|
994
|
+
| en (English) | 815 KB | Top 1000 words |
|
|
995
|
+
| el (Greek) | ~780 KB | Top 1000 words |
|
|
996
|
+
| it (Italian) | ~790 KB | Top 1000 words |
|
|
997
|
+
| no (Norwegian) | ~770 KB | Top 1000 words |
|
|
998
|
+
| ru (Russian) | 780 KB | Top 1000 words |
|
|
999
|
+
| sv (Swedish) | ~775 KB | Top 1000 words |
|
|
1000
|
+
|=========================================+
|
|
1001
|
+
|
|
1002
|
+
NOTE: Kelly frequency lists provide the top 1000 most common words from
|
|
1003
|
+
the Kelly Project (University of Leeds & University of Gothenburg).
|
|
1004
|
+
Languages not listed here require external frequency data sources.
|
|
1005
|
+
|
|
1006
|
+
=== Programmatic Usage
|
|
1007
|
+
|
|
1008
|
+
==== Using Language Cache
|
|
1009
|
+
|
|
1010
|
+
[source,ruby]
|
|
1011
|
+
----
|
|
1012
|
+
require 'kotoshu/cache/language_cache'
|
|
1013
|
+
|
|
1014
|
+
cache = Kotoshu::Cache::LanguageCache.new
|
|
1015
|
+
|
|
1016
|
+
# Get spelling dictionary
|
|
1017
|
+
dict = cache.get_spelling('en')
|
|
1018
|
+
puts "Dictionary: #{dict[:dic_path]}"
|
|
1019
|
+
puts "Words: #{File.readlines(dict[:dic_path]).count}"
|
|
1020
|
+
|
|
1021
|
+
# Get available languages
|
|
1022
|
+
cache.available_languages # => ["de", "en", "es", "fr", "pt", "ru"]
|
|
1023
|
+
|
|
1024
|
+
# Check if resource is cached
|
|
1025
|
+
cache.available?('en:spelling') # => true
|
|
1026
|
+
|
|
1027
|
+
# Get language info
|
|
1028
|
+
info = cache.language_info('en')
|
|
1029
|
+
puts "Language: #{info[:name]}"
|
|
1030
|
+
puts "Words: #{info[:word_count]}"
|
|
1031
|
+
puts "License: #{info[:license]}"
|
|
1032
|
+
----
|
|
1033
|
+
|
|
1034
|
+
==== Using Frequency Cache
|
|
1035
|
+
|
|
1036
|
+
[source,ruby]
|
|
1037
|
+
----
|
|
1038
|
+
require 'kotoshu/cache/frequency_cache'
|
|
1039
|
+
|
|
1040
|
+
cache = Kotoshu::Cache::FrequencyCache.new
|
|
1041
|
+
|
|
1042
|
+
# Get frequency data
|
|
1043
|
+
freq_data = cache.get('en')
|
|
1044
|
+
|
|
1045
|
+
# Access frequency tiers
|
|
1046
|
+
top_50 = freq_data[:tiers][:top_50]
|
|
1047
|
+
top_50.include?('the') # => true
|
|
1048
|
+
top_50.include?('hello') # => true (in top 200)
|
|
1049
|
+
|
|
1050
|
+
# Get available languages
|
|
1051
|
+
cache.available_languages # => ["ar", "zh", "en", "el", "it", "no", "ru", "sv"]
|
|
1052
|
+
----
|
|
1053
|
+
|
|
1054
|
+
==== Integration with Suggestion Strategies
|
|
1055
|
+
|
|
1056
|
+
[source,ruby]
|
|
1057
|
+
----
|
|
1058
|
+
require 'kotoshu/suggestions/strategies/edit_distance_strategy'
|
|
1059
|
+
|
|
1060
|
+
# Frequency bonuses automatically applied
|
|
1061
|
+
strategy = Kotoshu::Suggestions::Strategies::EditDistanceStrategy.new(
|
|
1062
|
+
language_code: 'en'
|
|
1063
|
+
)
|
|
1064
|
+
|
|
1065
|
+
# Suggestions are ranked by frequency
|
|
1066
|
+
suggestions = strategy.suggest('helo', max_results: 5)
|
|
1067
|
+
# => [
|
|
1068
|
+
# { word: "hello", score: 1200 }, # High frequency word
|
|
1069
|
+
# { word: "help", score: 1150 }, # Medium frequency word
|
|
1070
|
+
# ...
|
|
1071
|
+
# ]
|
|
1072
|
+
----
|
|
1073
|
+
|
|
1074
|
+
=== Cache TTL and Expiration
|
|
1075
|
+
|
|
1076
|
+
All cached resources have a Time-To-Live (TTL) and automatically expire:
|
|
1077
|
+
|
|
1078
|
+
* **LanguageCache**: 7 days (dictionaries change infrequently)
|
|
1079
|
+
* **FrequencyCache**: 7 days (frequency lists are stable)
|
|
1080
|
+
* **ModelCache**: 30 days (models are large and change rarely)
|
|
1081
|
+
|
|
1082
|
+
Expired resources are automatically re-downloaded on next access.
|
|
1083
|
+
|
|
1084
|
+
[source,ruby]
|
|
1085
|
+
----
|
|
1086
|
+
cache = Kotoshu::Cache::FrequencyCache.new
|
|
1087
|
+
|
|
1088
|
+
# Force re-download (ignores cache)
|
|
1089
|
+
freq_data = cache.get('en', force_download: true)
|
|
1090
|
+
|
|
1091
|
+
# Clean expired entries manually
|
|
1092
|
+
cache.clean
|
|
1093
|
+
----
|
|
1094
|
+
|
|
1095
|
+
=== Manual Cache Management
|
|
1096
|
+
|
|
1097
|
+
[source,ruby]
|
|
1098
|
+
----
|
|
1099
|
+
cache = Kotoshu::Cache::LanguageCache.new
|
|
1100
|
+
|
|
1101
|
+
# Clear specific resource
|
|
1102
|
+
cache.clear('en:spelling')
|
|
1103
|
+
|
|
1104
|
+
# Clear all resources
|
|
1105
|
+
cache.clear_all
|
|
1106
|
+
|
|
1107
|
+
# Check if resource exists
|
|
1108
|
+
cache.available?('en:spelling') # => true after download
|
|
1109
|
+
|
|
1110
|
+
# Get statistics
|
|
1111
|
+
stats = cache.stats
|
|
1112
|
+
puts "Hit rate: #{stats[:hit_rate] * 100}%"
|
|
1113
|
+
puts "Size: #{stats[:size_bytes]} bytes"
|
|
1114
|
+
----
|
|
1115
|
+
|
|
1116
|
+
=== GitHub Repository Structure
|
|
1117
|
+
|
|
1118
|
+
The kotoshu/dictionaries repository follows this structure:
|
|
1119
|
+
|
|
1120
|
+
[source]
|
|
1121
|
+
----
|
|
1122
|
+
kotoshu/dictionaries/
|
|
1123
|
+
├── en/
|
|
1124
|
+
│ ├── spelling/
|
|
1125
|
+
│ │ ├── index.dic # Hunspell dictionary
|
|
1126
|
+
│ │ ├── index.aff # Hunspell affix rules
|
|
1127
|
+
│ │ └── metadata.json # Version info
|
|
1128
|
+
│ ├── grammar/
|
|
1129
|
+
│ │ └── rules.yaml # Grammar rules (future)
|
|
1130
|
+
│ └── models/
|
|
1131
|
+
│ ├── fasttext/
|
|
1132
|
+
│ │ └── cc.en.300.vec # FastText vectors
|
|
1133
|
+
│ └── onnx/
|
|
1134
|
+
│ └── fasttext.en.onnx # ONNX model
|
|
1135
|
+
├── de/
|
|
1136
|
+
│ └── ... (same structure)
|
|
1137
|
+
└── README.md
|
|
1138
|
+
|
|
1139
|
+
kotoshu/frequency-list-kelly/
|
|
1140
|
+
├── data/
|
|
1141
|
+
│ ├── en.json # Kelly frequency data
|
|
1142
|
+
│ ├── ru.json
|
|
1143
|
+
│ └── ...
|
|
1144
|
+
└── README.md
|
|
1145
|
+
----
|
|
1146
|
+
|
|
1147
|
+
=== Adding New Languages
|
|
1148
|
+
|
|
1149
|
+
To add support for a new language:
|
|
1150
|
+
|
|
1151
|
+
1. **Dictionary**: Add Hunspell dictionary to `kotoshu/dictionaries/{code}/spelling/`
|
|
1152
|
+
2. **Frequency**: Add Kelly frequency data to `kotoshu/frequency-list-kelly/data/{code}.json`
|
|
1153
|
+
3. **Register**: Add to `AVAILABLE_LANGUAGES` in `LanguageCache`
|
|
1154
|
+
4. **Test**: Run integration tests to verify
|
|
1155
|
+
|
|
1156
|
+
See CONTRIBUTING.adoc for detailed guidelines.
|
|
1157
|
+
|
|
1158
|
+
== Model Repository
|
|
1159
|
+
|
|
1160
|
+
ONNX models are hosted at: https://github.com/kotoshu/dictionaries[kotoshu/dictionaries]
|
|
1161
|
+
|
|
1162
|
+
Download and setup:
|
|
1163
|
+
|
|
1164
|
+
[source,bash]
|
|
1165
|
+
----
|
|
1166
|
+
# Preferred: let kotoshu fetch and verify the model
|
|
1167
|
+
kotoshu setup en --want spelling,model
|
|
1168
|
+
|
|
1169
|
+
# Manual clone (advanced; bypasses manifest verification)
|
|
1170
|
+
git clone https://github.com/kotoshu/dictionaries.git ~/src/kotoshu/dictionaries
|
|
1171
|
+
----
|
|
1172
|
+
|
|
1173
|
+
== Writing a Document Plugin
|
|
1174
|
+
|
|
1175
|
+
Kotoshu ships only `Kotoshu::Documents::PlainTextDocument` for the
|
|
1176
|
+
plain-text case. Format-specific parsers (Markdown, AsciiDoc,
|
|
1177
|
+
reStructuredText, LaTeX, …) live in plugins — see the
|
|
1178
|
+
`kotoshu-document-plugin-boundary` design note. Kotoshu auto-discovers
|
|
1179
|
+
plugin files via `Gem.find_files("kotoshu_plugin/document/*.rb")` — install
|
|
1180
|
+
the gem and the parser registers itself on first use.
|
|
1181
|
+
|
|
1182
|
+
A plugin has three pieces:
|
|
1183
|
+
|
|
1184
|
+
1. *Parser class* — responds to
|
|
1185
|
+
`.from_string(text, language_code:)` (and optionally
|
|
1186
|
+
`.from_file(path, language_code:)`). Returns a
|
|
1187
|
+
`Kotoshu::Documents::Document` whose `text_nodes` carry proper
|
|
1188
|
+
`Documents::SourceRange`s.
|
|
1189
|
+
2. *Plugin file* — placed at
|
|
1190
|
+
`lib/kotoshu_plugin/document/<format>.rb` in the gem. The file body
|
|
1191
|
+
calls `Kotoshu::Documents.register(format, ParserClass)` at load
|
|
1192
|
+
time.
|
|
1193
|
+
3. *Gem dependency* — `kotoshu` (this gem) and whatever
|
|
1194
|
+
format-parsing library the parser wraps (e.g. `kramdown` for
|
|
1195
|
+
Markdown, `asciidoctor` for AsciiDoc).
|
|
1196
|
+
|
|
1197
|
+
Example skeleton (`my_markdown_parser` gem):
|
|
1198
|
+
|
|
1199
|
+
[source,ruby]
|
|
1200
|
+
----
|
|
1201
|
+
# lib/my_markdown_parser/document_parser.rb
|
|
1202
|
+
module MyMarkdownParser
|
|
1203
|
+
class DocumentParser
|
|
1204
|
+
# Build one or more TextNodes from the Markdown AST, each
|
|
1205
|
+
# carrying a SourceRange that points at the original source.
|
|
1206
|
+
def self.from_string(text, language_code: nil)
|
|
1207
|
+
# ... walk the AST, collect (text, source_offset) per text run,
|
|
1208
|
+
# build TextNodes with SourceRanges ...
|
|
1209
|
+
Kotoshu::Documents::Document.new(
|
|
1210
|
+
text_nodes: nodes,
|
|
1211
|
+
source: text,
|
|
1212
|
+
format: :markdown,
|
|
1213
|
+
language_code: language_code
|
|
1214
|
+
)
|
|
1215
|
+
end
|
|
1216
|
+
end
|
|
1217
|
+
end
|
|
1218
|
+
----
|
|
1219
|
+
|
|
1220
|
+
[source,ruby]
|
|
1221
|
+
----
|
|
1222
|
+
# lib/kotoshu_plugin/document/markdown.rb
|
|
1223
|
+
require "kotoshu"
|
|
1224
|
+
require "my_markdown_parser/document_parser"
|
|
1225
|
+
Kotoshu::Documents.register(:markdown, MyMarkdownParser::DocumentParser)
|
|
1226
|
+
----
|
|
1227
|
+
|
|
1228
|
+
After installing the gem, callers get Markdown parsing for free:
|
|
1229
|
+
|
|
1230
|
+
[source,ruby]
|
|
1231
|
+
----
|
|
1232
|
+
require "kotoshu"
|
|
1233
|
+
|
|
1234
|
+
doc = Kotoshu::Documents.parse(source, format: :markdown,
|
|
1235
|
+
language_code: "en")
|
|
1236
|
+
errors = Kotoshu::Spellchecker.new(
|
|
1237
|
+
resource_bundle: Kotoshu::ResourceManager.resolve(language: "en")
|
|
1238
|
+
).check(doc.flattened_text)
|
|
1239
|
+
errors.each do |error|
|
|
1240
|
+
puts "#{error.source_range}: #{error.original} → #{error.recommended_suggestion}"
|
|
1241
|
+
end
|
|
1242
|
+
----
|
|
1243
|
+
|
|
1244
|
+
The structure-aware contract: the analyzer scans
|
|
1245
|
+
`doc.flattened_text` (concatenation of every node's `text`), but every
|
|
1246
|
+
error carries a `source_range` from the original source. An editor can
|
|
1247
|
+
use that range to highlight the user's actual markup, not the
|
|
1248
|
+
stripped text. For example, on the source
|
|
1249
|
+
|
|
1250
|
+
I'm an **friend** of Tom
|
|
1251
|
+
|
|
1252
|
+
a grammar error "an friend" → "a friend" carries a source_range
|
|
1253
|
+
pointing at the original `an **friend**` range, not the flattened
|
|
1254
|
+
"an friend" range.
|
|
1255
|
+
|
|
1256
|
+
Plugin gems should publish to RubyGems with a runtime dependency on
|
|
1257
|
+
`kotoshu >= 0.5.0`. Format-specific parsers are the plugin's
|
|
1258
|
+
responsibility — kotoshu never owns document parsing.
|
|
1259
|
+
|
|
1260
|
+
== API Documentation
|
|
1261
|
+
|
|
1262
|
+
The YARD API reference for the Ruby API builds in CI on every change
|
|
1263
|
+
to `lib/` (the `docs` workflow); the generated HTML is attached to each
|
|
1264
|
+
run as the `yard-docs` artifact. A hosted copy is pending — it will
|
|
1265
|
+
live either on this repository's GitHub Pages or as an API subpath on
|
|
1266
|
+
https://kotoshu.github.io[kotoshu.github.io].
|
|
1267
|
+
|
|
1268
|
+
To build and browse it locally:
|
|
1269
|
+
|
|
1270
|
+
[source,bash]
|
|
1271
|
+
----
|
|
1272
|
+
bundle install
|
|
1273
|
+
bundle exec yard doc
|
|
1274
|
+
open doc/index.html
|
|
1275
|
+
----
|
|
1276
|
+
|
|
1277
|
+
== License
|
|
1278
|
+
|
|
1279
|
+
BSD 2-Clause — see the link:LICENSE[LICENSE] file for details.
|
|
1280
|
+
|
|
1281
|
+
Bundled dictionaries and frequency lists carry their own licenses; see the
|
|
1282
|
+
per-language `license` files in https://github.com/kotoshu/dictionaries[kotoshu/dictionaries].
|
|
1283
|
+
|
|
1284
|
+
== Contributing
|
|
1285
|
+
|
|
1286
|
+
Contributions are welcome! Please see CONTRIBUTING.adoc for guidelines.
|
|
1287
|
+
|
|
1288
|
+
== Acknowledgments
|
|
1289
|
+
|
|
1290
|
+
* FastText: https://fasttext.cc/[Facebook Research]
|
|
1291
|
+
* ONNX Runtime: https://onnxruntime.ai/[Microsoft]
|
|
1292
|
+
* Hunspell: https://hunspell.github.io/[László Németh]
|