kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,502 @@
|
|
|
1
|
+
# Kotoshu Cache System Architecture
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
The Kotoshu cache system provides a unified, object-oriented architecture for managing linguistic resources (dictionaries, frequency lists, and embedding models). The system implements automatic download, local caching, metadata management, and TTL-based expiration.
|
|
6
|
+
|
|
7
|
+
## Design Principles
|
|
8
|
+
|
|
9
|
+
### 1. DRY (Don't Repeat Yourself)
|
|
10
|
+
All cache types inherit from `BaseCache`, eliminating code duplication for:
|
|
11
|
+
- HTTP downloads
|
|
12
|
+
- Metadata serialization/deserialization
|
|
13
|
+
- Cache validation (existence, expiration)
|
|
14
|
+
- Statistics tracking (hits, misses, hit rate)
|
|
15
|
+
- TTL management
|
|
16
|
+
|
|
17
|
+
### 2. OOP (Object-Oriented Programming)
|
|
18
|
+
- Abstract base class defines the interface
|
|
19
|
+
- Concrete implementations provide type-specific behavior
|
|
20
|
+
- Template Method pattern for extensible operations
|
|
21
|
+
- Strategy pattern for resource type handling
|
|
22
|
+
|
|
23
|
+
### 3. Separation of Concerns
|
|
24
|
+
- `BaseCache`: Common cache operations
|
|
25
|
+
- `LanguageCache`: Dictionary and grammar resources
|
|
26
|
+
- `ModelCache`: Embedding model resources
|
|
27
|
+
- `FrequencyCache`: Frequency list resources
|
|
28
|
+
|
|
29
|
+
### 4. Single Responsibility
|
|
30
|
+
Each cache class has one responsibility:
|
|
31
|
+
- LanguageCache manages dictionaries
|
|
32
|
+
- ModelCache manages models
|
|
33
|
+
- FrequencyCache manages frequency lists
|
|
34
|
+
|
|
35
|
+
## Architecture Diagram
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
39
|
+
│ Application Layer │
|
|
40
|
+
│ (Kotoshu::SpellChecker, Kotoshu::Suggestions::Strategies::EditDistance) │
|
|
41
|
+
└─────────────────────────────────────────────────────────────────────────────┘
|
|
42
|
+
│
|
|
43
|
+
▼
|
|
44
|
+
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
45
|
+
│ Cache Interface Layer │
|
|
46
|
+
│ (CLI: kotoshu cache ...) │
|
|
47
|
+
└─────────────────────────────────────────────────────────────────────────────┘
|
|
48
|
+
│
|
|
49
|
+
┌─────────────────────┼─────────────────────┐
|
|
50
|
+
│ │ │
|
|
51
|
+
▼ ▼ ▼
|
|
52
|
+
┌───────────────────────────┐ ┌──────────────────┐ ┌─────────────────────────┐
|
|
53
|
+
│ LanguageCache │ │ ModelCache │ │ FrequencyCache │
|
|
54
|
+
│ (Dictionaries) │ │ (Embeddings) │ │ (Kelly Lists) │
|
|
55
|
+
├───────────────────────────┤ ├──────────────────┤ ├─────────────────────────┤
|
|
56
|
+
│ + get_spelling() │ │ + get_fasttext() │ │ + get_frequency() │
|
|
57
|
+
│ + get_grammar() │ │ + get_onnx() │ │ + available_languages() │
|
|
58
|
+
│ + available_languages() │ │ + available_... │ │ │
|
|
59
|
+
└───────────┬───────────────┘ └──────┬───────────┘ └───────────┬─────────────┘
|
|
60
|
+
│ │ │
|
|
61
|
+
│ │ │
|
|
62
|
+
▼ ▼ ▼
|
|
63
|
+
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
64
|
+
│ BaseCache (Abstract) │
|
|
65
|
+
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
|
66
|
+
│ │ Public API │ │
|
|
67
|
+
│ │ + get(resource_id, force_download: false) │ │
|
|
68
|
+
│ │ + available?(resource_id) │ │
|
|
69
|
+
│ │ + clear(resource_id) │ │
|
|
70
|
+
│ │ + clear_all() │ │
|
|
71
|
+
│ │ + stats │ │
|
|
72
|
+
│ │ + clean() │ │
|
|
73
|
+
│ └─────────────────────────────────────────────────────────────────────┘ │
|
|
74
|
+
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
|
75
|
+
│ │ Protected Abstract Methods (subclasses must implement) │ │
|
|
76
|
+
│ │ + download_resource(resource_id, dest_path) │ │
|
|
77
|
+
│ │ + load_cached(resource_id) │ │
|
|
78
|
+
│ │ + metadata_path_for(resource_id) │ │
|
|
79
|
+
│ │ + resource_dir_for(resource_id) │ │
|
|
80
|
+
│ │ + resource_files_exist?(resource_id) │ │
|
|
81
|
+
│ │ + supports_resource?(resource_id) │ │
|
|
82
|
+
│ │ + cached_resources │ │
|
|
83
|
+
│ └─────────────────────────────────────────────────────────────────────┘ │
|
|
84
|
+
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
|
85
|
+
│ │ Protected Helper Methods (available to subclasses) │ │
|
|
86
|
+
│ │ + download_url(url) │ │
|
|
87
|
+
│ │ + download_file(url, dest_path) │ │
|
|
88
|
+
│ │ + write_metadata(path, metadata) │ │
|
|
89
|
+
│ │ + read_metadata(path) │ │
|
|
90
|
+
│ │ + cached?(metadata_path) │ │
|
|
91
|
+
│ │ + expired?(metadata_path) │ │
|
|
92
|
+
│ │ + checksum(content) │ │
|
|
93
|
+
│ │ + parse_resource_id(resource_id) │ │
|
|
94
|
+
│ │ + extract_language(resource_id) │ │
|
|
95
|
+
│ │ + extract_type(resource_id) │ │
|
|
96
|
+
│ └─────────────────────────────────────────────────────────────────────┘ │
|
|
97
|
+
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
|
98
|
+
│ │ Private Methods │ │
|
|
99
|
+
│ │ + default_cache_path │ │
|
|
100
|
+
│ │ + default_url_base │ │
|
|
101
|
+
│ │ + default_cache_ttl │ │
|
|
102
|
+
│ └─────────────────────────────────────────────────────────────────────┘ │
|
|
103
|
+
└─────────────────────────────────────────────────────────────────────────────┘
|
|
104
|
+
│
|
|
105
|
+
┌─────────────────────┼─────────────────────┐
|
|
106
|
+
│ │ │
|
|
107
|
+
▼ ▼ ▼
|
|
108
|
+
┌─────────────────────┐ ┌──────────────┐ ┌─────────────────────────┐
|
|
109
|
+
│ ~/.kotoshu/ │ │ ~/.kotoshu/ │ │ ~/.kotoshu/ │
|
|
110
|
+
│ languages/ │ │ models/ │ │ frequency-lists/ │
|
|
111
|
+
│ ┌─────────────────┐ │ │ ┌──────────┐ │ │ ┌─────────────────────┐ │
|
|
112
|
+
│ │ en/ │ │ │ │ en/ │ │ │ │ en/ │ │
|
|
113
|
+
│ │ ├── spelling/ │ │ │ │ └──models/│ │ │ │ ├── frequency.json │ │
|
|
114
|
+
│ │ │ ├── index.dic│ │ │ │ fasttext│ │ │ │ └── metadata.json │ │
|
|
115
|
+
│ │ │ └── index.aff│ │ │ │ onnx/ │ │ │ └─────────────────────┘ │
|
|
116
|
+
│ │ └── metadata.json│ │ │ │ fasttext│ │ │ │
|
|
117
|
+
│ └─────────────────┘ │ │ │ onnx │ │ │ ru/, ar/, etc. │
|
|
118
|
+
│ de/, es/, fr/, etc. │ │ └──────────┘ │ │ │
|
|
119
|
+
└─────────────────────┘ └──────────────┘ └─────────────────────────┘
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
## Resource ID Format
|
|
123
|
+
|
|
124
|
+
All resources use a colon-separated identifier format: `{language}:{type}`
|
|
125
|
+
|
|
126
|
+
### Examples
|
|
127
|
+
- `en:spelling` - English spelling dictionary
|
|
128
|
+
- `en:fasttext` - English FastText model
|
|
129
|
+
- `en:frequency` - English frequency data (deprecated, use FrequencyCache)
|
|
130
|
+
- `ru:spelling` - Russian spelling dictionary
|
|
131
|
+
- `de:onnx` - German ONNX model
|
|
132
|
+
- `en:onnx:mini` - English mini-tier ONNX model (tiered form `{language}:onnx:{tier}`; tiers are `full`, `fluency`, `mini`; `full` uses the legacy two-part id)
|
|
133
|
+
|
|
134
|
+
## Model Tiers and the Registry
|
|
135
|
+
|
|
136
|
+
`ModelCache` supports tiered ONNX model resources (`full`, `fluency`, `mini`), resolved through the resource registry (`registry.json`) published by `kotoshu/models-fasttext-onnx` under stable ids (`kotoshu://models/{lang}/{tier}`). Approximate sizes: `full` ~120 MB, `fluency` ~15-18 MB, `mini` ~3 MB.
|
|
137
|
+
|
|
138
|
+
### Tiered on-disk layout
|
|
139
|
+
|
|
140
|
+
Tiered resource ids carry a third segment (`{language}:onnx:{tier}`). The `full` tier keeps the legacy two-part id (`{language}:onnx`) and the legacy on-disk layout, preserved byte-compatibly; `fluency` and `mini` nest under the onnx directory as sibling directories so tiers coexist:
|
|
141
|
+
|
|
142
|
+
```
|
|
143
|
+
{model cache}/
|
|
144
|
+
└── en/
|
|
145
|
+
└── models/
|
|
146
|
+
├── fasttext/
|
|
147
|
+
└── onnx/ # full tier (legacy layout, unchanged)
|
|
148
|
+
├── fasttext.en.onnx
|
|
149
|
+
├── fasttext.en.vocab.json
|
|
150
|
+
├── metadata.json
|
|
151
|
+
├── mini/ # tiered siblings
|
|
152
|
+
│ ├── fasttext.en.mini.onnx
|
|
153
|
+
│ ├── fasttext.en.mini.vocab.json
|
|
154
|
+
│ └── metadata.json
|
|
155
|
+
└── fluency/
|
|
156
|
+
└── ...
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
`ModelCache#cached_tiers(language)` lists the tiers currently cached for a language with a disk-only lookup (it never touches the network, so it is resolve-safe), in the fixed preference order `mini`, `fluency`, `full`. That order exists so ambiguity errors are deterministic across machines and runs — it is a reporting order, not a fallback chain; tiers are never silently substituted.
|
|
160
|
+
|
|
161
|
+
### The cached registry
|
|
162
|
+
|
|
163
|
+
`registry.json` is fetched at most once per cache instance and stored under `{model cache}/registry/`:
|
|
164
|
+
|
|
165
|
+
- `registry.json` — the raw registry bytes
|
|
166
|
+
- `metadata.json` — records the bytes' own SHA-256, source URL, and fetch time
|
|
167
|
+
|
|
168
|
+
The cached copy is reused while fresh (subject to the cache TTL). A copy that is absent, unparsable, or fails its recorded SHA-256 is treated as unusable and re-fetched rather than trusted.
|
|
169
|
+
|
|
170
|
+
`KOTOSHU_OFFLINE=1`: the registry is never fetched. A cached copy — even a stale one — is reused; when none is cached, `Kotoshu::Error` is raised (run once online, or copy `registry.json` there first).
|
|
171
|
+
|
|
172
|
+
The resolve path never consults the registry at all. It is cache-only, so a missing tier raises `ResourceNotSetupError` exactly like any other unset resource — resolve never downloads and never falls back to another tier.
|
|
173
|
+
|
|
174
|
+
### Download and verification
|
|
175
|
+
|
|
176
|
+
Tiered downloads (`ModelCache#download_tiered_model`) raise rather than degrade to `nil`, so setup callers can report `:unavailable`:
|
|
177
|
+
|
|
178
|
+
1. Look up the `(language, tier)` entry in the registry. No entry means the language publishes no such tier.
|
|
179
|
+
2. Download the model from `urls.primary`, falling back to `urls.mirror` (partial bytes from a failed attempt are removed first).
|
|
180
|
+
3. Verify the downloaded bytes' SHA-256 against the registry entry. On mismatch the corrupt file is deleted, the mismatch is recorded in the audit log, and `Kotoshu::IntegrityError` is raised with a remediation hint.
|
|
181
|
+
4. Download the vocab sibling (a missing vocab is a warning, not a failure).
|
|
182
|
+
|
|
183
|
+
## Cache Lifecycle
|
|
184
|
+
|
|
185
|
+
### 1. Initial Access (Cache Miss)
|
|
186
|
+
|
|
187
|
+
```
|
|
188
|
+
User calls cache.get('en:spelling')
|
|
189
|
+
│
|
|
190
|
+
▼
|
|
191
|
+
BaseCache.get() checks:
|
|
192
|
+
- supports_resource?('en:spelling') → true
|
|
193
|
+
- cached?(metadata_path) → false (first access)
|
|
194
|
+
│
|
|
195
|
+
▼
|
|
196
|
+
Download triggered:
|
|
197
|
+
- download_resource('en:spelling', dest_path)
|
|
198
|
+
- download_file(url, dest_path)
|
|
199
|
+
- write_metadata(metadata_path, metadata)
|
|
200
|
+
- track_miss
|
|
201
|
+
│
|
|
202
|
+
▼
|
|
203
|
+
Return: { aff_path, dic_path, cached: false, metadata }
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
### 2. Subsequent Access (Cache Hit)
|
|
207
|
+
|
|
208
|
+
```
|
|
209
|
+
User calls cache.get('en:spelling')
|
|
210
|
+
│
|
|
211
|
+
▼
|
|
212
|
+
BaseCache.get() checks:
|
|
213
|
+
- supports_resource?('en:spelling') → true
|
|
214
|
+
- cached?(metadata_path) → true
|
|
215
|
+
- expired?(metadata_path) → false (within TTL)
|
|
216
|
+
│
|
|
217
|
+
▼
|
|
218
|
+
Load from cache:
|
|
219
|
+
- load_cached('en:spelling')
|
|
220
|
+
- read_metadata(metadata_path)
|
|
221
|
+
- track_hit
|
|
222
|
+
│
|
|
223
|
+
▼
|
|
224
|
+
Return: { aff_path, dic_path, cached: true, metadata }
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
### 3. Expired Cache Access
|
|
228
|
+
|
|
229
|
+
```
|
|
230
|
+
User calls cache.get('en:spelling')
|
|
231
|
+
│
|
|
232
|
+
▼
|
|
233
|
+
BaseCache.get() checks:
|
|
234
|
+
- supports_resource?('en:spelling') → true
|
|
235
|
+
- cached?(metadata_path) → true
|
|
236
|
+
- expired?(metadata_path) → true (older than 7 days)
|
|
237
|
+
│
|
|
238
|
+
▼
|
|
239
|
+
Re-download triggered:
|
|
240
|
+
- download_resource('en:spelling', dest_path)
|
|
241
|
+
- (same as initial access)
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
## Metadata Format
|
|
245
|
+
|
|
246
|
+
All cached resources include metadata in JSON format:
|
|
247
|
+
|
|
248
|
+
```json
|
|
249
|
+
{
|
|
250
|
+
"version": "2025-01-15T10:30:00Z",
|
|
251
|
+
"url": "https://raw.githubusercontent.com/...",
|
|
252
|
+
"language": "en",
|
|
253
|
+
"type": "spelling",
|
|
254
|
+
"checksum": "a1b2c3d4e5f6...",
|
|
255
|
+
"cached_at": "2025-01-15T10:30:00Z"
|
|
256
|
+
}
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
### Metadata Fields
|
|
260
|
+
- `version`: Timestamp when resource was cached
|
|
261
|
+
- `url`: Source URL for the resource
|
|
262
|
+
- `language`: ISO 639-1 language code
|
|
263
|
+
- `type`: Resource type (spelling, grammar, fasttext, onnx, kelly_frequency)
|
|
264
|
+
- `checksum`: SHA256 checksum for integrity verification
|
|
265
|
+
- `cached_at`: Timestamp of caching
|
|
266
|
+
|
|
267
|
+
## Thread Safety
|
|
268
|
+
|
|
269
|
+
The cache system is designed for single-threaded use. For multi-threaded environments:
|
|
270
|
+
|
|
271
|
+
1. **Process-based parallelism**: Each process has its own cache
|
|
272
|
+
2. **File locking**: Not implemented (future enhancement)
|
|
273
|
+
3. **Recommended**: Use separate cache instances per thread
|
|
274
|
+
|
|
275
|
+
## Error Handling
|
|
276
|
+
|
|
277
|
+
### Download Failures
|
|
278
|
+
- HTTP errors raise `RuntimeError` with details
|
|
279
|
+
- Invalid JSON raises `JSON::ParserError`
|
|
280
|
+
- File I/O errors propagate to caller
|
|
281
|
+
|
|
282
|
+
### Cache Validation
|
|
283
|
+
- Missing metadata returns `nil`
|
|
284
|
+
- Expired cache triggers re-download
|
|
285
|
+
- Invalid checksum triggers re-download (future)
|
|
286
|
+
|
|
287
|
+
## Performance Considerations
|
|
288
|
+
|
|
289
|
+
### Cache Hit Rates
|
|
290
|
+
Target hit rates by cache type:
|
|
291
|
+
- **LanguageCache**: 85-95% (dictionaries rarely change)
|
|
292
|
+
- **FrequencyCache**: 95-99% (frequency lists very stable)
|
|
293
|
+
- **ModelCache**: 98-99% (models rarely change)
|
|
294
|
+
|
|
295
|
+
### Disk Usage
|
|
296
|
+
Typical disk usage per language:
|
|
297
|
+
- **Dictionary**: ~2-4 MB (Hunspell files)
|
|
298
|
+
- **Frequency**: ~800 KB (Kelly JSON)
|
|
299
|
+
- **Models**: ~600-2000 MB (FastText vectors, varies by language)
|
|
300
|
+
|
|
301
|
+
### Download Speeds
|
|
302
|
+
Typical download times (broadband):
|
|
303
|
+
- **Dictionary**: 2-5 seconds
|
|
304
|
+
- **Frequency**: 1-2 seconds
|
|
305
|
+
- **Models**: 30-120 seconds (large files)
|
|
306
|
+
|
|
307
|
+
## Extending the Cache System
|
|
308
|
+
|
|
309
|
+
### Adding a New Cache Type
|
|
310
|
+
|
|
311
|
+
1. **Create subclass of BaseCache**:
|
|
312
|
+
|
|
313
|
+
```ruby
|
|
314
|
+
class MyCache < Kotoshu::Cache::BaseCache
|
|
315
|
+
def initialize(cache_path: nil, url_base: nil, cache_ttl: nil)
|
|
316
|
+
super
|
|
317
|
+
end
|
|
318
|
+
|
|
319
|
+
protected
|
|
320
|
+
|
|
321
|
+
def download_resource(resource_id, dest_path)
|
|
322
|
+
# Download logic
|
|
323
|
+
end
|
|
324
|
+
|
|
325
|
+
def load_cached(resource_id)
|
|
326
|
+
# Load logic
|
|
327
|
+
end
|
|
328
|
+
|
|
329
|
+
def metadata_path_for(resource_id)
|
|
330
|
+
# Return path to metadata.json
|
|
331
|
+
end
|
|
332
|
+
|
|
333
|
+
def resource_dir_for(resource_id)
|
|
334
|
+
# Return directory path
|
|
335
|
+
end
|
|
336
|
+
|
|
337
|
+
def resource_files_exist?(resource_id)
|
|
338
|
+
# Check if files exist
|
|
339
|
+
end
|
|
340
|
+
|
|
341
|
+
def supports_resource?(resource_id)
|
|
342
|
+
# Check if resource is supported
|
|
343
|
+
end
|
|
344
|
+
|
|
345
|
+
def cached_resources
|
|
346
|
+
# Return list of cached resource IDs
|
|
347
|
+
end
|
|
348
|
+
end
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
2. **Implement abstract methods**
|
|
352
|
+
3. **Set appropriate TTL** (override `default_cache_ttl`)
|
|
353
|
+
4. **Set cache path** (override `default_cache_path`)
|
|
354
|
+
|
|
355
|
+
### Downloading from External Sources
|
|
356
|
+
|
|
357
|
+
For resources hosted outside GitHub (like FastText models), override the
|
|
358
|
+
download logic and URL generation:
|
|
359
|
+
|
|
360
|
+
```ruby
|
|
361
|
+
def model_url(language, type, filename)
|
|
362
|
+
case type
|
|
363
|
+
when "fasttext"
|
|
364
|
+
# Download from FastText CDN (Facebook Research)
|
|
365
|
+
# https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/
|
|
366
|
+
"https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/#{filename}"
|
|
367
|
+
else
|
|
368
|
+
"#{@url_base}/default/path/#{filename}"
|
|
369
|
+
end
|
|
370
|
+
end
|
|
371
|
+
|
|
372
|
+
def download_resource(resource_id, dest_path)
|
|
373
|
+
# ... existing code ...
|
|
374
|
+
|
|
375
|
+
filename = model_info[:file]
|
|
376
|
+
|
|
377
|
+
# Handle gzip compression
|
|
378
|
+
if filename.end_with?('.gz')
|
|
379
|
+
download_and_decompress(url, dest_path)
|
|
380
|
+
else
|
|
381
|
+
download_file(url, dest_path)
|
|
382
|
+
end
|
|
383
|
+
end
|
|
384
|
+
|
|
385
|
+
def download_and_decompress(url, dest_path)
|
|
386
|
+
# Download to temporary .gz file
|
|
387
|
+
temp_gz = "#{dest_path}.gz"
|
|
388
|
+
URI.open(url, open_timeout: 30, read_timeout: 300) do |uri|
|
|
389
|
+
File.write(temp_gz, uri.read, mode: 'wb')
|
|
390
|
+
end
|
|
391
|
+
|
|
392
|
+
# Decompress gzip
|
|
393
|
+
Zlib::GzipReader.open(temp_gz) do |gz|
|
|
394
|
+
File.write(dest_path, gz.read)
|
|
395
|
+
end
|
|
396
|
+
|
|
397
|
+
File.delete(temp_gz)
|
|
398
|
+
end
|
|
399
|
+
```
|
|
400
|
+
|
|
401
|
+
### FastText Model Sources
|
|
402
|
+
|
|
403
|
+
FastText models are downloaded from Facebook's public CDN:
|
|
404
|
+
|
|
405
|
+
- **Base URL**: `https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/`
|
|
406
|
+
- **Files**: `cc.{lang}.300.vec.gz` (gzipped FastText vectors)
|
|
407
|
+
- **Decompressed**: `cc.{lang}.300.vec` (text format, 300 dimensions)
|
|
408
|
+
- **Sizes**: ~1.2-2.7 GB per language (decompressed)
|
|
409
|
+
|
|
410
|
+
.File sizes by language
|
|
411
|
+
[width="100%",cols="^1,^1,^1",options="header"]
|
|
412
|
+
|=========================================
|
|
413
|
+
| Language | Compressed | Decompressed | Word Count |
|
|
414
|
+
|=========================================
|
|
415
|
+
| de (German) | 1.2 GB | ~2.5 GB | 1,000,000 |
|
|
416
|
+
| en (English) | 2.7 GB | ~5.0 GB | 2,000,000 |
|
|
417
|
+
| es (Spanish) | 1.2 GB | ~2.5 GB | 1,000,000 |
|
|
418
|
+
| fr (French) | 1.2 GB | ~2.5 GB | 1,000,000 |
|
|
419
|
+
| pt (Portuguese) | 1.2 GB | ~2.5 GB | 1,000,000 |
|
|
420
|
+
| ru (Russian) | 1.2 GB | ~2.5 GB | 1,000,000 |
|
|
421
|
+
|=========================================]
|
|
422
|
+
|
|
423
|
+
### Adding CLI Commands
|
|
424
|
+
|
|
425
|
+
To add CLI commands for a new cache type, edit `lib/kotoshu/cli/cache_command.rb`:
|
|
426
|
+
|
|
427
|
+
```ruby
|
|
428
|
+
desc 'download TYPE RESOURCE', 'Download a resource from GitHub'
|
|
429
|
+
def download(type, resource)
|
|
430
|
+
cache = cache_for_type(type)
|
|
431
|
+
|
|
432
|
+
case type
|
|
433
|
+
when 'mycache', 'mc'
|
|
434
|
+
download_mycache(cache, resource)
|
|
435
|
+
end
|
|
436
|
+
end
|
|
437
|
+
|
|
438
|
+
private
|
|
439
|
+
|
|
440
|
+
def download_mycache(cache, resource)
|
|
441
|
+
result = cache.get(resource, force_download: options[:force])
|
|
442
|
+
puts "Downloaded: #{result[:path]}"
|
|
443
|
+
end
|
|
444
|
+
|
|
445
|
+
def cache_for_type(type)
|
|
446
|
+
case type
|
|
447
|
+
when 'mycache', 'mc'
|
|
448
|
+
MyCache.new
|
|
449
|
+
# ... existing types
|
|
450
|
+
end
|
|
451
|
+
end
|
|
452
|
+
```
|
|
453
|
+
|
|
454
|
+
## Testing
|
|
455
|
+
|
|
456
|
+
### Unit Tests
|
|
457
|
+
Each cache type has comprehensive RSpec tests:
|
|
458
|
+
- `spec/kotoshu/cache/language_cache_spec.rb`
|
|
459
|
+
- `spec/kotoshu/cache/model_cache_spec.rb`
|
|
460
|
+
- `spec/kotoshu/cache/frequency_cache_spec.rb`
|
|
461
|
+
|
|
462
|
+
### Integration Tests
|
|
463
|
+
Run combined cache test:
|
|
464
|
+
```bash
|
|
465
|
+
ruby test_combined_cache.rb
|
|
466
|
+
```
|
|
467
|
+
|
|
468
|
+
## Troubleshooting
|
|
469
|
+
|
|
470
|
+
### Cache Not Downloading
|
|
471
|
+
Check:
|
|
472
|
+
1. Internet connection
|
|
473
|
+
2. GitHub repository accessibility
|
|
474
|
+
3. Available disk space
|
|
475
|
+
4. File permissions on `~/.kotoshu/`
|
|
476
|
+
|
|
477
|
+
### Stale Cache Data
|
|
478
|
+
Force re-download:
|
|
479
|
+
```ruby
|
|
480
|
+
cache.get('en:spelling', force_download: true)
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
Or use CLI:
|
|
484
|
+
```bash
|
|
485
|
+
kotoshu cache download language en
|
|
486
|
+
kotoshu cache purge language en
|
|
487
|
+
kotoshu cache download language en
|
|
488
|
+
```
|
|
489
|
+
|
|
490
|
+
### High Cache Miss Rate
|
|
491
|
+
- Check TTL settings
|
|
492
|
+
- Verify cache directory exists
|
|
493
|
+
- Check metadata files are valid JSON
|
|
494
|
+
|
|
495
|
+
## Future Enhancements
|
|
496
|
+
|
|
497
|
+
1. **File Locking**: Support multi-threaded access
|
|
498
|
+
2. **Compression**: Compress cached models to save disk space
|
|
499
|
+
3. **Partial Downloads**: Resume interrupted downloads
|
|
500
|
+
4. **Cache Warming**: Pre-download frequently used resources
|
|
501
|
+
5. **Distributed Caching**: Support Redis/memcached backends
|
|
502
|
+
6. **Incremental Updates**: Update only changed resources
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
## Summary
|
|
2
|
+
|
|
3
|
+
Plan 91: two correctness fixes for non-ASCII users, found by the site-flip agent (plan 90 residues).
|
|
4
|
+
|
|
5
|
+
### Track A — Unicode word detection
|
|
6
|
+
|
|
7
|
+
`Spellchecker#word_char?` accepted only ASCII letters, so Greek and Ukrainian users could not check any text: every non-Latin word was silently invisible and `kotoshu check -l el` reported `0 words`.
|
|
8
|
+
|
|
9
|
+
Word extraction now follows the language being checked:
|
|
10
|
+
|
|
11
|
+
- `Tokenizer::Base#spellcheck_word_regex` — single-character regex for spell-check extraction. Base default is the historical ASCII set (letters + apostrophe), so tokenizers without an override keep their exact behavior.
|
|
12
|
+
- Overrides: Greek `[\p{Greek}']` (apostrophe stays in-word for elisions like `απ’`), Cyrillic `[\p{Cyrillic}']` (`Мар'яна` stays one word), Latin `[\p{Latin}']` (covers capitals like `Å`/`Ä` that the old `à-ÿ` range missed, plus Turkish `ı ğ ş İ`).
|
|
13
|
+
- Digits are never word characters, so number-adjacent splitting is unchanged.
|
|
14
|
+
- `Spellchecker` resolves the tokenizer from the resource bundle language first (the facade passes the global `Configuration` next to the resolved bundle, so the config default is the wrong source) and falls back to the `Language::Registry` language, then to ASCII. Wrong-script words stay invisible: Greek text under `-l en` extracts nothing.
|
|
15
|
+
- Languages without a script tokenizer (en uses the components whitespace tokenizer, de has no override yet) keep the ASCII behavior — guarded by specs.
|
|
16
|
+
|
|
17
|
+
### Track B — Swedish aff `RegexpError`
|
|
18
|
+
|
|
19
|
+
`Readers::AffReader` raised `RegexpError: unmatched close parenthesis` on the Swedish dictionary (dictionaries pin 1829a3e). Root cause: `COMPOUNDRULE )k` uses `)` itself as a flag character ("for adjectives: )k", "for verbs: >j" per the aff comments), but `CompoundRule` only escaped `)` when preceded by a non-operator character. A leading or post-operator `)` survived raw into `Regexp.new` and raised. The fix escapes every flag atom (`Regexp.escape`) while keeping `*`/`?` operators and `(...)` groups intact. The full sv dictionary (152,719 words) now loads and checks.
|
|
20
|
+
|
|
21
|
+
## Verification
|
|
22
|
+
|
|
23
|
+
- Full suite: `3696 examples, 0 failures, 27 pending` (13 new examples)
|
|
24
|
+
- Frozen conformance replay: `rake kotoshu:conformance:ruby` — 2630 vectors, 0 failures, byte-identical
|
|
25
|
+
- rubocop clean on all changed files
|
|
26
|
+
- End-to-end through the real CLI (setup + check, isolated cache, dictionaries from the pin):
|
|
27
|
+
|
|
28
|
+
| lang | correct sentence | bogus word |
|
|
29
|
+
|---|---|---|
|
|
30
|
+
| el | OK, 5 words, no errors | `θθξκζψθ` flagged |
|
|
31
|
+
| tr | OK, 5 words, no errors | `ğşxqzz` flagged |
|
|
32
|
+
| sv | OK, 7 words, no errors | `ÅÅxxqqz` flagged |
|
|
33
|
+
| it | OK, 5 words, no errors | `zzqqxvv` flagged |
|
|
34
|
+
|
|
35
|
+
Note: remote `kotoshu setup it/el/tr/sv` currently 404s because upstream `dictionaries` only ships `en/` under the `spelling/` URL layout — verified instead with `setup LANG --from` against dictionaries fetched from pin 1829a3e. That URL-layout gap is upstream content, not this PR.
|
|
36
|
+
|
|
37
|
+
RBS: added `spellcheck_word_regex` to the declared Greek/Cyrillic tokenizers. No version changes.
|