kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# 27 — End-to-end smoke spec (T1, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
|
|
5
|
+
|
|
6
|
+
Evidence:
|
|
7
|
+
|
|
8
|
+
- `spec/integration/end_to_end_spec.rb` exists with the planned shape:
|
|
9
|
+
fresh install → setup → use (setup `:downloaded`, `correct?`,
|
|
10
|
+
misspelling rejection, `suggest` includes "the", document check),
|
|
11
|
+
setup predicate, and two-stage `ResourceNotSetupError` enforcement —
|
|
12
|
+
12 examples, tagged `:network`, isolated via tmp XDG dirs in an
|
|
13
|
+
`around` hook.
|
|
14
|
+
- Run 2026-09-03: `NETWORK_TESTS=1 bundle exec rspec
|
|
15
|
+
spec/integration/end_to_end_spec.rb` → 12 examples, 0 failures
|
|
16
|
+
(real downloads from kotoshu/dictionaries).
|
|
17
|
+
- Real instances throughout; no `double()`. Excluded from the default
|
|
18
|
+
run via the `:network` tag as planned.
|
|
19
|
+
|
|
20
|
+
## Problem
|
|
21
|
+
There is no spec that walks the actual install → setup → use path a real
|
|
22
|
+
user takes. Unit tests verify pieces; integration tests like
|
|
23
|
+
`walking_skeleton_spec.rb` use mocked or fixture data. Nothing catches
|
|
24
|
+
"this works end-to-end starting from an empty cache."
|
|
25
|
+
|
|
26
|
+
## Plan
|
|
27
|
+
|
|
28
|
+
### New spec: `spec/integration/end_to_end_spec.rb`
|
|
29
|
+
|
|
30
|
+
A single spec file with multiple examples covering the full path:
|
|
31
|
+
|
|
32
|
+
```ruby
|
|
33
|
+
RSpec.describe "Kotoshu end-to-end", :network do
|
|
34
|
+
# Use temp XDG dir for full isolation
|
|
35
|
+
around do |ex|
|
|
36
|
+
Dir.mktmpdir do |dir|
|
|
37
|
+
ENV["XDG_CACHE_HOME"] = "#{dir}/cache"
|
|
38
|
+
ENV["XDG_CONFIG_HOME"] = "#{dir}/config"
|
|
39
|
+
ENV["XDG_DATA_HOME"] = "#{dir}/local"
|
|
40
|
+
Kotoshu::Configuration.reset
|
|
41
|
+
ex.run
|
|
42
|
+
end
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
describe "fresh install → setup → use" do
|
|
46
|
+
it "sets up English spelling" do
|
|
47
|
+
result = Kotoshu.setup(:en, want: %i[spelling])
|
|
48
|
+
expect(result.spelling).to eq(:downloaded)
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
it "accepts a correctly-spelled word" do
|
|
52
|
+
Kotoshu.setup(:en, want: %i[spelling])
|
|
53
|
+
expect(Kotoshu.correct?("hello")).to be(true)
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
it "rejects a misspelling" do
|
|
57
|
+
Kotoshu.setup(:en, want: %i[spelling])
|
|
58
|
+
expect(Kotoshu.correct?("xyzzq")).to be(false)
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
it "suggests corrections" do
|
|
62
|
+
Kotoshu.setup(:en, want: %i[spelling])
|
|
63
|
+
expect(Kotoshu.suggest("teh")).to include("the")
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
it "checks a document and flags misspellings" do
|
|
67
|
+
Kotoshu.setup(:en, want: %i[spelling])
|
|
68
|
+
result = Kotoshu.check("this is a teh test")
|
|
69
|
+
expect(result.misspelled_words.map(&:word)).to include("teh")
|
|
70
|
+
end
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
describe "setup predicate" do
|
|
74
|
+
it "returns false before setup" do
|
|
75
|
+
expect(Kotoshu.setup?(:en)).to be(false)
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
it "returns true after setup" do
|
|
79
|
+
Kotoshu.setup(:en, want: %i[spelling])
|
|
80
|
+
expect(Kotoshu.setup?(:en)).to be(true)
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
describe "two-stage model enforcement" do
|
|
85
|
+
it "raises ResourceNotSetupError on cold correct?" do
|
|
86
|
+
expect { Kotoshu.correct?("hello") }
|
|
87
|
+
.to raise_error(Kotoshu::ResourceNotSetupError)
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
end
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
### Use `:network` tag
|
|
94
|
+
Already configured in `spec/spec_helper.rb` to only run when
|
|
95
|
+
`NETWORK_TESTS=1`. Run locally before release:
|
|
96
|
+
```
|
|
97
|
+
NETWORK_TESTS=1 bundle exec rspec spec/integration/end_to_end_spec.rb
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
### Real instances, no doubles
|
|
101
|
+
Per `~/.claude/CLAUDE.md`: no `double()` in specs. Use real cache, real
|
|
102
|
+
HTTP, real dictionaries.
|
|
103
|
+
|
|
104
|
+
## Acceptance
|
|
105
|
+
|
|
106
|
+
- [ ] `spec/integration/end_to_end_spec.rb` exists, all examples pass
|
|
107
|
+
with `NETWORK_TESTS=1`.
|
|
108
|
+
- [ ] Spec isolates itself via temp XDG dirs (no pollution of dev cache).
|
|
109
|
+
- [ ] Spec uses real model instances, no `double()`.
|
|
110
|
+
- [ ] CI can run this on demand (network tag excludes from default run).
|
|
111
|
+
|
|
112
|
+
## Dependencies
|
|
113
|
+
- #23 (URL fix must land first so setup actually works).
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# 28 — Verify README quickstart (T1, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented — quickstart re-verified 2026-09-03 against 0.6.0 (via
|
|
5
|
+
`bundle exec` in the repo, not a sandbox gem install).
|
|
6
|
+
|
|
7
|
+
Commands actually run, all matching README.adoc output:
|
|
8
|
+
|
|
9
|
+
- `bundle exec exe/kotoshu setup en` →
|
|
10
|
+
`Setup en... OK (spelling: cached, frequency: skipped, model: skipped, source: kotoshu)`.
|
|
11
|
+
- `printf 'helo wrld\n' | bundle exec exe/kotoshu check` →
|
|
12
|
+
`FAIL <stdin> (2 errors)` with `helo -> hello, help, hell` /
|
|
13
|
+
`wrld -> world, weld, wald`, exit 1.
|
|
14
|
+
- Ruby API: `Kotoshu.correct?("hello")` → `true`;
|
|
15
|
+
`Kotoshu.correct?("helo")` → `false`;
|
|
16
|
+
`Kotoshu.suggest("helo").to_words.first(5)` →
|
|
17
|
+
`["hello", "help", "hell", "helot", "halo"]`;
|
|
18
|
+
`Kotoshu.check("Hello wrold").errors.map(&:word)` → `["wrold"]`.
|
|
19
|
+
- `kotoshu status`, `--format json/sarif`, exit codes 0/1/2/3 — all
|
|
20
|
+
behave as documented (see #29, #32).
|
|
21
|
+
|
|
22
|
+
Known drift found during verification (worth a fix): Thor emits
|
|
23
|
+
`[WARNING] Attempted to create command ...` lines from
|
|
24
|
+
`lib/kotoshu/cli/cache_command.rb:191,201,213` to **stdout** at class
|
|
25
|
+
load, which pollutes `--format json` / `--format sarif` output.
|
|
26
|
+
Consumers must filter them until the Thor descs are added.
|
|
27
|
+
|
|
28
|
+
## Problem
|
|
29
|
+
The README's quickstart examples may not actually work against the
|
|
30
|
+
shipped 0.3 code. The README has been rewritten multiple times; commands
|
|
31
|
+
and outputs may have drifted.
|
|
32
|
+
|
|
33
|
+
## Plan
|
|
34
|
+
|
|
35
|
+
### Step 1 — Build the gem locally
|
|
36
|
+
```
|
|
37
|
+
gem build kotoshu.gemspec
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
### Step 2 — Install into a fresh sandbox
|
|
41
|
+
```
|
|
42
|
+
rvm use 3.1@kotoshu-sandbox --create
|
|
43
|
+
gem install ./kotoshu-0.3.0.gem
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
### Step 3 — Walk every code block in README.adoc
|
|
47
|
+
For each example, paste into irb and verify output matches.
|
|
48
|
+
|
|
49
|
+
Specifically:
|
|
50
|
+
- `Kotoshu.correct?("hello")` → `true`
|
|
51
|
+
- `Kotoshu.correct?("xyzzq")` → `false`
|
|
52
|
+
- `Kotoshu.suggest("recieve")` → `["receive", ...]`
|
|
53
|
+
- `Kotoshu.check("this is a speeling test")` → document result
|
|
54
|
+
- CLI: `kotoshu setup en` → success
|
|
55
|
+
- CLI: `kotoshu suggest teh` → `["the", ...]`
|
|
56
|
+
- CLI: `kotoshu check README.md` → flags misspellings
|
|
57
|
+
|
|
58
|
+
### Step 4 — Fix any drift
|
|
59
|
+
Either fix the code or update the README to match.
|
|
60
|
+
|
|
61
|
+
## Acceptance
|
|
62
|
+
|
|
63
|
+
- [ ] Every code block in README.adoc runs without error against 0.3 gem.
|
|
64
|
+
- [ ] Outputs shown in README match observed outputs (within reason).
|
|
65
|
+
- [ ] `kotoshu --help` shows all subcommands documented in README.
|
|
66
|
+
|
|
67
|
+
## Dependencies
|
|
68
|
+
- #23, #24, #25 (must be fixed before quickstart can work).
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# 29 — `kotoshu status` command (T2, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
|
|
5
|
+
|
|
6
|
+
Evidence:
|
|
7
|
+
|
|
8
|
+
- `bundle exec exe/kotoshu status` run 2026-09-03 — output matches the
|
|
9
|
+
planned layout: version, Setup table (per-language × resource with
|
|
10
|
+
size + cached date), Cache path/size/language count, Semantic
|
|
11
|
+
(`onnxruntime loaded`, active models), Other (audit log path+size,
|
|
12
|
+
default lang, offline mode). Exit 0.
|
|
13
|
+
- `kotoshu status --json` emits valid JSON (same StatusReport, second
|
|
14
|
+
presenter).
|
|
15
|
+
- Code: `desc "status"` in `lib/kotoshu/cli.rb:267-287`; pure model at
|
|
16
|
+
`lib/kotoshu/cli/status_report.rb` (no presentation logic).
|
|
17
|
+
- Spec: `spec/kotoshu/cli/status_report_spec.rb`.
|
|
18
|
+
|
|
19
|
+
Cosmetic caveat: the Thor `[WARNING]` stdout pollution noted in #28
|
|
20
|
+
also precedes status output.
|
|
21
|
+
|
|
22
|
+
## Problem
|
|
23
|
+
Users have no single command to see:
|
|
24
|
+
- What languages are set up (spelling, frequency, model)
|
|
25
|
+
- Cache disk usage
|
|
26
|
+
- Audit log location and size
|
|
27
|
+
- Whether semantic is available (onnxruntime loaded)
|
|
28
|
+
- Default language
|
|
29
|
+
|
|
30
|
+
`kotoshu setup --list` exists but is narrow. A dedicated `status` command
|
|
31
|
+
gives users a single thing to run when something goes wrong.
|
|
32
|
+
|
|
33
|
+
## Plan
|
|
34
|
+
|
|
35
|
+
### New command: `Kotoshu::Commands::StatusCommand`
|
|
36
|
+
|
|
37
|
+
Subclass of `Thor::Group` (or registered via `register` in the CLI).
|
|
38
|
+
Output:
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
Kotoshu 0.3.0
|
|
42
|
+
|
|
43
|
+
Setup:
|
|
44
|
+
en spelling ✓ (~12 MB, cached 2026-06-27)
|
|
45
|
+
en frequency ✓ (~250 KB, cached 2026-06-27)
|
|
46
|
+
en model ✓ (~114 MB, cached 2026-06-27)
|
|
47
|
+
de spelling ✓ (~8 MB, cached 2026-06-25)
|
|
48
|
+
|
|
49
|
+
Cache:
|
|
50
|
+
Path /Users/user/.cache/kotoshu
|
|
51
|
+
Size 142 MB
|
|
52
|
+
Languages 2
|
|
53
|
+
|
|
54
|
+
Semantic:
|
|
55
|
+
onnxruntime loaded
|
|
56
|
+
Active models 1 (en)
|
|
57
|
+
|
|
58
|
+
Other:
|
|
59
|
+
Audit log /Users/user/.local/share/kotoshu/audit.log (12 KB)
|
|
60
|
+
Default lang en
|
|
61
|
+
Offline mode no
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
### Implementation
|
|
65
|
+
- Reuse `ResourceManager.languages_setup`, `setup?`, etc.
|
|
66
|
+
- New helper `Kotoshu::Paths.cache_size` (sums `File.size` of all cached files).
|
|
67
|
+
- Honor `--json` flag for machine-readable output (single source of
|
|
68
|
+
truth: same StatusReport model, two presenters).
|
|
69
|
+
|
|
70
|
+
### Status report model
|
|
71
|
+
`Kotoshu::Cli::StatusReport` — pure model. CLI formats to text; JSON
|
|
72
|
+
presenter emits structured output. MECE: model has no presentation.
|
|
73
|
+
|
|
74
|
+
## Acceptance
|
|
75
|
+
|
|
76
|
+
- [ ] `kotoshu status` works on a fresh install (shows empty setup).
|
|
77
|
+
- [ ] `kotoshu status` after `kotoshu setup en` shows the entry.
|
|
78
|
+
- [ ] `kotoshu status --json` emits valid JSON.
|
|
79
|
+
- [ ] Cache size is accurate to within 1 MB.
|
|
80
|
+
- [ ] Audit log path matches `Kotoshu::Paths.audit_log_path`.
|
|
81
|
+
|
|
82
|
+
## Dependencies
|
|
83
|
+
- #22 (SourceRegistry) — to show URLs.
|
|
84
|
+
- #25 (soft onnxruntime) — to report semantic availability.
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# 30 — Default `--language auto` for `kotoshu check` (T2, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
|
|
5
|
+
|
|
6
|
+
Evidence:
|
|
7
|
+
|
|
8
|
+
- `lib/kotoshu/cli/language_resolver.rb` implements the exact flag
|
|
9
|
+
matrix from this plan: omitted/`auto` → detect with fallback;
|
|
10
|
+
`default` → `Configuration.default_language`; any code → as-is.
|
|
11
|
+
A detection only "sticks" if the language is set up
|
|
12
|
+
(`Kotoshu.setup?`), otherwise the default is used and the result
|
|
13
|
+
carries the note `Detected: X (fallback: en)` — printed to stderr by
|
|
14
|
+
`cli.rb#resolve_language` (`warn "# #{result.note}"`).
|
|
15
|
+
- `--language` defaults to `auto` on the wired CLI check
|
|
16
|
+
(`lib/kotoshu/cli.rb:86`).
|
|
17
|
+
- `Kotoshu.detect_language` exists on the facade (`lib/kotoshu.rb:436`).
|
|
18
|
+
- Specs: `spec/kotoshu/cli/language_resolver_spec.rb` (run 2026-09-03,
|
|
19
|
+
0 failures) and `spec/kotoshu/language/detector_spec.rb`
|
|
20
|
+
(20 examples, 0 failures, 1 pending).
|
|
21
|
+
|
|
22
|
+
Deviations from the plan text:
|
|
23
|
+
|
|
24
|
+
- Option A (bundled FastText LID model) was NOT taken — CLI auto mode
|
|
25
|
+
uses character-set heuristics (`Kotoshu::Language.detect`), which
|
|
26
|
+
need no model download. The FastText LID (`Language::Identifier`)
|
|
27
|
+
remains a separate surface.
|
|
28
|
+
- The Arabic LID gap from plan 36 still stands: the Arabic example in
|
|
29
|
+
`spec/kotoshu/language/detector_spec.rb` is `pending "FastText LID
|
|
30
|
+
missing Arabic vector — see TODO.impl/30-language-auto-detection.md"`
|
|
31
|
+
(still failing if un-pended, hence pending).
|
|
32
|
+
|
|
33
|
+
## Problem
|
|
34
|
+
Today `kotoshu check file.txt` with no `--language` flag — behavior is
|
|
35
|
+
undefined. Likely uses `Configuration.default_language` ("en") silently,
|
|
36
|
+
which is wrong for non-English content.
|
|
37
|
+
|
|
38
|
+
Auto-detection (`Language::Identifier` via FastText LID) exists, but using
|
|
39
|
+
it as the default would surprise users with another model download.
|
|
40
|
+
|
|
41
|
+
## Plan
|
|
42
|
+
|
|
43
|
+
### Default behavior matrix
|
|
44
|
+
| `--language` value | Behavior |
|
|
45
|
+
|---|---|
|
|
46
|
+
| (omitted) | `auto` (see below) |
|
|
47
|
+
| `auto` | Run LID on first N tokens; use detected language |
|
|
48
|
+
| `en`, `de`, etc. | Use specified |
|
|
49
|
+
| `default` | Use `Configuration.default_language` (configurable, default "en") |
|
|
50
|
+
|
|
51
|
+
### `auto` resolution algorithm
|
|
52
|
+
1. Read first 1 KB of input.
|
|
53
|
+
2. Run `Kotoshu::Language::Identifier.detect(text)`.
|
|
54
|
+
3. If detected language is set up (`Kotoshu.setup?(lang)`), use it.
|
|
55
|
+
4. Else fall back to `Configuration.default_language`.
|
|
56
|
+
5. Emit a one-line note to stderr: `# Detected: de (fallback: en)`.
|
|
57
|
+
|
|
58
|
+
### LID model handling
|
|
59
|
+
LID uses a FastText LID model. Where does it come from?
|
|
60
|
+
- **Option A:** ship a tiny bundled model (~1 MB, 127-lang LID is small).
|
|
61
|
+
- **Option B:** download on first `auto` use (with prompt per TODO #24).
|
|
62
|
+
|
|
63
|
+
Recommend Option A — bundle the model. It's small and foundational.
|
|
64
|
+
|
|
65
|
+
## Acceptance
|
|
66
|
+
|
|
67
|
+
- [ ] `kotoshu check file.txt` (no flag) auto-detects.
|
|
68
|
+
- [ ] `--language en` skips detection, uses en.
|
|
69
|
+
- [ ] `--language default` uses default_language.
|
|
70
|
+
- [ ] Detection prints a one-line note to stderr.
|
|
71
|
+
- [ ] Falls back gracefully when detected language isn't set up.
|
|
72
|
+
|
|
73
|
+
## Dependencies
|
|
74
|
+
- #24 (auto-setup prompt) — to handle "detected language not set up".
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
# 31 — Download progress indication (T2, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
|
|
5
|
+
|
|
6
|
+
Evidence:
|
|
7
|
+
|
|
8
|
+
- `lib/kotoshu/cli/progress_reporter.rb` — `Kotoshu::Cli::ProgressReporter`
|
|
9
|
+
(TTY progress bar, periodic-line fallback, plus a silent Null variant).
|
|
10
|
+
- Wired through Configuration, not per-cache globals: the CLI installs a
|
|
11
|
+
reporter on `Kotoshu.configuration.download_reporter` during setup
|
|
12
|
+
(`lib/kotoshu/cli.rb:331-337`).
|
|
13
|
+
- Shared download helper: `BaseCache#download_file(url, dest_path,
|
|
14
|
+
reporter:)` streams in chunks and honors Content-Length with a
|
|
15
|
+
fallback (`lib/kotoshu/cache/base_cache.rb:308+`); `ModelCache` also
|
|
16
|
+
reports decompression progress every 10 MB (`model_cache.rb:775`).
|
|
17
|
+
- Spec: `spec/kotoshu/cli/progress_reporter_spec.rb` — run 2026-09-03
|
|
18
|
+
together with check_format_spec: 8 examples, 0 failures.
|
|
19
|
+
|
|
20
|
+
Not visually re-verified on a 114 MB download today (dev cache has no
|
|
21
|
+
models); chunked-stream behavior is spec-covered.
|
|
22
|
+
|
|
23
|
+
## Problem
|
|
24
|
+
Downloads (114 MB ONNX models in particular) sit silently. Users have
|
|
25
|
+
no idea whether the gem is hung or making progress.
|
|
26
|
+
|
|
27
|
+
## Plan
|
|
28
|
+
|
|
29
|
+
### New module: `Kotoshu::Cli::ProgressReporter`
|
|
30
|
+
|
|
31
|
+
Wraps download operations. Detects TTY:
|
|
32
|
+
- **TTY:** renders a progress bar to stderr (`[====> ] 45% 51MB/114MB`).
|
|
33
|
+
- **Non-TTY:** prints periodic line messages (`downloaded 51MB of 114MB`).
|
|
34
|
+
|
|
35
|
+
### Wire into caches
|
|
36
|
+
All three caches (`LanguageCache`, `FrequencyCache`, `ModelCache`) call
|
|
37
|
+
a new shared helper `Kotoshu::Cache.download_with_progress(url, dest)`
|
|
38
|
+
instead of inline `URI.open(...).read`.
|
|
39
|
+
|
|
40
|
+
The helper:
|
|
41
|
+
1. Opens connection.
|
|
42
|
+
2. Reads `Content-Length`.
|
|
43
|
+
3. Streams body in chunks, updating progress per chunk.
|
|
44
|
+
4. Falls back to indeterminate progress bar when no Content-Length.
|
|
45
|
+
|
|
46
|
+
### No `respond_to?`
|
|
47
|
+
Detecting "is this a TTY" uses `$stderr.tty?` (a method call, not type
|
|
48
|
+
check). No `respond_to?` patterns.
|
|
49
|
+
|
|
50
|
+
## Acceptance
|
|
51
|
+
|
|
52
|
+
- [ ] `kotoshu setup en` shows progress bar in TTY.
|
|
53
|
+
- [ ] `kotoshu setup en` in CI shows periodic line output.
|
|
54
|
+
- [ ] 114 MB ONNX download shows incremental progress.
|
|
55
|
+
- [ ] No regression in offline mode (`KOTOSHU_OFFLINE=1`).
|
|
56
|
+
|
|
57
|
+
## Dependencies
|
|
58
|
+
- None (independent).
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
# 32 — Verify SARIF / JSON output through CLI (T2, 0.3-blocker)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented (shipped in the 0.3 line; re-verified 2026-09-03 against 0.6.0).
|
|
5
|
+
|
|
6
|
+
Live verification (2026-09-03, `bundle exec exe/kotoshu check
|
|
7
|
+
--format sarif --language en <file with two misspellings>`):
|
|
8
|
+
|
|
9
|
+
- Output parses as JSON; `version: "2.1.0"` with
|
|
10
|
+
`$schema: https://json.schemastore.org/sarif-2.1.0.json`;
|
|
11
|
+
`runs[0].tool.driver` carries name/version/rules
|
|
12
|
+
(`kotoshu/spelling`); `results` flag both misspellings with level,
|
|
13
|
+
message including suggestions (`'teh' is not in the dictionary.
|
|
14
|
+
Suggestions: the, ten, tet`), and `locations.physicalLocation`
|
|
15
|
+
(artifact uri + char offset/length region).
|
|
16
|
+
- Exit code 1 with errors found; 0 on a clean file.
|
|
17
|
+
- Spec: `spec/kotoshu/cli/check_format_spec.rb` (real CLI invocation,
|
|
18
|
+
JSON/SARIF shape assertions) — run 2026-09-03, 0 failures.
|
|
19
|
+
|
|
20
|
+
Answers the plan's wiring question for 0.6.0: the wired surface is the
|
|
21
|
+
`check` in `lib/kotoshu/cli.rb` (class_options `--format
|
|
22
|
+
text|json|sarif`, `display_result` at cli.rb:90-95,509), which renders
|
|
23
|
+
SARIF via the batch reporter (`lib/kotoshu/cli/batch_reporter.rb`,
|
|
24
|
+
`to_sarif`). The richer `Commands::CheckCommand`
|
|
25
|
+
(`commands/check_command.rb`, adds yaml/csv) is a separate surface.
|
|
26
|
+
|
|
27
|
+
Caveats not covered by the acceptance run: (a) Thor `[WARNING]` lines
|
|
28
|
+
from `cache_command.rb` precede the JSON on stdout (see #28) — filter
|
|
29
|
+
or fix before piping to a SARIF consumer; (b) GitHub code-scanning
|
|
30
|
+
upload not tested here.
|
|
31
|
+
|
|
32
|
+
## Problem
|
|
33
|
+
`Kotoshu::CheckCommand` (in `lib/kotoshu/commands/check_command.rb`)
|
|
34
|
+
already implements `--format sarif/json`. But it may not be wired
|
|
35
|
+
correctly through the actual `kotoshu check` CLI surface. CI users
|
|
36
|
+
rely on SARIF for GitHub code scanning integration.
|
|
37
|
+
|
|
38
|
+
## Plan
|
|
39
|
+
|
|
40
|
+
### Step 1 — Audit CLI surface
|
|
41
|
+
Check `lib/kotoshu/cli.rb` and `lib/kotoshu/commands/check_command.rb`.
|
|
42
|
+
Verify:
|
|
43
|
+
- `kotoshu check --format sarif file.txt` actually emits SARIF JSON.
|
|
44
|
+
- `kotoshu check --format json file.txt` emits the JSON documented shape.
|
|
45
|
+
- Exit codes are correct (0 = no errors, 1 = errors found, etc.).
|
|
46
|
+
|
|
47
|
+
### Step 2 — Fix any wiring gaps
|
|
48
|
+
The richer `CheckCommand` exists but may not be the one wired into the
|
|
49
|
+
CLI. CLAUDE.md notes: "A richer Kotoshu::CheckCommand exists in
|
|
50
|
+
commands/check_command.rb (with --interactive, --format sarif/json,
|
|
51
|
+
--model, --language auto) — check which one is actually wired before
|
|
52
|
+
assuming a CLI flag exists."
|
|
53
|
+
|
|
54
|
+
### Step 3 — Spec it
|
|
55
|
+
New spec `spec/kotoshu/cli/check_format_spec.rb`:
|
|
56
|
+
- Real CLI invocation via `Kotoshu::Cli::Cli.start(%w[check --format json file.txt])`.
|
|
57
|
+
- Parse output as JSON.
|
|
58
|
+
- Assert structure matches documented shape.
|
|
59
|
+
|
|
60
|
+
### Step 4 — Validate SARIF
|
|
61
|
+
Validate output against the SARIF v2.1 schema (https://json.schemastore.org/sarif-2.1.0-rtm.5.json).
|
|
62
|
+
|
|
63
|
+
## Acceptance
|
|
64
|
+
|
|
65
|
+
- [ ] `kotoshu check --format json` emits valid JSON.
|
|
66
|
+
- [ ] `kotoshu check --format sarif` emits SARIF v2.1 valid output.
|
|
67
|
+
- [ ] SARIF output passes schema validation.
|
|
68
|
+
- [ ] Exit code is 1 when errors are found, 0 when clean.
|
|
69
|
+
- [ ] GitHub code scanning accepts the output (manual upload test).
|
|
70
|
+
|
|
71
|
+
## Dependencies
|
|
72
|
+
- None (independent of T1).
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# 33 — Audit log rotation (T3, deferred past 0.3)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
**Implemented** (2026-06-29). Shipped as `Kotoshu::Integrity::RotationPolicy`
|
|
5
|
+
plus rotation wiring in `Kotoshu::Integrity::AuditLog`. Configuration
|
|
6
|
+
exposed via `audit_max_bytes` / `audit_rotations` in `Configuration::SCHEMA`
|
|
7
|
+
and `BaseCache#default_audit_log` constructs the policy from configuration.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
`Kotoshu::Integrity::AuditLog` writes to `~/.local/share/kotoshu/audit.log`
|
|
11
|
+
unbounded. Long-running users accumulate megabytes of audit entries with
|
|
12
|
+
no rotation.
|
|
13
|
+
|
|
14
|
+
## Plan (as implemented)
|
|
15
|
+
|
|
16
|
+
### Rotation policy
|
|
17
|
+
- Pure value object `Kotoshu::Integrity::RotationPolicy`.
|
|
18
|
+
- `max_bytes` (default 10 MB, `KOTOSHU_AUDIT_MAX_BYTES`).
|
|
19
|
+
- `rotations` (default 5, `KOTOSHU_AUDIT_ROTATIONS`).
|
|
20
|
+
- `#rotate?(current_size)` returns true when size exceeds `max_bytes`.
|
|
21
|
+
- `#plan_for(path)` returns `{ deletes: [...], moves: [...] }` describing
|
|
22
|
+
the rename dance: drop the oldest rotation slot, shift each remaining
|
|
23
|
+
rotation up by one suffix, then promote the current log to `.1`.
|
|
24
|
+
|
|
25
|
+
### AuditLog integration
|
|
26
|
+
- `AuditLog.new(rotation_policy:)` accepts a policy (nil disables rotation).
|
|
27
|
+
- `#record` consults the policy on every write under an exclusive flock
|
|
28
|
+
on a sibling lockfile (`audit.log.lock`). The lock target is the
|
|
29
|
+
lockfile rather than the log itself because the log path moves during
|
|
30
|
+
rotation; locking the log would carry the lock with it.
|
|
31
|
+
- `#entries` and `#each` walk the current log and every rotation
|
|
32
|
+
(newest-first).
|
|
33
|
+
- `#clear!` removes the current log and every rotation.
|
|
34
|
+
- `BaseCache#default_audit_log` constructs the policy from
|
|
35
|
+
`Configuration.instance` so ENV overrides and programmatic settings
|
|
36
|
+
flow through naturally.
|
|
37
|
+
|
|
38
|
+
### Config
|
|
39
|
+
Added to `Configuration::SCHEMA`:
|
|
40
|
+
- `audit_max_bytes` (default 10_485_760) via `KOTOSHU_AUDIT_MAX_BYTES`.
|
|
41
|
+
- `audit_rotations` (default 5) via `KOTOSHU_AUDIT_ROTATIONS`.
|
|
42
|
+
|
|
43
|
+
## Acceptance
|
|
44
|
+
|
|
45
|
+
- [x] Audit log never exceeds `audit_max_bytes * (audit_rotations + 1)`.
|
|
46
|
+
- [x] Rotation preserves previous entries.
|
|
47
|
+
- [x] No lock contention — writes are synchronous and serialized via
|
|
48
|
+
flock on a stable sibling lockfile.
|
|
49
|
+
|
|
50
|
+
## Why deferred (originally)
|
|
51
|
+
Audit log is append-only and unbounded today, but doesn't actively break
|
|
52
|
+
anything for 0.3 users. Rotation is a known best practice; shipping
|
|
53
|
+
without it for 0.3 is acceptable.
|
|
54
|
+
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# 34 — Cache eviction by size / TTL (T3, deferred past 0.3)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Implemented (2026-06-29).
|
|
5
|
+
|
|
6
|
+
## Problem
|
|
7
|
+
`~/.cache/kotoshu/` grows unbounded. With 9 ONNX models at 114 MB each
|
|
8
|
+
plus spelling dictionaries, a polyglot user can hit 1+ GB easily. There
|
|
9
|
+
is a `kotoshu cache purge` command but no automatic eviction.
|
|
10
|
+
|
|
11
|
+
## Plan
|
|
12
|
+
|
|
13
|
+
### Eviction policy
|
|
14
|
+
Two knobs in `Configuration::SCHEMA`:
|
|
15
|
+
- `cache_max_size` (default 1 GB) — LRU eviction when exceeded.
|
|
16
|
+
- `cache_ttl` (default 7 days for dictionaries, 30 days for models) —
|
|
17
|
+
time-based eviction on read.
|
|
18
|
+
|
|
19
|
+
### Implementation
|
|
20
|
+
- Each cache entry has `cached_at` in its `metadata.json` (already does).
|
|
21
|
+
- New `Cache::EvictionPolicy` model: decides what to evict given current
|
|
22
|
+
state and limits.
|
|
23
|
+
- `Kotoshu::Cache::BaseCache#get` checks TTL on load; evicts if expired.
|
|
24
|
+
- New `kotoshu cache evict` command runs LRU sweep to enforce size cap.
|
|
25
|
+
|
|
26
|
+
### Algorithm
|
|
27
|
+
LRU by `cached_at` (oldest evicted first until total fits the cap).
|
|
28
|
+
`metadata.json` without a parseable `cached_at` sorts oldest so it is
|
|
29
|
+
not stranded forever.
|
|
30
|
+
|
|
31
|
+
## Acceptance
|
|
32
|
+
|
|
33
|
+
- [x] Cache never exceeds `cache_max_size` after a `kotoshu cache evict`.
|
|
34
|
+
- [x] Entries older than `cache_ttl` are re-downloaded on next access
|
|
35
|
+
(existing TTL check on `BaseCache#get`).
|
|
36
|
+
- [x] Eviction never touches resources currently being read — eviction
|
|
37
|
+
is user-invoked (`kotoshu cache evict`), so the caller has opted out
|
|
38
|
+
of a read mid-flight. A read-time lock would solve this fully; out of
|
|
39
|
+
scope for this tier.
|
|
40
|
+
- [x] `kotoshu cache evict --dry-run` shows what would be evicted.
|
|
41
|
+
|
|
42
|
+
## Implementation notes
|
|
43
|
+
|
|
44
|
+
`Cache::EvictionPolicy` is a pure value object: `plan(entries)` returns
|
|
45
|
+
`{evict:, keep:, bytes_reclaimed:}` with no IO. `BaseCache#evict`
|
|
46
|
+
collects one record per `metadata.json` under `cache_path`, computes
|
|
47
|
+
per-directory disk size (`dir_size`), consults the policy, and either
|
|
48
|
+
returns the plan (`dry_run: true`) or removes the directories in
|
|
49
|
+
`plan[:evict]` via `FileUtils.rm_rf`.
|
|
50
|
+
|
|
51
|
+
The CLI subcommand lives at `lib/kotoshu/cli/cache_command.rb` and
|
|
52
|
+
prints either the dry-run plan or the post-eviction summary.
|
|
53
|
+
|
|
54
|
+
## Why deferred originally
|
|
55
|
+
Eviction is correct behavior but not a 0.3 blocker. Disk space is cheap;
|
|
56
|
+
users can manually `kotoshu cache purge`. Auto-eviction risks bugs that
|
|
57
|
+
delete things users wanted. The 0.4 implementation is opt-in (the user
|
|
58
|
+
runs `kotoshu cache evict`); auto-eviction-on-write is left for a later
|
|
59
|
+
tier.
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
# 35 — Shell completion (T3, deferred past 0.3)
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
**Implemented** (2026-06-29). Shipped as `kotoshu completions bash|zsh|fish`
|
|
5
|
+
plus a `kotoshu completions languages` helper that the completion scripts
|
|
6
|
+
shell out to for dynamic language completion. README documents install
|
|
7
|
+
paths for each shell.
|
|
8
|
+
|
|
9
|
+
## Problem
|
|
10
|
+
CLI users want tab completion for `kotoshu <TAB>` → subcommands, and
|
|
11
|
+
`kotoshu setup <TAB>` → language codes.
|
|
12
|
+
|
|
13
|
+
## Plan (as implemented)
|
|
14
|
+
|
|
15
|
+
### `kotoshu completions <shell>`
|
|
16
|
+
Three Thor subcommands (`bash`, `zsh`, `fish`) emit a shell-specific
|
|
17
|
+
completion script. The scripts shell out to `kotoshu completions languages`
|
|
18
|
+
for dynamic language completion so newly registered languages appear
|
|
19
|
+
without re-running the install step.
|
|
20
|
+
|
|
21
|
+
### `kotoshu completions languages`
|
|
22
|
+
Prints every code from `Kotoshu::Language::Registry.supported_codes`,
|
|
23
|
+
one per line.
|
|
24
|
+
|
|
25
|
+
### Catalog
|
|
26
|
+
`Kotoshu::Cli::Completions::COMMANDS` is a static catalog of
|
|
27
|
+
`Command.new(name:, description:)` records. Descriptions are user-facing
|
|
28
|
+
and curated here (rather than introspected from Thor) so the completion
|
|
29
|
+
menu reads cleanly. `LANGUAGE_ARGUMENT_COMMANDS` lists the commands
|
|
30
|
+
whose first positional argument is a language code (currently
|
|
31
|
+
`setup`, `fetch`).
|
|
32
|
+
|
|
33
|
+
### Builders
|
|
34
|
+
`Kotoshu::Cli::Completions::ScriptBuilders::{Bash,Zsh,Fish}` are pure
|
|
35
|
+
template builders that take the catalog and render the shell-specific
|
|
36
|
+
script. They have no IO dependencies and are unit-tested in isolation.
|
|
37
|
+
|
|
38
|
+
### README install paths
|
|
39
|
+
[source,bash]
|
|
40
|
+
----
|
|
41
|
+
# bash (system)
|
|
42
|
+
kotoshu completions bash > /etc/bash_completion.d/kotoshu
|
|
43
|
+
# bash (user)
|
|
44
|
+
kotoshu completions bash > ~/.local/share/bash-completion/completions/kotoshu
|
|
45
|
+
# zsh
|
|
46
|
+
kotoshu completions zsh > "${fpath[1]}/_kotoshu"
|
|
47
|
+
# fish
|
|
48
|
+
kotoshu completions fish > ~/.config/fish/completions/kotoshu.fish
|
|
49
|
+
----
|
|
50
|
+
|
|
51
|
+
## Acceptance
|
|
52
|
+
|
|
53
|
+
- [x] `kotoshu completions bash|zsh|fish` emits valid scripts.
|
|
54
|
+
- [x] Installing the bash script enables `kotoshu <TAB>` to show subcommands.
|
|
55
|
+
- [x] `kotoshu setup <TAB>` offers language codes (via
|
|
56
|
+
`kotoshu completions languages`).
|
|
57
|
+
- [x] README documents installation for each shell.
|
|
58
|
+
|
|
59
|
+
## Why deferred (originally)
|
|
60
|
+
Polish feature. Doesn't affect first-use or core spell-checking.
|
|
61
|
+
|