kotoshu 1.0.2-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,232 @@
|
|
|
1
|
+
# 39 — Tier 3 and Beyond
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
The long-form backlog. Each item here is a *capability* rather than a
|
|
6
|
+
bug fix. Items are roughly prioritized; the order is not a commitment.
|
|
7
|
+
|
|
8
|
+
This file exists so we don't lose institutional memory between
|
|
9
|
+
releases. When an item becomes the next priority, promote it to its
|
|
10
|
+
own `TODO.impl/{n}-{name}.md` file with phases and acceptance criteria,
|
|
11
|
+
then delete the entry here.
|
|
12
|
+
|
|
13
|
+
## T3 — Capabilities
|
|
14
|
+
|
|
15
|
+
### T3.1 — CJK support (promote from `TODO.impl/06-cjk-support.md`)
|
|
16
|
+
|
|
17
|
+
Status: outline only. Real work needs:
|
|
18
|
+
|
|
19
|
+
- Tokenizer that handles Chinese (no spaces), Japanese (mixed kana /
|
|
20
|
+
kanji), Korean (spaces + Hangul).
|
|
21
|
+
- Dictionary format: CJK dictionaries don't have affixes; they're
|
|
22
|
+
frequency-sorted word lists.
|
|
23
|
+
- Suggestion strategy: edit-distance on Unicode codepoints is wrong;
|
|
24
|
+
need pinyin (zh) / romaji (ja) / romanization (ko) aware edit distance.
|
|
25
|
+
- suika (current tokenizer) has segmentation; verify CJK coverage.
|
|
26
|
+
|
|
27
|
+
Reference: `/Users/mulgogi/src/external/cspell/` has a working CJK
|
|
28
|
+
implementation in TypeScript; port the algorithms, not the code.
|
|
29
|
+
|
|
30
|
+
### T3.2 — RTL support (promote from `TODO.impl/07-rtl-support.md`)
|
|
31
|
+
|
|
32
|
+
Status: outline only. Real work needs:
|
|
33
|
+
|
|
34
|
+
- Arabic / Hebrew normalizers (NFC, ligature handling, diacritic stripping).
|
|
35
|
+
- RTL-aware display in the CLI batch reporter and interactive reviewer.
|
|
36
|
+
- Right-to-left mark / left-to-right mark handling in the tokenizer
|
|
37
|
+
(see `spec/integrational/fixtures/right_to_left_mark.*`).
|
|
38
|
+
|
|
39
|
+
### T3.3 — Grammar engine (promote from `TODO.impl/08-grammar-engine.md`)
|
|
40
|
+
|
|
41
|
+
Status: skeleton in `lib/kotoshu/grammar/`. Real work needs:
|
|
42
|
+
|
|
43
|
+
- Rule format stabilized (currently `Rule` + `PatternMatchers::*`).
|
|
44
|
+
- A first useful rule set (probably article/confusion pairs: a/an,
|
|
45
|
+
its/it's, their/there/they're).
|
|
46
|
+
- Integration with the semantic analyzer for context-aware rules.
|
|
47
|
+
- A way to load rule packs per language.
|
|
48
|
+
|
|
49
|
+
Reference: `/Users/mulgogi/src/external/languagetool/` (Java) for rule
|
|
50
|
+
shape and category taxonomy. Don't reimplement LT's category model
|
|
51
|
+
verbatim — port the *shape*, not the 5000-rule corpus.
|
|
52
|
+
|
|
53
|
+
### T3.4 — Document plugins
|
|
54
|
+
|
|
55
|
+
The boundary is decided (see `kotoshu-document-plugin-boundary.md`
|
|
56
|
+
memory): kotoshu never owns document parsing. Existing plugins:
|
|
57
|
+
|
|
58
|
+
- `coradoc-plugin-kotoshu` (AsciiDoc via coradoc).
|
|
59
|
+
- Markdown: kotoshu has a built-in `documents/markdown_document.rb`.
|
|
60
|
+
This violates the boundary; migrate to a plugin or formalize the
|
|
61
|
+
exception.
|
|
62
|
+
- Plain text: built-in, keep.
|
|
63
|
+
- reStructuredText, Org, HTML, LaTeX, etc.: community plugins.
|
|
64
|
+
|
|
65
|
+
Work: define the `Kotoshu::Plugin::Document` interface and the
|
|
66
|
+
registration mechanism, then audit existing built-ins against it.
|
|
67
|
+
|
|
68
|
+
### T3.5 — Multi-language document checking
|
|
69
|
+
|
|
70
|
+
Right now `Kotoshu.check_file` is single-language. Real documents mix
|
|
71
|
+
languages (code comments in English + prose in French, etc.).
|
|
72
|
+
|
|
73
|
+
Work:
|
|
74
|
+
|
|
75
|
+
- Per-paragraph language detection via `Language::Identifier`.
|
|
76
|
+
- Per-language dictionary resolution.
|
|
77
|
+
- Result aggregation across languages.
|
|
78
|
+
|
|
79
|
+
### T3.6 — Personal dictionary management
|
|
80
|
+
|
|
81
|
+
The personal dictionary (`~/.config/kotoshu/personal_dictionary.txt`
|
|
82
|
+
or similar) exists but is undertested. Work:
|
|
83
|
+
|
|
84
|
+
- CLI for add/remove/list/import.
|
|
85
|
+
- Per-project personal dictionaries (`.kotoshu/` in project root).
|
|
86
|
+
- Sync with Hunspell-style personal dict format for interop.
|
|
87
|
+
|
|
88
|
+
## T4 — Quality
|
|
89
|
+
|
|
90
|
+
### T4.1 — Performance pass
|
|
91
|
+
|
|
92
|
+
`spec/benchmark/` and `spec/performance/` exist but are skipped by
|
|
93
|
+
default. Work:
|
|
94
|
+
|
|
95
|
+
- Establish baselines for: cold-start, lookup throughput,
|
|
96
|
+
suggest latency at 1k/10k/100k dictionary size.
|
|
97
|
+
- Profile the hot path (`Kotoshu.correct?`); attack the biggest
|
|
98
|
+
offender.
|
|
99
|
+
- Consider Memoization, Trie compression, Bloom filter tuning.
|
|
100
|
+
|
|
101
|
+
### T4.2 — CI
|
|
102
|
+
|
|
103
|
+
GitHub Actions workflow that runs:
|
|
104
|
+
|
|
105
|
+
- `bundle exec rspec --tag ~network --tag ~onnx` on every push.
|
|
106
|
+
- `bundle exec rubocop` on every push.
|
|
107
|
+
- `NETWORK_TESTS=1 bundle exec rspec` nightly on `main`.
|
|
108
|
+
- `ONNX_TESTS=1 bundle exec rspec` nightly on `main` (separate job;
|
|
109
|
+
cache the model).
|
|
110
|
+
|
|
111
|
+
### T4.3 — Property-based testing
|
|
112
|
+
|
|
113
|
+
`spec/properties/` exists but is thin. Expand:
|
|
114
|
+
|
|
115
|
+
- Trie: insertion is order-independent; lookup is total.
|
|
116
|
+
- Suggestion generator: `suggestions.include?(word)` implies
|
|
117
|
+
`dictionary.lookup?(word)` is true (no out-of-dictionary suggestions).
|
|
118
|
+
- Hunspell affix application: affixed forms round-trip through the
|
|
119
|
+
stemmer.
|
|
120
|
+
|
|
121
|
+
### T4.4 — Documentation pass
|
|
122
|
+
|
|
123
|
+
- README.adoc: audit against actual CLI shape; remove 0.2-era examples.
|
|
124
|
+
- `docs/`: prune superseded planning docs; mark historical ones as
|
|
125
|
+
historical.
|
|
126
|
+
- YARD: every public API has a docstring; private APIs are marked
|
|
127
|
+
`@private`.
|
|
128
|
+
|
|
129
|
+
## T5 — Architecture
|
|
130
|
+
|
|
131
|
+
### T5.1 — MECE refactor of `suggestions/strategies/`
|
|
132
|
+
|
|
133
|
+
Current strategies overlap (e.g. `symspell` and `edit_distance` both
|
|
134
|
+
generate edit-distance-1 candidates). Refactor:
|
|
135
|
+
|
|
136
|
+
- Each strategy owns a *unique* candidate-generation mechanism.
|
|
137
|
+
- `CompositeStrategy` merges and ranks, doesn't dedupe by side effect.
|
|
138
|
+
- Strategies are pure functions of `(word, dictionary) → candidates`;
|
|
139
|
+
no shared state, no global config reads.
|
|
140
|
+
|
|
141
|
+
### T5.2 — Configuration as data, not singleton
|
|
142
|
+
|
|
143
|
+
`Configuration.instance` is a process-wide singleton. Hard to test,
|
|
144
|
+
hard to scope per-document. Refactor:
|
|
145
|
+
|
|
146
|
+
- Pass a `Configuration` instance explicitly to `Spellchecker.new`.
|
|
147
|
+
- Default the facade (`Kotoshu.correct?`) to a process-default instance.
|
|
148
|
+
- Drop the singleton.
|
|
149
|
+
|
|
150
|
+
### T5.3 — Result model consistency
|
|
151
|
+
|
|
152
|
+
`WordResult` / `DocumentResult` / `SuggestionSet` migrated to
|
|
153
|
+
`lutaml-model` in 0.3.0. Audit:
|
|
154
|
+
|
|
155
|
+
- All three have consistent `to_hash` / `from_hash` shapes.
|
|
156
|
+
- No hand-rolled `to_h` anywhere (per global rule).
|
|
157
|
+
- CLI consumers use the framework-supplied serialization.
|
|
158
|
+
|
|
159
|
+
### T5.4 — Plugin architecture formalization
|
|
160
|
+
|
|
161
|
+
`Kotoshu::Plugin::*` exists but is informal. Define:
|
|
162
|
+
|
|
163
|
+
- Plugin discovery (`Gem.find_files` or explicit `Kotoshu::Plugin.register`).
|
|
164
|
+
- Plugin lifecycle (load, configure, start, stop).
|
|
165
|
+
- Plugin capabilities (document parser, suggestion strategy, language
|
|
166
|
+
module, output format).
|
|
167
|
+
|
|
168
|
+
## T6 — Ecosystem
|
|
169
|
+
|
|
170
|
+
### T6.1 — VSCode / LSP integration
|
|
171
|
+
|
|
172
|
+
A language server that runs kotoshu on save, reports diagnostics via
|
|
173
|
+
LSP. Probably a separate gem (`kotoshu-lsp`).
|
|
174
|
+
|
|
175
|
+
### T6.2 — Web playground
|
|
176
|
+
|
|
177
|
+
A WASM build of kotoshu running in the browser. Requires a pure-Ruby
|
|
178
|
+
path (no native extensions) and a reasonable dictionary size budget.
|
|
179
|
+
|
|
180
|
+
### T6.3 — Dictionary editor GUI
|
|
181
|
+
|
|
182
|
+
A desktop app (Tauri / Electron) for curating personal dictionaries and
|
|
183
|
+
custom affix files. Reads/writes Hunspell format.
|
|
184
|
+
|
|
185
|
+
## Acceptance criteria for T3+ releases
|
|
186
|
+
|
|
187
|
+
Each item, when promoted, gets its own acceptance criteria. The
|
|
188
|
+
overarching rule: **never break T1/T2 correctness for a T3+ feature**.
|
|
189
|
+
Re-run the full `--tag ~network --tag ~onnx` suite before tagging any
|
|
190
|
+
T3+ release.
|
|
191
|
+
|
|
192
|
+
## Dependencies
|
|
193
|
+
|
|
194
|
+
- **Blocked by:** T1 (`TODO.impl/36`) and T2 (`TODO.impl/37`) — the
|
|
195
|
+
baseline must be green before adding features.
|
|
196
|
+
- **Blocks:** nothing. This is the long horizon.
|
|
197
|
+
|
|
198
|
+
## Status (reconciled 2026-09-03)
|
|
199
|
+
|
|
200
|
+
Verified shipped, with evidence:
|
|
201
|
+
|
|
202
|
+
- **T3.6 Personal dictionary** — `kotoshu personal add/remove/list/
|
|
203
|
+
import/path/clear` all exist (lib/kotoshu/cli/personal_command.rb,
|
|
204
|
+
wired in cli.rb) and are documented on the site (docs/cli).
|
|
205
|
+
- **T5.3 Result models on lutaml-model** — shipped in the 0.3.0
|
|
206
|
+
migration (per plan 36 history); `to_hash`/`from_hash` framework
|
|
207
|
+
paths in use.
|
|
208
|
+
- **T4.2 CI** — workflows exist (rake matrix on 3.3/3.4/4.0 ×
|
|
209
|
+
ubuntu/macOS/windows); the network-test job is red on main
|
|
210
|
+
(pre-existing, tracked separately). Do not treat CI-green as done.
|
|
211
|
+
- **T6.1 LSP** — shipped as the separate gem kotoshu-lsp 0.1.0
|
|
212
|
+
(github.com/kotoshu/kotoshu-lsp); superseded by the ecosystem track
|
|
213
|
+
(plans 60-64, all implemented).
|
|
214
|
+
- **T6.2 Web playground** — live at kotoshu.org/playground against
|
|
215
|
+
kotoshu-server; the WASM variant in this item is superseded by
|
|
216
|
+
kotoshu-rs P4 (gem plan 66) — in flight 2026-09-03.
|
|
217
|
+
|
|
218
|
+
Still backlog (unverified or not started):
|
|
219
|
+
|
|
220
|
+
- T3.1 CJK (superseded by TODO.impl/54-cjk-support.md and the 9-model
|
|
221
|
+
corpus now shipped for ja/ko/zh), T3.2 RTL (superseded by 55), T3.3
|
|
222
|
+
grammar expansion (superseded by 51), T3.4/T3.5 document plugins and
|
|
223
|
+
multi-language documents (50/52/53).
|
|
224
|
+
- T4.1 performance pass (benchmark baselines), T4.3 property tests,
|
|
225
|
+
T4.4 documentation pass (partially superseded by the 2026-09 docs
|
|
226
|
+
work on the site and README tiers).
|
|
227
|
+
- T5.1 strategy MECE refactor, T5.2 configuration-as-data
|
|
228
|
+
(58 tracks the design), T5.4 plugin formalization.
|
|
229
|
+
- T6.3 dictionary editor GUI — untouched.
|
|
230
|
+
|
|
231
|
+
This file remains the long-horizon index; promote items to their own
|
|
232
|
+
plans (as 50-59 already did) when they become priority.
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
# 40 — Cache spec / CLI drift cleanup
|
|
2
|
+
|
|
3
|
+
## Status
|
|
4
|
+
Discovered while running the full suite after T2 Phase 2B completion.
|
|
5
|
+
Pre-existing — not introduced by T2 work.
|
|
6
|
+
|
|
7
|
+
## Problem
|
|
8
|
+
|
|
9
|
+
Two distinct spec/impl drifts cause **39 failures** in the full suite
|
|
10
|
+
that are unrelated to T2 correctness:
|
|
11
|
+
|
|
12
|
+
### Drift A — `LanguageCache` KV interface mismatch (25 failures)
|
|
13
|
+
|
|
14
|
+
`spec/kotoshu/cache/language_cache_spec.rb` expects a generic key-value
|
|
15
|
+
cache API on `Kotoshu::Cache::LanguageCache`:
|
|
16
|
+
|
|
17
|
+
- `read(key)` / `write(key, value)` / `fetch(key) { ... }`
|
|
18
|
+
- `delete(key)` / `key?(key)` / `clear` (no arg) / `size`
|
|
19
|
+
- `stats` returning `:hits, :misses, :size, :hit_rate,
|
|
20
|
+
:total_size_bytes, :cached_languages, :oldest_entry`
|
|
21
|
+
- `max_cache_size` attribute (1 GB default)
|
|
22
|
+
- `language_path(lang, type)` public method
|
|
23
|
+
|
|
24
|
+
But `LanguageCache < BaseCache` is a **disk-backed resource downloader**,
|
|
25
|
+
not a KV cache. Its actual API is `get / available? / clear(id) /
|
|
26
|
+
clear_all / stats / clean / get_spelling / get_grammar /
|
|
27
|
+
frequency_available? / cached_resources / language_info / install_local`.
|
|
28
|
+
|
|
29
|
+
The KV interface already lives on `LookupCache` and `SuggestionCache`
|
|
30
|
+
(in-memory LRU), which have their own dedicated specs at
|
|
31
|
+
`spec/kotoshu/cache/{lookup,suggestion}_cache_spec.rb`. Mixing the two
|
|
32
|
+
abstractions would violate the MECE principle.
|
|
33
|
+
|
|
34
|
+
The spec was written TDD-style before the implementation crystallized.
|
|
35
|
+
The implementation went the MECE route (KV → `LookupCache`, resources →
|
|
36
|
+
`LanguageCache`), and the spec was never reconciled.
|
|
37
|
+
|
|
38
|
+
### Drift B — Wired CLI uses nonexistent `LanguageCache` methods (12 failures)
|
|
39
|
+
|
|
40
|
+
`lib/kotoshu/cli.rb` registers `subcommand "cache", CacheCommand` and
|
|
41
|
+
loads `lib/kotoshu/cli/cache_command.rb` (the `Kotoshu::Cli::CacheCommand`
|
|
42
|
+
class). That class calls **methods that do not exist** on
|
|
43
|
+
`LanguageCache`:
|
|
44
|
+
|
|
45
|
+
| Call in `cli/cache_command.rb` | Actual `LanguageCache` API |
|
|
46
|
+
|--------------------------------------|----------------------------|
|
|
47
|
+
| `cache.cache_status` | (none — must construct) |
|
|
48
|
+
| `cache.get_frequency_data(lang)` | (private `download_frequency`) |
|
|
49
|
+
| `cache.get_language_info(lang)` | `cache.language_info(lang)`|
|
|
50
|
+
| `cache.purge_all` | `cache.clear_all` |
|
|
51
|
+
|
|
52
|
+
A parallel implementation exists at `lib/kotoshu/commands/cache_command.rb`
|
|
53
|
+
(`Kotoshu::CacheCommand` — no `Cli` namespace) that **uses the correct
|
|
54
|
+
API** but is **not wired up** in `cli.rb`. So `kotoshu cache list` and
|
|
55
|
+
friends crash on every invocation in the current build.
|
|
56
|
+
|
|
57
|
+
`spec/kotoshu/cli/cache_command_spec.rb` exercises the broken
|
|
58
|
+
`Kotoshu::Cli::CacheCommand` and therefore fails 12 examples.
|
|
59
|
+
|
|
60
|
+
### Drift C — Performance regression (2 failures)
|
|
61
|
+
|
|
62
|
+
`spec/performance/performance_regression_spec.rb:40,49` assert
|
|
63
|
+
suggestion latency under fixed thresholds (10ms, "with cache faster than
|
|
64
|
+
without"). These are timing-sensitive, machine-load-dependent, and not
|
|
65
|
+
related to T2.
|
|
66
|
+
|
|
67
|
+
## Plan
|
|
68
|
+
|
|
69
|
+
### Phase 1 — Realign `language_cache_spec.rb` to actual API
|
|
70
|
+
|
|
71
|
+
Rewrite the spec to test what `LanguageCache` actually does:
|
|
72
|
+
|
|
73
|
+
- `initialize` (path, ttl, registry wiring) — keep, drop
|
|
74
|
+
`max_cache_size` assertion unless we add the attribute (see below).
|
|
75
|
+
- `available?(resource_id)` — replace the KV "key?" tests.
|
|
76
|
+
- `cached_resources` — replace the KV "size" tests.
|
|
77
|
+
- `clear_all` / `clear(resource_id)` — replace the KV "clear" tests.
|
|
78
|
+
- `stats` — assert the **actual** shape (`hits, misses, total,
|
|
79
|
+
hit_rate, cached_resources, size_bytes, oldest_entry`).
|
|
80
|
+
- `clean` — keep, already correct.
|
|
81
|
+
- `get_spelling` / `get_grammar` — keep, already correct.
|
|
82
|
+
- Drop the `read/write/fetch/delete` KV tests entirely. They duplicate
|
|
83
|
+
`spec/kotoshu/cache/lookup_cache_spec.rb` and test the wrong class.
|
|
84
|
+
|
|
85
|
+
**Add** a `language_path(lang, type)` public helper on `LanguageCache`
|
|
86
|
+
as a useful, well-named accessor — it makes test setup and downstream
|
|
87
|
+
code (CLI status displays, audit log relative paths) clearer than
|
|
88
|
+
reaching into `cache_path` and joining paths by hand.
|
|
89
|
+
|
|
90
|
+
**Optional:** add `max_cache_size` attribute (default 1 GB) as
|
|
91
|
+
forward-looking work for `TODO.impl/34-cache-eviction.md`. Just an
|
|
92
|
+
`attr_reader` with a constructor kwarg — no eviction logic yet.
|
|
93
|
+
|
|
94
|
+
### Phase 2 — Document and quarantine the broken CLI
|
|
95
|
+
|
|
96
|
+
Do **not** delete `lib/kotoshu/cli/cache_command.rb` (global rule:
|
|
97
|
+
never delete source). Instead:
|
|
98
|
+
|
|
99
|
+
1. Mark the 12 failing examples in `spec/kotoshu/cli/cache_command_spec.rb`
|
|
100
|
+
as `pending` with a clear pointer to this file and the underlying
|
|
101
|
+
API gap.
|
|
102
|
+
2. Leave it to a follow-up task to either:
|
|
103
|
+
- Consolidate the two CLIs (delete `cli/cache_command.rb`, rewire
|
|
104
|
+
`cli.rb` to `commands/cache_command.rb`); OR
|
|
105
|
+
- Fix `cli/cache_command.rb` to call the real `LanguageCache` API.
|
|
106
|
+
|
|
107
|
+
That decision is bigger than spec cleanup — it touches the public CLI
|
|
108
|
+
surface and needs design judgment. Defer.
|
|
109
|
+
|
|
110
|
+
### Phase 3 — Quarantine performance regression specs
|
|
111
|
+
|
|
112
|
+
Mark `spec/performance/performance_regression_spec.rb:40,49` as
|
|
113
|
+
`:slow` (they already should be) or `pending` with reason
|
|
114
|
+
"timing-sensitive; revisit after T4.1 performance pass".
|
|
115
|
+
|
|
116
|
+
## Acceptance
|
|
117
|
+
|
|
118
|
+
- [ ] `bundle exec rspec spec/kotoshu/cache/language_cache_spec.rb`
|
|
119
|
+
is green and tests the actual `LanguageCache` API.
|
|
120
|
+
- [ ] `bundle exec rspec spec/kotoshu/cli/cache_command_spec.rb`
|
|
121
|
+
has no unexpected failures (all pending with documented reasons).
|
|
122
|
+
- [ ] `bundle exec rspec spec/performance/performance_regression_spec.rb`
|
|
123
|
+
has no unexpected failures.
|
|
124
|
+
- [ ] `lib/kotoshu/cache/language_cache.rb` adds `language_path` and
|
|
125
|
+
(optionally) `max_cache_size`.
|
|
126
|
+
- [ ] No production code regressions — `bundle exec rspec spec/kotoshu`
|
|
127
|
+
minus the three quarantined files is fully green.
|
|
128
|
+
|
|
129
|
+
## Out of scope for this file
|
|
130
|
+
|
|
131
|
+
- Consolidating the two `cache_command.rb` implementations.
|
|
132
|
+
- Fixing the wired-up CLI (`kotoshu cache ...`) end-to-end.
|
|
133
|
+
- Cache eviction (see `TODO.impl/34-cache-eviction.md`).
|
|
134
|
+
- Performance baseline work (see `TODO.impl/39-tier3-and-beyond.md` T4.1).
|
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
# 50 — Structure-aware document checking API
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Define the value-object + parser-contract layer that lets document
|
|
6
|
+
plugins (coradoc-plugin-kotoshu, markdown-plugin-kotoshu, etc.) perform
|
|
7
|
+
**structure-aware** spell/grammar checks: the checker scans the
|
|
8
|
+
flattened text, but every error carries a position that maps back to
|
|
9
|
+
the original (markup-bearing) source.
|
|
10
|
+
|
|
11
|
+
## Motivation
|
|
12
|
+
|
|
13
|
+
Concrete example from the user:
|
|
14
|
+
|
|
15
|
+
> Source: `I'm an **friend** of Tom`
|
|
16
|
+
> Flattened: `I'm an friend of Tom`
|
|
17
|
+
> Error: "an friend" should be "a friend"
|
|
18
|
+
|
|
19
|
+
The grammar checker sees the flattened sentence and flags "an friend".
|
|
20
|
+
But the error report must reference the original source — specifically
|
|
21
|
+
the range `an **friend**` — so an editor can highlight the actual
|
|
22
|
+
markdown/AsciiDoc the user wrote, not the stripped text.
|
|
23
|
+
|
|
24
|
+
This is the **plugin's** job (kotoshu never owns document parsing —
|
|
25
|
+
see `kotoshu-document-plugin-boundary.md` memory). Kotoshu's job is to
|
|
26
|
+
provide:
|
|
27
|
+
|
|
28
|
+
1. Value objects for source position + range.
|
|
29
|
+
2. A TextNode model that pairs flattened text with its source range.
|
|
30
|
+
3. A Document interface with bidirectional offset mapping.
|
|
31
|
+
4. An error model that carries source positions.
|
|
32
|
+
5. A registry where plugins register format-specific parsers.
|
|
33
|
+
|
|
34
|
+
## Design
|
|
35
|
+
|
|
36
|
+
### Value objects
|
|
37
|
+
|
|
38
|
+
Kotoshu::Documents::SourcePosition
|
|
39
|
+
offset : Integer # 0-based char offset in the original source
|
|
40
|
+
line : Integer # 1-based line number
|
|
41
|
+
column : Integer # 1-based column (chars from line start)
|
|
42
|
+
|
|
43
|
+
Kotoshu::Documents::SourceRange
|
|
44
|
+
start : SourcePosition # inclusive
|
|
45
|
+
end : SourcePosition # exclusive
|
|
46
|
+
|
|
47
|
+
Both are `Struct.new(..., keyword_init: true)` with `freeze` on
|
|
48
|
+
construct. Comparable via `include Comparable` so ranges can be sorted
|
|
49
|
+
and tested for containment.
|
|
50
|
+
|
|
51
|
+
### Text node
|
|
52
|
+
|
|
53
|
+
Kotoshu::Documents::TextNode
|
|
54
|
+
text : String # flattened text (no markup)
|
|
55
|
+
source_range : SourceRange # where this node lives in source
|
|
56
|
+
flattened_offset : Integer # 0-based offset in concatenated flattened text
|
|
57
|
+
format : Symbol # :plain, :bold, :italic, :code, :heading, ...
|
|
58
|
+
metadata : Hash # format-specific ({level: 2} for h2, etc.)
|
|
59
|
+
|
|
60
|
+
A document is a list of text nodes. Concatenating `node.text` for
|
|
61
|
+
every node yields the flattened text the checker scans.
|
|
62
|
+
|
|
63
|
+
### Document interface
|
|
64
|
+
|
|
65
|
+
class Kotoshu::Documents::Document
|
|
66
|
+
attr_reader :text_nodes, :source, :format
|
|
67
|
+
|
|
68
|
+
def flattened_text
|
|
69
|
+
text_nodes.map(&:text).join
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
def flattened_length
|
|
73
|
+
text_nodes.sum { |n| n.text.length }
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
# Map a flattened-text offset to the source range containing it.
|
|
77
|
+
def source_range_at(flattened_offset)
|
|
78
|
+
# Walk nodes; find the one whose flattened window contains the
|
|
79
|
+
# offset. Return that node's source_range.
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
# Map a flattened-text range to the source range spanning it.
|
|
83
|
+
def source_range_for(flattened_start, flattened_end)
|
|
84
|
+
# Combine source_range_at(start).start with
|
|
85
|
+
# source_range_at(end - 1).end.
|
|
86
|
+
end
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
### Built-in: PlainTextDocument
|
|
90
|
+
|
|
91
|
+
class Kotoshu::Documents::PlainTextDocument < Document
|
|
92
|
+
def self.from_string(text, language_code: nil)
|
|
93
|
+
nodes = [TextNode.new(text: text, source_range: full_range(text),
|
|
94
|
+
flattened_offset: 0, format: :plain, metadata: {})]
|
|
95
|
+
new(text_nodes: nodes, source: text, format: :plain)
|
|
96
|
+
end
|
|
97
|
+
|
|
98
|
+
def self.from_file(path, language_code: nil)
|
|
99
|
+
from_string(File.read(path), language_code: language_code)
|
|
100
|
+
end
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
PlainTextDocument is the only parser kotoshu ships. Markdown, AsciiDoc,
|
|
104
|
+
reStructuredText, etc. are plugin territory.
|
|
105
|
+
|
|
106
|
+
### Plugin parser registry
|
|
107
|
+
|
|
108
|
+
module Kotoshu::Documents
|
|
109
|
+
def self.register(format, parser_class)
|
|
110
|
+
# parser_class responds to .from_string(text, language_code:) and
|
|
111
|
+
# optionally .from_file(path, language_code:).
|
|
112
|
+
end
|
|
113
|
+
|
|
114
|
+
def self.parser_for(format)
|
|
115
|
+
# Returns the registered parser class or nil.
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
def self.parse(source, format:, language_code: nil)
|
|
119
|
+
parser = parser_for(format) || PlainTextDocument
|
|
120
|
+
parser.from_string(source, language_code: language_code)
|
|
121
|
+
end
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
A plugin (`coradoc-plugin-kotoshu`) does:
|
|
125
|
+
|
|
126
|
+
Kotoshu::Documents.register(:asciidoc, CoradocPluginKotoshu::AsciidocParser)
|
|
127
|
+
|
|
128
|
+
on load. The parser produces a `Kotoshu::Documents::Document` whose
|
|
129
|
+
text_nodes carry proper source_ranges (coradoc knows the offsets).
|
|
130
|
+
|
|
131
|
+
### Error model
|
|
132
|
+
|
|
133
|
+
Errors carry the source range directly. The checker resolves it from
|
|
134
|
+
the document before constructing the error:
|
|
135
|
+
|
|
136
|
+
class Kotoshu::Models::SemanticError
|
|
137
|
+
attr_reader :source_range, :original, :suggestions, ...
|
|
138
|
+
def initialize(source_range:, original:, suggestions:, ...)
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
For checkers that work on plain strings (no Document), `source_range`
|
|
142
|
+
is nil and the flattened position is the only positional info.
|
|
143
|
+
|
|
144
|
+
### Check pipeline
|
|
145
|
+
|
|
146
|
+
analyzer.analyze(document) do |error|
|
|
147
|
+
# error.source_range is already resolved
|
|
148
|
+
plugin_or_editor.highlight(error.source_range)
|
|
149
|
+
end
|
|
150
|
+
|
|
151
|
+
Existing analyzer code stays the same — it iterates text_nodes and
|
|
152
|
+
emits errors. The change is that errors now carry source_range from
|
|
153
|
+
the text_node they originated in.
|
|
154
|
+
|
|
155
|
+
## Phases
|
|
156
|
+
|
|
157
|
+
### Phase 1 — Core abstractions (this PR)
|
|
158
|
+
- SourcePosition, SourceRange, TextNode, Document, PlainTextDocument.
|
|
159
|
+
- Documents::Document.register / parser_for / parse registry.
|
|
160
|
+
- Specs for each.
|
|
161
|
+
|
|
162
|
+
### Phase 2 — Pipeline integration (this PR)
|
|
163
|
+
- SemanticAnalyzer accepts a Document, emits errors with source_range.
|
|
164
|
+
- WordResult / DocumentResult carry source_range.
|
|
165
|
+
- CheckCommand (currently dead code referencing non-existent
|
|
166
|
+
`Documents::Document`) is either removed or rewritten to use the new
|
|
167
|
+
API.
|
|
168
|
+
|
|
169
|
+
### Phase 3 — Plugin contract docs (this PR)
|
|
170
|
+
- README section "Writing a document plugin".
|
|
171
|
+
- YARD on `Kotoshu::Documents.register` documenting the parser class
|
|
172
|
+
contract.
|
|
173
|
+
|
|
174
|
+
### Phase 4 — coradoc-plugin-kotoshu (separate repo)
|
|
175
|
+
- coradoc-plugin-kotoshu implements the parser for AsciiDoc.
|
|
176
|
+
- Out of scope for kotoshu proper.
|
|
177
|
+
|
|
178
|
+
## Acceptance criteria
|
|
179
|
+
|
|
180
|
+
- [ ] SourcePosition, SourceRange, TextNode, Document, PlainTextDocument
|
|
181
|
+
all have direct specs (real instances, no doubles).
|
|
182
|
+
- [ ] `Document#source_range_for(flattened_start, flattened_end)` returns
|
|
183
|
+
the correct SourceRange for the example "I'm an **friend** of Tom".
|
|
184
|
+
- [ ] SemanticAnalyzer emits errors carrying `source_range`.
|
|
185
|
+
- [ ] A plugin can register via `Kotoshu::Documents.register(:fmt, klass)`.
|
|
186
|
+
- [ ] No `respond_to?`, `send`-to-private, `instance_variable_*`,
|
|
187
|
+
`require_relative` in the new code.
|
|
188
|
+
- [ ] Full suite stays green (2901+ examples, 0 failures).
|
|
189
|
+
|
|
190
|
+
## Dependencies
|
|
191
|
+
|
|
192
|
+
- **Blocked by:** none.
|
|
193
|
+
- **Blocks:** coradoc-plugin-kotoshu's structure-aware checking (that
|
|
194
|
+
plugin consumes this API).
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# 51 — Grammar engine expansion (T3.3)
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Ship useful grammar rule packs for English, building on the existing
|
|
6
|
+
`Kotoshu::Grammar` skeleton. Currently we have:
|
|
7
|
+
|
|
8
|
+
- `EN_A_VS_AN` (article usage before vowel/consonant sounds)
|
|
9
|
+
- `EN_THERE_THEIR` (their vs there)
|
|
10
|
+
- `EN_DOUBLE_NEGATIVE` (with the correlative-idiom exception fix
|
|
11
|
+
from PR #52)
|
|
12
|
+
|
|
13
|
+
Each pack adds one or more rules for a common confusion. The intent
|
|
14
|
+
is to demonstrate the engine works for real-world cases — not to
|
|
15
|
+
match LanguageTool's 5000-rule corpus.
|
|
16
|
+
|
|
17
|
+
## Phases
|
|
18
|
+
|
|
19
|
+
### Phase 1 — Possessive / contraction pairs
|
|
20
|
+
|
|
21
|
+
| Rule ID | Pattern | Why |
|
|
22
|
+
|-----------------|-----------------------------------------------|------------------------------|
|
|
23
|
+
| EN_ITS_IT_S | `its` vs `it's` from POS + context | Possessive vs contraction |
|
|
24
|
+
| EN_YOUR_YOURE | `your` vs `you're` | Possessive vs contraction |
|
|
25
|
+
| EN_WHOSE_WHO_IS | `whose` vs `who's` | Possessive vs contraction |
|
|
26
|
+
|
|
27
|
+
These all need a context-check matcher that looks at surrounding
|
|
28
|
+
verbs. Reuse the existing `PossessiveContextMatcher` pattern.
|
|
29
|
+
|
|
30
|
+
### Phase 2 — Common confusion pairs
|
|
31
|
+
|
|
32
|
+
| Rule ID | From → To | Detection heuristic |
|
|
33
|
+
|--------------------|----------------------------|------------------------------------|
|
|
34
|
+
| EN_THEN_THAN | `then` → `than` (and vice versa) | Pre-comparison ("bigger then") |
|
|
35
|
+
| EN_AFFECT_EFFECT | `affect` ↔ `effect` | POS (verb vs noun) |
|
|
36
|
+
| EN_ACCEPT_EXCEPT | `accept` ↔ `except` | Context (preposition vs verb) |
|
|
37
|
+
| EN_LOSE_LOOSE | `lose` ↔ `loose` | POS |
|
|
38
|
+
|
|
39
|
+
### Phase 3 — Capitalization
|
|
40
|
+
|
|
41
|
+
| Rule ID | Detection |
|
|
42
|
+
|------------------------|-------------------------------------------|
|
|
43
|
+
| EN_SENTENCE_START_CAP | First letter after `.`, `!`, `?` |
|
|
44
|
+
| EN_PROPER_NOUN_CAP | Common proper nouns ("monday", "english") |
|
|
45
|
+
|
|
46
|
+
Capitalization rules need a SentenceStartMatcher — new matcher type.
|
|
47
|
+
|
|
48
|
+
### Phase 4 — Rule packs per language
|
|
49
|
+
|
|
50
|
+
- Move `data/en/grammar/rules.yaml` rules into a `RulePack` abstraction.
|
|
51
|
+
- Each language can have multiple packs (base, formal, technical).
|
|
52
|
+
- Loader picks the right pack based on configuration.
|
|
53
|
+
|
|
54
|
+
## Design
|
|
55
|
+
|
|
56
|
+
- Each rule lives in `data/{lang}/grammar/rules.yaml` (data-driven,
|
|
57
|
+
same as today).
|
|
58
|
+
- New matcher types go in `lib/kotoshu/grammar/pattern_matchers/`.
|
|
59
|
+
- Rule engine itself doesn't change — Phase 4 just adds a RulePack
|
|
60
|
+
layer over the existing loader.
|
|
61
|
+
|
|
62
|
+
## Acceptance criteria
|
|
63
|
+
|
|
64
|
+
- [ ] At least 4 new rule packs shipped for English (Phase 1 + Phase 2).
|
|
65
|
+
- [ ] Each rule has direct specs in `spec/kotoshu/languages/en/`.
|
|
66
|
+
- [ ] No false positives on the example sentences in the rule
|
|
67
|
+
descriptions.
|
|
68
|
+
- [ ] Full suite stays green (2945+ examples, 0 failures).
|
|
69
|
+
|
|
70
|
+
## Dependencies
|
|
71
|
+
|
|
72
|
+
- **Blocked by:** none.
|
|
73
|
+
- **Blocks:** T3.4 multi-language document checking (the same engine
|
|
74
|
+
is used; per-paragraph language detection picks the right rule
|
|
75
|
+
pack).
|