kotoshu 1.0.2-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.kotoshu-baseline.json +1 -0
- data/.pre-commit-hooks.yaml +18 -0
- data/.rspec +3 -0
- data/.rubocop.yml +351 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +813 -0
- data/CLAUDE.md +176 -0
- data/CODE_OF_CONDUCT.md +132 -0
- data/Cargo.lock +359 -0
- data/Cargo.toml +15 -0
- data/LICENSE +31 -0
- data/README.adoc +1292 -0
- data/Rakefile +150 -0
- data/SECURITY.md +93 -0
- data/TODO.impl/00-cut-0.1.md +121 -0
- data/TODO.impl/00-cut-0.2.md +175 -0
- data/TODO.impl/00-vision.md +281 -0
- data/TODO.impl/01-hunspell-correctness.md +111 -0
- data/TODO.impl/02-cli-unification.md +74 -0
- data/TODO.impl/03-dynamic-download.md +107 -0
- data/TODO.impl/04-language-modules.md +89 -0
- data/TODO.impl/05-semantic-path.md +83 -0
- data/TODO.impl/06-cjk-support.md +105 -0
- data/TODO.impl/07-rtl-support.md +71 -0
- data/TODO.impl/08-grammar-engine.md +106 -0
- data/TODO.impl/09-integrity-security.md +90 -0
- data/TODO.impl/10-testing-ci.md +98 -0
- data/TODO.impl/100-full-feature-batch3.md +28 -0
- data/TODO.impl/101-int4-tier-experiment.md +32 -0
- data/TODO.impl/102-lid-chain.md +28 -0
- data/TODO.impl/103-oov-bucket-tables.md +27 -0
- data/TODO.impl/104-vscode-marketplace-readiness.md +32 -0
- data/TODO.impl/105-personal-dictionary-in-check.md +34 -0
- data/TODO.impl/106-native-lid-detection.md +36 -0
- data/TODO.impl/107-basic-support-every-staged-language.md +42 -0
- data/TODO.impl/108-ko-ne-full-feature.md +37 -0
- data/TODO.impl/109-ten-audit.md +26 -0
- data/TODO.impl/11-release-v1.md +95 -0
- data/TODO.impl/110-sr-latn-and-polish.md +33 -0
- data/TODO.impl/111-embedding-bakeoff.md +33 -0
- data/TODO.impl/112-deployability-and-gating.md +26 -0
- data/TODO.impl/113-language-packs.md +26 -0
- data/TODO.impl/114-typo-biencoder-thread.md +33 -0
- data/TODO.impl/115-hybrid-retrieval-pricing.md +22 -0
- data/TODO.impl/116-baseline-suggestion-skip.md +81 -0
- data/TODO.impl/117-deterministic-frequency-tiers.md +110 -0
- data/TODO.impl/118-server-gem-floor.md +33 -0
- data/TODO.impl/119-embedded-kelly-tiers.md +56 -0
- data/TODO.impl/120-setup-metadata-refresh.md +35 -0
- data/TODO.impl/121-jekyll-baseline-skip.md +33 -0
- data/TODO.impl/122-pack-compression-kpk2.md +66 -0
- data/TODO.impl/123-typo-corpus-synthesis.md +55 -0
- data/TODO.impl/124-owner-residue.md +19 -0
- data/TODO.impl/125-consumer-floor-guards.md +43 -0
- data/TODO.impl/126-tier-error-budgets.md +33 -0
- data/TODO.impl/127-suggestion-feedback-loop.md +34 -0
- data/TODO.impl/128-hybrid-pricing-verdict.md +47 -0
- data/TODO.impl/129-pt-fr-corpora.md +23 -0
- data/TODO.impl/130-news-truth-0912.md +23 -0
- data/TODO.impl/131-hybrid-productization.md +42 -0
- data/TODO.impl/132-techniques-page-and-prose.md +27 -0
- data/TODO.impl/133-rust-first-distribution.md +68 -0
- data/TODO.impl/20-encapsulation-cleanup.md +47 -0
- data/TODO.impl/21-autoload-migration.md +64 -0
- data/TODO.impl/22-url-pin-single-source-of-truth.md +120 -0
- data/TODO.impl/23-fix-url-defaults.md +57 -0
- data/TODO.impl/24-cli-auto-setup-prompt.md +76 -0
- data/TODO.impl/25-soft-onnxruntime-require.md +75 -0
- data/TODO.impl/26-onnx-vocab-json-generation.md +96 -0
- data/TODO.impl/27-end-to-end-smoke-spec.md +113 -0
- data/TODO.impl/28-readme-quickstart-verification.md +68 -0
- data/TODO.impl/29-status-command.md +84 -0
- data/TODO.impl/30-language-auto-detection.md +74 -0
- data/TODO.impl/31-download-progress.md +58 -0
- data/TODO.impl/32-sarif-output-verification.md +72 -0
- data/TODO.impl/33-audit-log-rotation.md +54 -0
- data/TODO.impl/34-cache-eviction.md +59 -0
- data/TODO.impl/35-shell-completion.md +61 -0
- data/TODO.impl/36-cleanup-regressions-0.3.1.md +134 -0
- data/TODO.impl/37-hunspell-correctness-tier2.md +203 -0
- data/TODO.impl/38-onnx-semantic-gating.md +218 -0
- data/TODO.impl/39-tier3-and-beyond.md +232 -0
- data/TODO.impl/40-spec-drift-cleanup.md +134 -0
- data/TODO.impl/50-structure-aware-document-api.md +194 -0
- data/TODO.impl/51-grammar-engine-expansion.md +75 -0
- data/TODO.impl/52-multi-language-documents.md +95 -0
- data/TODO.impl/53-document-plugin-architecture.md +94 -0
- data/TODO.impl/54-cjk-support.md +117 -0
- data/TODO.impl/55-rtl-support.md +109 -0
- data/TODO.impl/56-t4-t5-t6-quality-architecture-ecosystem.md +126 -0
- data/TODO.impl/57-t5.1-step3-pure-function-strategies.md +138 -0
- data/TODO.impl/58-t5.2-configuration-as-data.md +113 -0
- data/TODO.impl/59-yard-api-documentation.md +75 -0
- data/TODO.impl/60-lsp-server.md +174 -0
- data/TODO.impl/61-editor-ecosystem.md +167 -0
- data/TODO.impl/62-framework-integrations.md +159 -0
- data/TODO.impl/63-web-playground-wasm.md +186 -0
- data/TODO.impl/64-http-api-and-sdks.md +191 -0
- data/TODO.impl/65-universal-kotoshu.md +251 -0
- data/TODO.impl/66-kotoshu-core.md +169 -0
- data/TODO.impl/67-kotoshu-rs-and-access-libraries.md +137 -0
- data/TODO.impl/68-sota-adoption.md +152 -0
- data/TODO.impl/69-sota-typo-eval.md +74 -0
- data/TODO.impl/70-sota-confidence-cascade.md +59 -0
- data/TODO.impl/71-sota-nested-model.md +63 -0
- data/TODO.impl/72-model2vec-evaluation.md +49 -0
- data/TODO.impl/73-rl-reranker.md +42 -0
- data/TODO.impl/74-distilled-reranker.md +36 -0
- data/TODO.impl/75-llm-evaluator.md +35 -0
- data/TODO.impl/76-optional-llm-backend.md +37 -0
- data/TODO.impl/77-model-coverage-expansion.md +96 -0
- data/TODO.impl/78-in-browser-playground.md +69 -0
- data/TODO.impl/79-news-and-releases.md +60 -0
- data/TODO.impl/80-ecosystem-docs.md +56 -0
- data/TODO.impl/81-editor-reach.md +68 -0
- data/TODO.impl/82-doc-formats-and-ignores.md +81 -0
- data/TODO.impl/83-models-batch-2.md +56 -0
- data/TODO.impl/84-language-modules-wave2.md +75 -0
- data/TODO.impl/85-wasm-model-api.md +48 -0
- data/TODO.impl/86-site-truth-wave2.md +45 -0
- data/TODO.impl/87-lsp-personal-dictionary.md +36 -0
- data/TODO.impl/88-cli-directory-mode.md +54 -0
- data/TODO.impl/89-framework-integrations-exec.md +59 -0
- data/TODO.impl/90-gem-correctness-residues.md +44 -0
- data/TODO.impl/91-unicode-and-sv-aff.md +47 -0
- data/TODO.impl/92-tier-mirror-cors.md +88 -0
- data/TODO.impl/93-python-wheels.md +32 -0
- data/TODO.impl/94-site-integrations-docs.md +35 -0
- data/TODO.impl/95-editor-action-polish.md +27 -0
- data/TODO.impl/96-server-semantic-models.md +50 -0
- data/TODO.impl/97-yard-execution.md +73 -0
- data/TODO.impl/98-campaign-cleanup.md +78 -0
- data/TODO.impl/99-site-truth-performance-docs.md +31 -0
- data/conformance/vectors.jsonl +2630 -0
- data/data/en/grammar/rules.yaml +851 -0
- data/docs/STABILITY-1.0-AUDIT.md +251 -0
- data/docs/STABILITY-1.0.md +137 -0
- data/docs/cache-architecture.md +502 -0
- data/docs/verification-evidence/correct3/pr_body.md +37 -0
- data/docs/verification-evidence/correct3/rspec_final.log +5588 -0
- data/docs/verification-evidence/correct3/rspec_full.log +4504 -0
- data/docs/verification-evidence/correct3/sv.aff +856 -0
- data/docs/verification-evidence/correct3/sv.dic +153717 -0
- data/examples/01_basic_word_checking.rb +38 -0
- data/examples/02_text_document_checking.rb +77 -0
- data/examples/03_dictionary_backends.rb +137 -0
- data/examples/04_trie_data_structure.rb +146 -0
- data/examples/05_suggestion_algorithms.rb +239 -0
- data/examples/06_configuration_advanced.rb +287 -0
- data/examples/07_multi_language_dictionaries.rb +278 -0
- data/exe/kotoshu +6 -0
- data/lib/kotoshu/algorithms/capitalization.rb +276 -0
- data/lib/kotoshu/algorithms/edit_distance.rb +128 -0
- data/lib/kotoshu/algorithms/lookup/affix_form.rb +132 -0
- data/lib/kotoshu/algorithms/lookup/compound_checks.rb +209 -0
- data/lib/kotoshu/algorithms/lookup/compound_form.rb +50 -0
- data/lib/kotoshu/algorithms/lookup.rb +841 -0
- data/lib/kotoshu/algorithms/ngram_suggest.rb +367 -0
- data/lib/kotoshu/algorithms/permutations.rb +283 -0
- data/lib/kotoshu/algorithms/phonet_suggest.rb +162 -0
- data/lib/kotoshu/algorithms/soundex.rb +92 -0
- data/lib/kotoshu/algorithms/suggest.rb +794 -0
- data/lib/kotoshu/algorithms.rb +23 -0
- data/lib/kotoshu/analyzers/semantic_analyzer.rb +405 -0
- data/lib/kotoshu/analyzers.rb +8 -0
- data/lib/kotoshu/baseline/application.rb +18 -0
- data/lib/kotoshu/baseline/entry.rb +33 -0
- data/lib/kotoshu/baseline/store.rb +175 -0
- data/lib/kotoshu/baseline.rb +20 -0
- data/lib/kotoshu/cache/base_cache.rb +778 -0
- data/lib/kotoshu/cache/cache.rb +91 -0
- data/lib/kotoshu/cache/eviction_policy.rb +66 -0
- data/lib/kotoshu/cache/frequency_cache.rb +283 -0
- data/lib/kotoshu/cache/language_cache.rb +511 -0
- data/lib/kotoshu/cache/lookup_cache.rb +166 -0
- data/lib/kotoshu/cache/model_cache.rb +1147 -0
- data/lib/kotoshu/cache/model_registry.rb +109 -0
- data/lib/kotoshu/cache/staged_languages.rb +45 -0
- data/lib/kotoshu/cache/suggestion_cache.rb +113 -0
- data/lib/kotoshu/cache.rb +49 -0
- data/lib/kotoshu/cli/auto_setup.rb +71 -0
- data/lib/kotoshu/cli/baseline_command.rb +119 -0
- data/lib/kotoshu/cli/batch_reporter.rb +321 -0
- data/lib/kotoshu/cli/cache_command.rb +282 -0
- data/lib/kotoshu/cli/completions_command.rb +210 -0
- data/lib/kotoshu/cli/dict_command.rb +64 -0
- data/lib/kotoshu/cli/directory_check.rb +146 -0
- data/lib/kotoshu/cli/directory_walker.rb +132 -0
- data/lib/kotoshu/cli/display_formatter.rb +429 -0
- data/lib/kotoshu/cli/errors.rb +36 -0
- data/lib/kotoshu/cli/ignore_matcher.rb +205 -0
- data/lib/kotoshu/cli/interactive_reviewer.rb +314 -0
- data/lib/kotoshu/cli/language_resolver.rb +91 -0
- data/lib/kotoshu/cli/navigation_manager.rb +273 -0
- data/lib/kotoshu/cli/personal_command.rb +113 -0
- data/lib/kotoshu/cli/progress_reporter.rb +114 -0
- data/lib/kotoshu/cli/status_report.rb +130 -0
- data/lib/kotoshu/cli.rb +880 -0
- data/lib/kotoshu/components/passthrough_spell_checker.rb +70 -0
- data/lib/kotoshu/components/pos_tagger.rb +98 -0
- data/lib/kotoshu/components/spell_checker.rb +73 -0
- data/lib/kotoshu/components/synthesizer.rb +60 -0
- data/lib/kotoshu/components/tokenizer.rb +58 -0
- data/lib/kotoshu/components/whitespace_tokenizer.rb +94 -0
- data/lib/kotoshu/components.rb +13 -0
- data/lib/kotoshu/configuration/builder.rb +207 -0
- data/lib/kotoshu/configuration/resolver.rb +124 -0
- data/lib/kotoshu/configuration.rb +829 -0
- data/lib/kotoshu/conformance_exporter.rb +256 -0
- data/lib/kotoshu/conformance_runner.rb +176 -0
- data/lib/kotoshu/core/exceptions.rb +186 -0
- data/lib/kotoshu/core/indexed_dictionary.rb +293 -0
- data/lib/kotoshu/core/models/affix_rule.rb +262 -0
- data/lib/kotoshu/core/models/result/document_result.rb +132 -0
- data/lib/kotoshu/core/models/result/word_result.rb +131 -0
- data/lib/kotoshu/core/models/word.rb +131 -0
- data/lib/kotoshu/core/trie/builder.rb +119 -0
- data/lib/kotoshu/core/trie/node.rb +94 -0
- data/lib/kotoshu/core/trie/trie.rb +249 -0
- data/lib/kotoshu/core.rb +28 -0
- data/lib/kotoshu/data/common_words/de.yml +1800 -0
- data/lib/kotoshu/data/common_words/en.yml +1215 -0
- data/lib/kotoshu/data/common_words/es.yml +750 -0
- data/lib/kotoshu/data/common_words/fr.yml +1015 -0
- data/lib/kotoshu/data/common_words/pt.yml +870 -0
- data/lib/kotoshu/data/common_words/ru.yml +484 -0
- data/lib/kotoshu/data/common_words_loader.rb +141 -0
- data/lib/kotoshu/data/frozen_kelly/en.json +1154 -0
- data/lib/kotoshu/data/frozen_kelly.rb +38 -0
- data/lib/kotoshu/data.rb +8 -0
- data/lib/kotoshu/data_structures/bloom_filter.rb +188 -0
- data/lib/kotoshu/data_structures.rb +8 -0
- data/lib/kotoshu/debug_logger.rb +146 -0
- data/lib/kotoshu/debug_mode.rb +136 -0
- data/lib/kotoshu/defaults.rb +84 -0
- data/lib/kotoshu/dictionaries/catalog.rb +814 -0
- data/lib/kotoshu/dictionaries.rb +8 -0
- data/lib/kotoshu/dictionary/base.rb +324 -0
- data/lib/kotoshu/dictionary/cspell.rb +243 -0
- data/lib/kotoshu/dictionary/custom.rb +225 -0
- data/lib/kotoshu/dictionary/hunspell.rb +520 -0
- data/lib/kotoshu/dictionary/plain_text.rb +302 -0
- data/lib/kotoshu/dictionary/repository.rb +246 -0
- data/lib/kotoshu/dictionary/unified.rb +236 -0
- data/lib/kotoshu/dictionary/unix_words.rb +218 -0
- data/lib/kotoshu/dictionary.rb +15 -0
- data/lib/kotoshu/documents/document.rb +120 -0
- data/lib/kotoshu/documents/plain_text_document.rb +73 -0
- data/lib/kotoshu/documents/source_position.rb +45 -0
- data/lib/kotoshu/documents/source_range.rb +76 -0
- data/lib/kotoshu/documents/suppressions/scanner.rb +149 -0
- data/lib/kotoshu/documents/suppressions/suppression.rb +75 -0
- data/lib/kotoshu/documents/suppressions.rb +44 -0
- data/lib/kotoshu/documents/text_node.rb +49 -0
- data/lib/kotoshu/documents.rb +178 -0
- data/lib/kotoshu/embeddings/embedding_pipeline.rb +238 -0
- data/lib/kotoshu/embeddings/lru_cache.rb +244 -0
- data/lib/kotoshu/embeddings/onnx_runtime_model.rb +411 -0
- data/lib/kotoshu/embeddings/protocol.rb +96 -0
- data/lib/kotoshu/embeddings/protocols.rb +23 -0
- data/lib/kotoshu/embeddings/registry.rb +184 -0
- data/lib/kotoshu/embeddings/search.rb +204 -0
- data/lib/kotoshu/embeddings/similarity_engine.rb +278 -0
- data/lib/kotoshu/embeddings/similarity_search.rb +328 -0
- data/lib/kotoshu/embeddings/vocabulary.rb +306 -0
- data/lib/kotoshu/embeddings.rb +87 -0
- data/lib/kotoshu/fluent_checker.rb +94 -0
- data/lib/kotoshu/grammar/pattern_matchers/base_matcher.rb +48 -0
- data/lib/kotoshu/grammar/pattern_matchers/double_negative_matcher.rb +179 -0
- data/lib/kotoshu/grammar/pattern_matchers/phrase_matcher.rb +86 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_context_matcher.rb +75 -0
- data/lib/kotoshu/grammar/pattern_matchers/possessive_contraction_matcher.rb +87 -0
- data/lib/kotoshu/grammar/pattern_matchers/sentence_start_matcher.rb +84 -0
- data/lib/kotoshu/grammar/pattern_matchers/vowel_sound_matcher.rb +81 -0
- data/lib/kotoshu/grammar/pattern_matchers/word_list_matcher.rb +66 -0
- data/lib/kotoshu/grammar/rule.rb +99 -0
- data/lib/kotoshu/grammar/rule_engine.rb +96 -0
- data/lib/kotoshu/grammar/rule_loader.rb +30 -0
- data/lib/kotoshu/grammar.rb +24 -0
- data/lib/kotoshu/integrity/audit_log.rb +167 -0
- data/lib/kotoshu/integrity/manifest.rb +116 -0
- data/lib/kotoshu/integrity/net_http.rb +51 -0
- data/lib/kotoshu/integrity/rotation_policy.rb +88 -0
- data/lib/kotoshu/integrity.rb +26 -0
- data/lib/kotoshu/jekyll.rb +101 -0
- data/lib/kotoshu/keyboard/layout.rb +116 -0
- data/lib/kotoshu/keyboard/layouts/arabic.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/azerty.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/bulgarian.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/devanagari_inscript.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/dubeolsik.rb +56 -0
- data/lib/kotoshu/keyboard/layouts/dvorak.rb +54 -0
- data/lib/kotoshu/keyboard/layouts/greek.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/hebrew.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/hrsl.rb +94 -0
- data/lib/kotoshu/keyboard/layouts/jcuken.rb +57 -0
- data/lib/kotoshu/keyboard/layouts/latin.rb +266 -0
- data/lib/kotoshu/keyboard/layouts/persian.rb +58 -0
- data/lib/kotoshu/keyboard/layouts/qwerty.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/qwertz.rb +55 -0
- data/lib/kotoshu/keyboard/layouts/serbian.rb +59 -0
- data/lib/kotoshu/keyboard/layouts/turkish_q.rb +52 -0
- data/lib/kotoshu/keyboard/layouts/ukrainian.rb +51 -0
- data/lib/kotoshu/keyboard/registry.rb +201 -0
- data/lib/kotoshu/keyboard.rb +96 -0
- data/lib/kotoshu/kotoshu_native.so +0 -0
- data/lib/kotoshu/language/detector.rb +242 -0
- data/lib/kotoshu/language/identifier.rb +376 -0
- data/lib/kotoshu/language/languages/base.rb +271 -0
- data/lib/kotoshu/language/lid_detector.rb +188 -0
- data/lib/kotoshu/language/normalizer/arabic.rb +113 -0
- data/lib/kotoshu/language/normalizer/base.rb +133 -0
- data/lib/kotoshu/language/normalizer/greek.rb +63 -0
- data/lib/kotoshu/language/normalizer/hebrew.rb +72 -0
- data/lib/kotoshu/language/normalizer/persian.rb +36 -0
- data/lib/kotoshu/language/normalizer/turkish.rb +38 -0
- data/lib/kotoshu/language/registry.rb +209 -0
- data/lib/kotoshu/language/resources/ar/common_words.txt +6753 -0
- data/lib/kotoshu/language/resources/ar/confusion_sets.txt +11 -0
- data/lib/kotoshu/language/resources/de/common_words.txt +10003 -0
- data/lib/kotoshu/language/resources/de/confusion_sets.txt +246 -0
- data/lib/kotoshu/language/resources/en/common_words.txt +9979 -0
- data/lib/kotoshu/language/resources/en/confusion_sets.txt +871 -0
- data/lib/kotoshu/language/resources/es/common_words.txt +9992 -0
- data/lib/kotoshu/language/resources/es/confusion_sets.txt +17 -0
- data/lib/kotoshu/language/resources/fr/common_words.txt +9993 -0
- data/lib/kotoshu/language/resources/fr/confusion_sets.txt +76 -0
- data/lib/kotoshu/language/resources/pt/common_words.txt +9977 -0
- data/lib/kotoshu/language/resources/pt/confusion_sets.txt +18 -0
- data/lib/kotoshu/language/resources/ru/common_words.txt +9951 -0
- data/lib/kotoshu/language/resources/ru/confusion_sets.txt +5 -0
- data/lib/kotoshu/language/script.rb +128 -0
- data/lib/kotoshu/language/segmenter.rb +122 -0
- data/lib/kotoshu/language/suika.rb +38 -0
- data/lib/kotoshu/language/tokenizer/base.rb +187 -0
- data/lib/kotoshu/language/tokenizer/cyrillic_tokenizer.rb +76 -0
- data/lib/kotoshu/language/tokenizer/devanagari_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/french_tokenizer.rb +172 -0
- data/lib/kotoshu/language/tokenizer/german_tokenizer.rb +41 -0
- data/lib/kotoshu/language/tokenizer/greek_tokenizer.rb +72 -0
- data/lib/kotoshu/language/tokenizer/hangul_tokenizer.rb +70 -0
- data/lib/kotoshu/language/tokenizer/japanese_tokenizer.rb +55 -0
- data/lib/kotoshu/language/tokenizer/latin_tokenizer.rb +153 -0
- data/lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb +160 -0
- data/lib/kotoshu/language/tokenizer/russian_tokenizer.rb +93 -0
- data/lib/kotoshu/language/tokenizer/script_tokenizer.rb +62 -0
- data/lib/kotoshu/language/tokenizer/spanish_tokenizer.rb +122 -0
- data/lib/kotoshu/language.rb +127 -0
- data/lib/kotoshu/languages/ar/language.rb +98 -0
- data/lib/kotoshu/languages/bg/language.rb +37 -0
- data/lib/kotoshu/languages/ca/language.rb +22 -0
- data/lib/kotoshu/languages/cs/language.rb +22 -0
- data/lib/kotoshu/languages/da/language.rb +22 -0
- data/lib/kotoshu/languages/de/language.rb +554 -0
- data/lib/kotoshu/languages/el/language.rb +50 -0
- data/lib/kotoshu/languages/en/language.rb +470 -0
- data/lib/kotoshu/languages/es/language.rb +476 -0
- data/lib/kotoshu/languages/et/language.rb +24 -0
- data/lib/kotoshu/languages/fa/language.rb +91 -0
- data/lib/kotoshu/languages/fr/language.rb +505 -0
- data/lib/kotoshu/languages/he/language.rb +98 -0
- data/lib/kotoshu/languages/hr/language.rb +24 -0
- data/lib/kotoshu/languages/hu/language.rb +22 -0
- data/lib/kotoshu/languages/id/language.rb +25 -0
- data/lib/kotoshu/languages/it/language.rb +22 -0
- data/lib/kotoshu/languages/ja/language.rb +476 -0
- data/lib/kotoshu/languages/ko/language.rb +56 -0
- data/lib/kotoshu/languages/latin_base.rb +53 -0
- data/lib/kotoshu/languages/lt/language.rb +24 -0
- data/lib/kotoshu/languages/lv/language.rb +24 -0
- data/lib/kotoshu/languages/nb/language.rb +25 -0
- data/lib/kotoshu/languages/ne/language.rb +60 -0
- data/lib/kotoshu/languages/nl/language.rb +22 -0
- data/lib/kotoshu/languages/nn/language.rb +26 -0
- data/lib/kotoshu/languages/pl/language.rb +22 -0
- data/lib/kotoshu/languages/pt/language.rb +439 -0
- data/lib/kotoshu/languages/ro/language.rb +22 -0
- data/lib/kotoshu/languages/ru/language.rb +420 -0
- data/lib/kotoshu/languages/sk/language.rb +25 -0
- data/lib/kotoshu/languages/sl/language.rb +25 -0
- data/lib/kotoshu/languages/sr/language.rb +39 -0
- data/lib/kotoshu/languages/sr-Latn/language.rb +35 -0
- data/lib/kotoshu/languages/sv/language.rb +22 -0
- data/lib/kotoshu/languages/tr/language.rb +36 -0
- data/lib/kotoshu/languages/uk/language.rb +38 -0
- data/lib/kotoshu/languages/vi/language.rb +22 -0
- data/lib/kotoshu/languages.rb +64 -0
- data/lib/kotoshu/metrics_collector.rb +238 -0
- data/lib/kotoshu/metrics_module.rb +121 -0
- data/lib/kotoshu/models/context.rb +119 -0
- data/lib/kotoshu/models/embedding_model.rb +181 -0
- data/lib/kotoshu/models/fasttext_model.rb +212 -0
- data/lib/kotoshu/models/nearest_neighbor.rb +96 -0
- data/lib/kotoshu/models/onnx_model.rb +330 -0
- data/lib/kotoshu/models/semantic_error.rb +204 -0
- data/lib/kotoshu/models/suggestion.rb +106 -0
- data/lib/kotoshu/models/word_embedding.rb +97 -0
- data/lib/kotoshu/models.rb +27 -0
- data/lib/kotoshu/multi_language_checker.rb +127 -0
- data/lib/kotoshu/native.rb +83 -0
- data/lib/kotoshu/native_backend.rb +131 -0
- data/lib/kotoshu/paths.rb +79 -0
- data/lib/kotoshu/personal_dictionary.rb +105 -0
- data/lib/kotoshu/plugins/plugin.rb +61 -0
- data/lib/kotoshu/plugins/registry.rb +118 -0
- data/lib/kotoshu/plugins.rb +9 -0
- data/lib/kotoshu/project_config.rb +76 -0
- data/lib/kotoshu/readers/aff_data.rb +530 -0
- data/lib/kotoshu/readers/aff_reader.rb +393 -0
- data/lib/kotoshu/readers/condition_checker.rb +125 -0
- data/lib/kotoshu/readers/dic_reader.rb +208 -0
- data/lib/kotoshu/readers/file_reader.rb +377 -0
- data/lib/kotoshu/readers/lookup_builder.rb +584 -0
- data/lib/kotoshu/readers/readers.rb +17 -0
- data/lib/kotoshu/readers.rb +25 -0
- data/lib/kotoshu/resource_bundle.rb +55 -0
- data/lib/kotoshu/resource_manager.rb +600 -0
- data/lib/kotoshu/results/result.rb +165 -0
- data/lib/kotoshu/results.rb +8 -0
- data/lib/kotoshu/rspec.rb +163 -0
- data/lib/kotoshu/scripts/fasttext_to_onnx.py +275 -0
- data/lib/kotoshu/source_registry.rb +104 -0
- data/lib/kotoshu/spellchecker/parallel_checker.rb +90 -0
- data/lib/kotoshu/spellchecker.rb +406 -0
- data/lib/kotoshu/string_metrics.rb +153 -0
- data/lib/kotoshu/suggestions/context.rb +43 -0
- data/lib/kotoshu/suggestions/frequency_provider.rb +80 -0
- data/lib/kotoshu/suggestions/generator.rb +169 -0
- data/lib/kotoshu/suggestions/pipeline.rb +135 -0
- data/lib/kotoshu/suggestions/semantic_cascade.rb +73 -0
- data/lib/kotoshu/suggestions/strategies/base_strategy.rb +315 -0
- data/lib/kotoshu/suggestions/strategies/composite_strategy.rb +197 -0
- data/lib/kotoshu/suggestions/strategies/edit_distance_strategy.rb +631 -0
- data/lib/kotoshu/suggestions/strategies/keyboard_proximity_strategy.rb +207 -0
- data/lib/kotoshu/suggestions/strategies/ngram_strategy.rb +165 -0
- data/lib/kotoshu/suggestions/strategies/phonetic_strategy.rb +284 -0
- data/lib/kotoshu/suggestions/strategies/semantic_strategy.rb +317 -0
- data/lib/kotoshu/suggestions/strategies/symspell_strategy.rb +270 -0
- data/lib/kotoshu/suggestions/strategies/word_index.rb +78 -0
- data/lib/kotoshu/suggestions/suggestion.rb +154 -0
- data/lib/kotoshu/suggestions/suggestion_set.rb +270 -0
- data/lib/kotoshu/suggestions/sweep_index.rb +137 -0
- data/lib/kotoshu/suggestions.rb +29 -0
- data/lib/kotoshu/tasks/check_task.rb +104 -0
- data/lib/kotoshu/tasks.rb +11 -0
- data/lib/kotoshu/validators/spelling_validator.rb +101 -0
- data/lib/kotoshu/validators.rb +9 -0
- data/lib/kotoshu/version.rb +5 -0
- data/lib/kotoshu.rb +538 -0
- data/rakelib/compile.rake +10 -0
- data/script/validate_all_dictionaries.rb +444 -0
- data/scripts/build_native_gem.rb +45 -0
- data/scripts/extract_eval_grids.rb +37 -0
- data/scripts/generate_frozen_tiers.rb +44 -0
- data/sig/kotoshu.rbs +435 -0
- data/test_oop.rb +79 -0
- metadata +546 -0
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
# 52 — Multi-language document checking (T3.5)
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Right now `Kotoshu.check_file` and `Kotoshu::Spellchecker` are
|
|
6
|
+
single-language. Real documents mix languages: code comments in
|
|
7
|
+
English embedded in French prose, a Japanese paper with English
|
|
8
|
+
abstract, etc. This TODO adds per-paragraph language detection and
|
|
9
|
+
per-language resolution so a single document is checked with the
|
|
10
|
+
right dictionary + rules for each segment.
|
|
11
|
+
|
|
12
|
+
## Design
|
|
13
|
+
|
|
14
|
+
### Layer 1: Per-segment detection
|
|
15
|
+
|
|
16
|
+
`Kotoshu::Language::Identifier` (FastText LID) already exists. Wire
|
|
17
|
+
it into a `DocumentSegmenter` that splits a `Kotoshu::Documents::Document`
|
|
18
|
+
into language-tagged segments.
|
|
19
|
+
|
|
20
|
+
class Kotoshu::Language::Segmenter
|
|
21
|
+
# @param document [Documents::Document]
|
|
22
|
+
# @param threshold [Float] detection confidence threshold (default 0.7)
|
|
23
|
+
# @return [Array<Segment>] language-tagged segments
|
|
24
|
+
def segment(document, threshold: 0.7)
|
|
25
|
+
# Walk document.text_nodes.
|
|
26
|
+
# Group consecutive nodes by detected language.
|
|
27
|
+
# Detect per node (or per paragraph break).
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
class Kotoshu::Language::Segment
|
|
32
|
+
# @param language_code [String]
|
|
33
|
+
# @param text_nodes [Array<Documents::TextNode>]
|
|
34
|
+
# @param confidence [Float]
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
### Layer 2: Per-segment resource resolution
|
|
38
|
+
|
|
39
|
+
For each segment, resolve the right dictionary + rules via the
|
|
40
|
+
existing two-stage `ResourceManager`:
|
|
41
|
+
|
|
42
|
+
Kotoshu::ResourceManager.resolve(language: segment.language_code, want: %i[spelling grammar])
|
|
43
|
+
|
|
44
|
+
### Layer 3: Multi-language check pipeline
|
|
45
|
+
|
|
46
|
+
class Kotoshu::MultiLanguageChecker
|
|
47
|
+
def check(document)
|
|
48
|
+
segments = segmenter.segment(document)
|
|
49
|
+
segments.flat_map do |segment|
|
|
50
|
+
bundle = ResourceManager.resolve(language: segment.language_code)
|
|
51
|
+
spellchecker = Spellchecker.new(resource_bundle: bundle)
|
|
52
|
+
segment.text_nodes.flat_map { |node|
|
|
53
|
+
spellchecker.check_text(node.text)
|
|
54
|
+
}
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
### Layer 4: Result aggregation
|
|
60
|
+
|
|
61
|
+
Errors carry `source_range` from TODO 50. Per-segment errors already
|
|
62
|
+
have positions; aggregation is just a concatenation sorted by
|
|
63
|
+
source_range.start.
|
|
64
|
+
|
|
65
|
+
## Phases
|
|
66
|
+
|
|
67
|
+
### Phase 1 — Segmenter (this PR series)
|
|
68
|
+
- `Language::Segmenter` with paragraph-level detection.
|
|
69
|
+
- Specs with mixed-language fixtures (en + de, en + ja).
|
|
70
|
+
|
|
71
|
+
### Phase 2 — MultiLanguageChecker (this PR series)
|
|
72
|
+
- Composes segmenter + spellchecker.
|
|
73
|
+
- Carries `source_range` through.
|
|
74
|
+
|
|
75
|
+
### Phase 3 — CLI integration
|
|
76
|
+
- `kotoshu check --language auto FILE` picks the multi-language path.
|
|
77
|
+
- Default for `auto` becomes multi-language; single-language mode is
|
|
78
|
+
opt-in via `--language en`.
|
|
79
|
+
|
|
80
|
+
## Acceptance criteria
|
|
81
|
+
|
|
82
|
+
- [ ] A document with English prose and French quotes is checked with
|
|
83
|
+
both dictionaries, errors from both languages surface in the
|
|
84
|
+
output with correct source positions.
|
|
85
|
+
- [ ] A document with English code comments and Japanese prose is
|
|
86
|
+
checked with the right language per segment.
|
|
87
|
+
- [ ] Performance: < 2x the single-language cost for typical mixed
|
|
88
|
+
documents.
|
|
89
|
+
- [ ] Full suite stays green.
|
|
90
|
+
|
|
91
|
+
## Dependencies
|
|
92
|
+
|
|
93
|
+
- **Blocked by:** TODO 50 (document API with source positions).
|
|
94
|
+
- **Blocks:** T3.4 document plugin architecture (plugins want
|
|
95
|
+
multi-language support to come for free).
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# 53 — Document plugin architecture (T3.4)
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Formalize the boundary between kotoshu-core and format-specific
|
|
6
|
+
document parsers. The boundary is already decided (see
|
|
7
|
+
`kotoshu-document-plugin-boundary` memory): kotoshu never owns
|
|
8
|
+
document parsing. Plugins like `coradoc-plugin-kotoshu` provide
|
|
9
|
+
parsers that produce `Kotoshu::Documents::Document` instances.
|
|
10
|
+
|
|
11
|
+
This TODO formalizes the plugin contract, the registration mechanism,
|
|
12
|
+
and the audit that moves the existing built-in Markdown parser out of
|
|
13
|
+
core kotoshu (if/when a `markdown-plugin-kotoshu` exists).
|
|
14
|
+
|
|
15
|
+
## Design
|
|
16
|
+
|
|
17
|
+
### Plugin contract
|
|
18
|
+
|
|
19
|
+
A document plugin is a Ruby gem that:
|
|
20
|
+
|
|
21
|
+
1. Depends on `kotoshu`.
|
|
22
|
+
2. On load, calls `Kotoshu::Documents.register(format, ParserClass)`.
|
|
23
|
+
3. The parser class responds to:
|
|
24
|
+
- `.from_string(text, language_code:)` → `Kotoshu::Documents::Document`
|
|
25
|
+
- `.from_file(path, language_code:)` → `Kotoshu::Documents::Document` (optional)
|
|
26
|
+
4. The Document's `text_nodes` carry proper `SourceRange`s pointing
|
|
27
|
+
into the original markup-bearing source.
|
|
28
|
+
|
|
29
|
+
### Discovery
|
|
30
|
+
|
|
31
|
+
Two discovery mechanisms:
|
|
32
|
+
|
|
33
|
+
1. **Explicit**: user code calls `Kotoshu::Documents.register` after
|
|
34
|
+
requiring the plugin.
|
|
35
|
+
2. **Gem find-files**: a plugin ships a `kotoshu_plugin/document/*.rb`
|
|
36
|
+
file. Kotoshu loads every such file at startup via
|
|
37
|
+
`Gem.find_files("kotoshu_plugin/document/*.rb")`.
|
|
38
|
+
|
|
39
|
+
Both are supported. Explicit is the default; gem find-files is the
|
|
40
|
+
"just install the gem and it works" path.
|
|
41
|
+
|
|
42
|
+
### Built-in parsers
|
|
43
|
+
|
|
44
|
+
- **PlainTextDocument**: built-in, never moves. Plain text needs no
|
|
45
|
+
parser.
|
|
46
|
+
- **MarkdownDocument**: currently lives in `lib/kotoshu/documents/`
|
|
47
|
+
(was removed in TODO 50; verify and re-add as plugin if needed).
|
|
48
|
+
- **AsciiDocDocument**: same — was removed; should come from
|
|
49
|
+
`coradoc-plugin-kotoshu`.
|
|
50
|
+
|
|
51
|
+
### Audit checklist
|
|
52
|
+
|
|
53
|
+
For each currently-shipped document parser, decide:
|
|
54
|
+
|
|
55
|
+
1. **Keep in core** (e.g. PlainTextDocument) — generic enough.
|
|
56
|
+
2. **Move to a plugin** (e.g. MarkdownDocument) — needs format-specific
|
|
57
|
+
parsing logic that violates the boundary.
|
|
58
|
+
|
|
59
|
+
## Phases
|
|
60
|
+
|
|
61
|
+
### Phase 1 — Plugin API (DONE in TODO 50)
|
|
62
|
+
- `Kotoshu::Documents.register`, `parser_for`, `parse`,
|
|
63
|
+
`registered_formats`, `reset!`.
|
|
64
|
+
|
|
65
|
+
### Phase 2 — Gem find-files discovery
|
|
66
|
+
- On kotoshu load, walk `Gem.find_files("kotoshu_plugin/document/*.rb")`.
|
|
67
|
+
- Each file is responsible for calling `Documents.register`.
|
|
68
|
+
- Add `Kotoshu::Documents.discovered_plugins` reader.
|
|
69
|
+
|
|
70
|
+
### Phase 3 — Audit existing parsers
|
|
71
|
+
- Confirm `lib/kotoshu/documents/` only contains core abstractions
|
|
72
|
+
(Location, TextNode, Document, PlainTextDocument).
|
|
73
|
+
- If a Markdown or AsciiDoc parser exists in core, document it as a
|
|
74
|
+
temporary exception OR migrate.
|
|
75
|
+
|
|
76
|
+
### Phase 4 — Plugin authoring docs
|
|
77
|
+
- README section "Writing a document plugin".
|
|
78
|
+
- Example plugin skeleton.
|
|
79
|
+
- YARD on the contract.
|
|
80
|
+
|
|
81
|
+
## Acceptance criteria
|
|
82
|
+
|
|
83
|
+
- [ ] A plugin installed as a gem is auto-discovered without user code.
|
|
84
|
+
- [ ] `Kotoshu::Documents.discovered_plugins` lists discovered plugins.
|
|
85
|
+
- [ ] No format-specific parsing code in `lib/kotoshu/documents/`
|
|
86
|
+
beyond PlainTextDocument.
|
|
87
|
+
- [ ] README has a "Writing a document plugin" section.
|
|
88
|
+
- [ ] Full suite stays green.
|
|
89
|
+
|
|
90
|
+
## Dependencies
|
|
91
|
+
|
|
92
|
+
- **Blocked by:** TODO 50 (document API).
|
|
93
|
+
- **Blocks:** coradoc-plugin-kotoshu's structure-aware checking
|
|
94
|
+
(that plugin is the canonical consumer).
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# 54 — CJK support (T3.1)
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Real CJK (Chinese, Japanese, Korean) spell and grammar checking.
|
|
6
|
+
This is the largest tier 3 item: CJK languages need word-segmentation
|
|
7
|
+
(no spaces in zh/ja), different dictionary formats, and
|
|
8
|
+
romanization-aware suggestion strategies.
|
|
9
|
+
|
|
10
|
+
Status: outline only. This TODO documents the design space; real work
|
|
11
|
+
needs upstream library choices and per-language fixture data.
|
|
12
|
+
|
|
13
|
+
## Per-language needs
|
|
14
|
+
|
|
15
|
+
### Chinese (zh)
|
|
16
|
+
|
|
17
|
+
- **Tokenizer**: needs word segmentation. Options:
|
|
18
|
+
- `jieba-ruby` (port of the Python jieba) — HMM + dictionary-based.
|
|
19
|
+
- `suika` (already a kotoshu dependency for Japanese) has some CJK
|
|
20
|
+
support — verify coverage.
|
|
21
|
+
- **Dictionary format**: frequency-sorted word lists, not Hunspell.
|
|
22
|
+
CJK dictionaries don't have affixes.
|
|
23
|
+
- **Suggestions**: edit distance on Unicode codepoints is wrong (zh
|
|
24
|
+
characters are logographic). Need pinyin-aware edit distance:
|
|
25
|
+
- Map each character to its pinyin romanization.
|
|
26
|
+
- Edit-distance on pinyin, then map back to characters.
|
|
27
|
+
- Requires a `pinyin-ruby` gem or port of `pypinyin`.
|
|
28
|
+
|
|
29
|
+
### Japanese (ja)
|
|
30
|
+
|
|
31
|
+
- **Tokenizer**: `suika` already integrated. Works for kana + kanji.
|
|
32
|
+
- **Dictionary format**: frequency-sorted word lists.
|
|
33
|
+
- **Suggestions**: romaji-aware edit distance. Map kana → romaji via
|
|
34
|
+
`romaji` gem, then edit-distance on romaji.
|
|
35
|
+
|
|
36
|
+
### Korean (ko)
|
|
37
|
+
|
|
38
|
+
- **Tokenizer**: Korean HAS spaces but morphological analysis is
|
|
39
|
+
needed for accurate word boundaries. Options:
|
|
40
|
+
- `korean-stemmer` (pure Ruby port of mecab-ko).
|
|
41
|
+
- Or treat Korean like Latin-family languages (use space-split).
|
|
42
|
+
- **Dictionary format**: word lists.
|
|
43
|
+
- **Suggestions**: romanization via `romaja` or similar.
|
|
44
|
+
|
|
45
|
+
## Design space
|
|
46
|
+
|
|
47
|
+
### New tokenizer base classes
|
|
48
|
+
|
|
49
|
+
class Kotoshu::Language::Tokenizer::CJKTokenizer < Base
|
|
50
|
+
# Subclasses: ChineseTokenizer, JapaneseTokenizer, KoreanTokenizer
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
Each owns its segmentation algorithm and dictionary interaction.
|
|
54
|
+
|
|
55
|
+
### New suggestion strategy
|
|
56
|
+
|
|
57
|
+
class Kotoshu::Suggestions::Strategies::RomanizationStrategy < BaseStrategy
|
|
58
|
+
# Edit distance on romanized form, then map candidates back to
|
|
59
|
+
# the original script.
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
Registered alongside the existing edit_distance, phonetic, etc.
|
|
63
|
+
|
|
64
|
+
### New dictionary backend
|
|
65
|
+
|
|
66
|
+
class Kotoshu::Dictionary::FrequencySorted < Base
|
|
67
|
+
# CJK dictionaries don't have affixes. Just a frequency-sorted
|
|
68
|
+
# word list, optionally with POS tags.
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
## Phases
|
|
72
|
+
|
|
73
|
+
### Phase 0 — Survey (this TODO)
|
|
74
|
+
- Pick segmentation libraries (suika / jieba-ruby / etc.).
|
|
75
|
+
- Confirm license compatibility.
|
|
76
|
+
- Identify fixture sources (CC-CEDICT, JMDict, KRDict).
|
|
77
|
+
|
|
78
|
+
### Phase 1 — Japanese (smallest scope)
|
|
79
|
+
- Tokenizer already exists (suika-backed `JapaneseTokenizer`).
|
|
80
|
+
- Add frequency-sorted dictionary backend.
|
|
81
|
+
- Add romaji-aware suggestion strategy.
|
|
82
|
+
- Specs with a small JMDict extract.
|
|
83
|
+
|
|
84
|
+
### Phase 2 — Chinese
|
|
85
|
+
- Add `jieba-ruby` as a soft dependency.
|
|
86
|
+
- New `ChineseTokenizer`.
|
|
87
|
+
- Pinyin mapping.
|
|
88
|
+
- Specs with a small CC-CEDICT extract.
|
|
89
|
+
|
|
90
|
+
### Phase 3 — Korean
|
|
91
|
+
- Decision on morphological analysis (mecab-ko vs simple split).
|
|
92
|
+
- Specs.
|
|
93
|
+
|
|
94
|
+
## Acceptance criteria
|
|
95
|
+
|
|
96
|
+
- [ ] `kotoshu setup :ja` produces a working Japanese checker that
|
|
97
|
+
detects and suggests corrections for non-words.
|
|
98
|
+
- [ ] Same for Chinese and Korean.
|
|
99
|
+
- [ ] Suggestion quality on CJK typos is reasonable (top-3 contains
|
|
100
|
+
the intended word in > 70% of test cases).
|
|
101
|
+
- [ ] No CJK-specific code in core kotoshu outside `languages/ja`,
|
|
102
|
+
`languages/zh`, `languages/ko`, and the new strategy/backend.
|
|
103
|
+
|
|
104
|
+
## Dependencies
|
|
105
|
+
|
|
106
|
+
- **Blocked by:** none directly, but Phases 1-3 each need their
|
|
107
|
+
upstream library chosen and bundled as soft deps.
|
|
108
|
+
- **Blocks:** nothing.
|
|
109
|
+
|
|
110
|
+
## Open questions
|
|
111
|
+
|
|
112
|
+
- Should we ship CJK dictionaries in `data/{lang}/` or only via
|
|
113
|
+
dynamic download? Current direction: dynamic download (matches
|
|
114
|
+
Latin languages).
|
|
115
|
+
- Is suika's Korean support real? Needs verification.
|
|
116
|
+
- For Chinese, jieba vs jieba-ruby vs python-subprocess? Performance
|
|
117
|
+
and packaging tradeoffs.
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
# 55 — RTL support (T3.2)
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Right-to-left script support: Arabic, Hebrew, Persian, Urdu. Covers
|
|
6
|
+
normalization, display, and tokenizer handling.
|
|
7
|
+
|
|
8
|
+
## Per-language needs
|
|
9
|
+
|
|
10
|
+
### Arabic (ar)
|
|
11
|
+
|
|
12
|
+
- **Normalizer**: NFC + ligature handling + diacritic stripping.
|
|
13
|
+
Arabic has many presentation forms (U+FE70..U+FEFF) that should be
|
|
14
|
+
canonicalized to base forms (U+0627..U+0645).
|
|
15
|
+
- **Tokenizer**: Arabic HAS spaces but morphological prefixes
|
|
16
|
+
(ال, و, ب, ف, ل) attach to words. Need light stemming for lookup.
|
|
17
|
+
- **Suggestions**: edit distance on Unicode codepoints works (unlike
|
|
18
|
+
CJK). Phonetic similarity via Soundex-like algorithm possible.
|
|
19
|
+
|
|
20
|
+
### Hebrew (he)
|
|
21
|
+
|
|
22
|
+
- **Normalizer**: NFC + dagesh + niqqud handling. Modern Hebrew is
|
|
23
|
+
typically written without niqqud; for spell-checking, strip niqqud
|
|
24
|
+
before lookup.
|
|
25
|
+
- **Tokenizer**: Hebrew HAS spaces. No prefix issues as severe as
|
|
26
|
+
Arabic.
|
|
27
|
+
- **Suggestions**: edit distance on Hebrew codepoints.
|
|
28
|
+
|
|
29
|
+
### Persian (fa)
|
|
30
|
+
|
|
31
|
+
- Like Arabic but with Persian-specific letters (گ چ پ ژ ک ی).
|
|
32
|
+
- Normalizer should map Arabic Yeh (ي) to Persian Yeh (ی).
|
|
33
|
+
|
|
34
|
+
### Urdu (ur)
|
|
35
|
+
|
|
36
|
+
- Like Persian but written in Nastaliq style (calligraphic).
|
|
37
|
+
- Tokenization is the same as Persian; the visual rendering is a
|
|
38
|
+
display concern, not a spell-check concern.
|
|
39
|
+
|
|
40
|
+
## Design
|
|
41
|
+
|
|
42
|
+
### New normalizer base class
|
|
43
|
+
|
|
44
|
+
class Kotoshu::Language::Normalizer::Rtl < Base
|
|
45
|
+
# Subclasses: ArabicNormalizer, HebrewNormalizer, PersianNormalizer.
|
|
46
|
+
# Each overrides normalize_word(word) to do NFC + script-specific
|
|
47
|
+
# canonicalization.
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
### Tokenizer flag
|
|
51
|
+
|
|
52
|
+
`Kotoshu::Language::Base#rtl?` already returns false for Latin and
|
|
53
|
+
true for Arabic (verify). Use this flag to:
|
|
54
|
+
|
|
55
|
+
1. Pick the right normalizer.
|
|
56
|
+
2. Tell the CLI batch reporter to align output RTL.
|
|
57
|
+
3. Tell the tokenizer to handle RTL marks (U+200E, U+200F, U+202A-E).
|
|
58
|
+
|
|
59
|
+
### RTL mark handling
|
|
60
|
+
|
|
61
|
+
RTL/LTR marks are zero-width characters that affect display. The
|
|
62
|
+
tokenizer should:
|
|
63
|
+
|
|
64
|
+
1. Strip marks before tokenizing (so they don't become "characters"
|
|
65
|
+
in a word).
|
|
66
|
+
2. Preserve them in the source mapping (so the editor shows them
|
|
67
|
+
correctly).
|
|
68
|
+
|
|
69
|
+
The document API (TODO 50) already supports this naturally: marks
|
|
70
|
+
live in the source between text nodes; text nodes themselves are
|
|
71
|
+
mark-free.
|
|
72
|
+
|
|
73
|
+
## Phases
|
|
74
|
+
|
|
75
|
+
### Phase 1 — Arabic normalizer
|
|
76
|
+
- New `lib/kotoshu/language/normalizers/arabic.rb`.
|
|
77
|
+
- NFC + presentation-form canonicalization + optional diacritic strip.
|
|
78
|
+
- Specs with the existing Arabic fixtures in `data/ar/`.
|
|
79
|
+
|
|
80
|
+
### Phase 2 — Arabic tokenizer + dictionary integration
|
|
81
|
+
- Verify `languages/ar/language.rb` uses the new normalizer.
|
|
82
|
+
- Wire Arabic into the `Language::Registry` and the spell-checker
|
|
83
|
+
pipeline.
|
|
84
|
+
- Specs with a small Arabic word list.
|
|
85
|
+
|
|
86
|
+
### Phase 3 — Hebrew + Persian normalizers
|
|
87
|
+
- Same shape as Arabic; different canonicalization rules.
|
|
88
|
+
- Specs per language.
|
|
89
|
+
|
|
90
|
+
### Phase 4 — RTL-aware display in CLI
|
|
91
|
+
- `cli/display_formatter.rb` aligns error context RTL when the
|
|
92
|
+
detected language is RTL.
|
|
93
|
+
- `cli/batch_reporter.rb` aligns suggestions RTL.
|
|
94
|
+
|
|
95
|
+
## Acceptance criteria
|
|
96
|
+
|
|
97
|
+
- [ ] Arabic text is normalized identically regardless of input form
|
|
98
|
+
(presentation form vs base form).
|
|
99
|
+
- [ ] `kotoshu check ARABIC_FILE` reports errors with correct source
|
|
100
|
+
positions, RTL-aligned context.
|
|
101
|
+
- [ ] Same for Hebrew and Persian.
|
|
102
|
+
- [ ] The pre-existing `spec/integrational/fixtures/right_to_left_mark.*`
|
|
103
|
+
fixtures pass.
|
|
104
|
+
- [ ] Full suite stays green.
|
|
105
|
+
|
|
106
|
+
## Dependencies
|
|
107
|
+
|
|
108
|
+
- **Blocked by:** TODO 50 (document API for source positions).
|
|
109
|
+
- **Blocks:** nothing.
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
# 56 — T4+ Quality, Architecture, and Ecosystem backlog
|
|
2
|
+
|
|
3
|
+
This file rolls up the T4 (Quality), T5 (Architecture), and T6
|
|
4
|
+
(Ecosystem) items from TODO.impl/39-tier3-and-beyond.md into
|
|
5
|
+
promotable units with concrete acceptance criteria.
|
|
6
|
+
|
|
7
|
+
## T4 — Quality
|
|
8
|
+
|
|
9
|
+
### T4.1 — Performance pass
|
|
10
|
+
|
|
11
|
+
Status: outline. Baseline measurements needed before optimization.
|
|
12
|
+
|
|
13
|
+
- `spec/benchmark/` and `spec/performance/` exist but are skipped by
|
|
14
|
+
default (SLOW_TESTS=1).
|
|
15
|
+
- Establish baselines for:
|
|
16
|
+
- Cold-start time (time to first `correct?` call).
|
|
17
|
+
- Lookup throughput (correct? calls/sec on a 100k-word dictionary).
|
|
18
|
+
- Suggest latency at 1k / 10k / 100k dictionary sizes.
|
|
19
|
+
- Profile the hot path (`Kotoshu.correct?`); attack the biggest
|
|
20
|
+
offender. Likely candidates:
|
|
21
|
+
- `IndexedDictionary` lookup: is it O(1)?
|
|
22
|
+
- `Suggestions::Generator#generate`: is the strategy composition
|
|
23
|
+
optimal?
|
|
24
|
+
- `Trie` traversal: any obvious wins?
|
|
25
|
+
- Consider Memoization, Trie compression, Bloom filter tuning.
|
|
26
|
+
|
|
27
|
+
### T4.2 — CI matrix
|
|
28
|
+
|
|
29
|
+
- Workflow that runs `bundle exec rspec --tag ~network --tag ~onnx`
|
|
30
|
+
on every push (already done).
|
|
31
|
+
- Workflow that runs `bundle exec rubocop` on every push (already done).
|
|
32
|
+
- Nightly workflow on main that runs `NETWORK_TESTS=1` (already done).
|
|
33
|
+
- Nightly workflow on main that runs `ONNX_TESTS=1` with cached models
|
|
34
|
+
(already done).
|
|
35
|
+
- Add: weekly workflow that runs `SLOW_TESTS=1` benchmarks and
|
|
36
|
+
publishes deltas to a dashboard.
|
|
37
|
+
|
|
38
|
+
### T4.3 — Property-based testing
|
|
39
|
+
|
|
40
|
+
Status: skeleton exists in `spec/properties/`.
|
|
41
|
+
|
|
42
|
+
- Trie: insertion is order-independent; lookup is total.
|
|
43
|
+
- Suggestion generator: `suggestions.include?(word)` implies
|
|
44
|
+
`dictionary.lookup?(word)` is true (no out-of-dictionary
|
|
45
|
+
suggestions).
|
|
46
|
+
- Hunspell affix application: affixed forms round-trip through the
|
|
47
|
+
stemmer.
|
|
48
|
+
|
|
49
|
+
Expand each with at least 5 properties.
|
|
50
|
+
|
|
51
|
+
### T4.4 — Documentation pass
|
|
52
|
+
|
|
53
|
+
- README.adoc: audit against actual CLI shape; remove 0.2-era
|
|
54
|
+
examples.
|
|
55
|
+
- `docs/`: prune superseded planning docs; mark historical ones as
|
|
56
|
+
historical.
|
|
57
|
+
- YARD: every public API has a docstring; private APIs are marked
|
|
58
|
+
`@private`.
|
|
59
|
+
- Add API docs publishing to CI (gh-pages).
|
|
60
|
+
|
|
61
|
+
## T5 — Architecture
|
|
62
|
+
|
|
63
|
+
### T5.1 — MECE refactor of `suggestions/strategies/`
|
|
64
|
+
|
|
65
|
+
Current strategies overlap (e.g. `symspell` and `edit_distance` both
|
|
66
|
+
generate edit-distance-1 candidates). Refactor:
|
|
67
|
+
|
|
68
|
+
- Each strategy owns a *unique* candidate-generation mechanism.
|
|
69
|
+
- `CompositeStrategy` merges and ranks, doesn't dedupe by side
|
|
70
|
+
effect.
|
|
71
|
+
- Strategies are pure functions of `(word, dictionary) → candidates`;
|
|
72
|
+
no shared state, no global config reads.
|
|
73
|
+
|
|
74
|
+
### T5.2 — Configuration as data, not singleton
|
|
75
|
+
|
|
76
|
+
`Configuration.instance` is a process-wide singleton. Hard to test,
|
|
77
|
+
hard to scope per-document. Refactor:
|
|
78
|
+
|
|
79
|
+
- Pass a `Configuration` instance explicitly to `Spellchecker.new`.
|
|
80
|
+
- Default the facade (`Kotoshu.correct?`) to a process-default
|
|
81
|
+
instance.
|
|
82
|
+
- Drop the singleton.
|
|
83
|
+
|
|
84
|
+
### T5.3 — Result model consistency
|
|
85
|
+
|
|
86
|
+
`WordResult` / `DocumentResult` / `SuggestionSet` migrated to
|
|
87
|
+
`lutaml-model` in 0.3.0. Audit:
|
|
88
|
+
|
|
89
|
+
- All three have consistent `to_hash` / `from_hash` shapes.
|
|
90
|
+
- No hand-rolled `to_h` anywhere (per global rule).
|
|
91
|
+
- CLI consumers use the framework-supplied serialization.
|
|
92
|
+
|
|
93
|
+
### T5.4 — Plugin architecture formalization
|
|
94
|
+
|
|
95
|
+
See TODO.impl/53.
|
|
96
|
+
|
|
97
|
+
## T6 — Ecosystem
|
|
98
|
+
|
|
99
|
+
### T6.1 — VSCode / LSP integration
|
|
100
|
+
|
|
101
|
+
A language server that runs kotoshu on save, reports diagnostics via
|
|
102
|
+
LSP. Probably a separate gem (`kotoshu-lsp`).
|
|
103
|
+
|
|
104
|
+
### T6.2 — Web playground
|
|
105
|
+
|
|
106
|
+
A WASM build of kotoshu running in the browser. Requires a pure-Ruby
|
|
107
|
+
path (no native extensions) and a reasonable dictionary size budget.
|
|
108
|
+
|
|
109
|
+
### T6.3 — Dictionary editor GUI
|
|
110
|
+
|
|
111
|
+
A desktop app (Tauri / Electron) for curating personal dictionaries
|
|
112
|
+
and custom affix files. Reads/writes Hunspell format.
|
|
113
|
+
|
|
114
|
+
## Acceptance criteria per promotable item
|
|
115
|
+
|
|
116
|
+
Each item, when promoted from this file to its own
|
|
117
|
+
`TODO.impl/{n}-{name}.md`, gets its own acceptance criteria. The
|
|
118
|
+
overarching rule: **never break T1/T2 correctness for a T3+ feature**.
|
|
119
|
+
Re-run the full `--tag ~network --tag ~onnx` suite before tagging any
|
|
120
|
+
T3+ release.
|
|
121
|
+
|
|
122
|
+
## Dependencies
|
|
123
|
+
|
|
124
|
+
- **Blocked by:** T1 (`TODO.impl/36`) and T2 (`TODO.impl/37`) — the
|
|
125
|
+
baseline must be green before adding features.
|
|
126
|
+
- **Blocks:** nothing. This is the long horizon.
|
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
# 57 — T5.1 step 3: Strategies as pure functions
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Finish the MECE refactor of `suggestions/strategies/` started in
|
|
6
|
+
TODO 56 T5.1. Steps 1 and 2 are done (PRs #75, #76):
|
|
7
|
+
|
|
8
|
+
- Step 1 (PR #75): Extracted `SuggestionSet#deduplicate!` as a named
|
|
9
|
+
method with a `duplicates_removed` counter. Was previously buried
|
|
10
|
+
as a side effect inside `sort_and_limit!`.
|
|
11
|
+
- Step 2 (PR #76): `CompositeStrategy#generate` now collects all
|
|
12
|
+
strategy outputs via `flat_map` and builds one `SuggestionSet` at
|
|
13
|
+
the end — was running sort/dedup/limit N times per merge.
|
|
14
|
+
|
|
15
|
+
Step 3 remains: make strategies **pure functions** of
|
|
16
|
+
`(word, dictionary) → candidates`, with no shared state and no
|
|
17
|
+
global config reads.
|
|
18
|
+
|
|
19
|
+
## Current state (what's wrong)
|
|
20
|
+
|
|
21
|
+
`EditDistanceStrategy` is the worst offender. Its constructor:
|
|
22
|
+
|
|
23
|
+
```ruby
|
|
24
|
+
def initialize(name: :edit_distance, language_code: 'en',
|
|
25
|
+
keyboard_layout: nil, **config)
|
|
26
|
+
super(name: name, **config)
|
|
27
|
+
@language_code = language_code
|
|
28
|
+
@keyboard_layout = keyboard_layout
|
|
29
|
+
@frequency_data = load_frequency_data(language_code) # disk IO at construction
|
|
30
|
+
@keyboard_layout = resolve_keyboard_layout(keyboard_layout)
|
|
31
|
+
end
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Consequences:
|
|
35
|
+
|
|
36
|
+
- Strategy is **bound to one language** at construction. Switching
|
|
37
|
+
languages requires a new strategy instance (and re-loads frequency
|
|
38
|
+
data from disk).
|
|
39
|
+
- Strategy reads `Kotoshu::Cache::FrequencyCache` (global state) via
|
|
40
|
+
`load_frequency_data`. Hard to test without network/disk fixtures.
|
|
41
|
+
- Strategy is **stateful** — same instance returns different scores
|
|
42
|
+
for the same `(word, dictionary)` pair if the underlying cache
|
|
43
|
+
changes between calls.
|
|
44
|
+
- Per-document configuration (different frequency list, different
|
|
45
|
+
keyboard layout) requires reconstructing the strategy pipeline.
|
|
46
|
+
|
|
47
|
+
`SymspellStrategy`, `NgramStrategy`, `PhoneticStrategy` have the
|
|
48
|
+
same shape; `SemanticStrategy` already takes its model via context.
|
|
49
|
+
|
|
50
|
+
## Target design
|
|
51
|
+
|
|
52
|
+
Strategies become stateless. Per-call state (language, keyboard,
|
|
53
|
+
frequency data) flows through `Suggestions::Context`.
|
|
54
|
+
|
|
55
|
+
```ruby
|
|
56
|
+
class EditDistanceStrategy < BaseStrategy
|
|
57
|
+
def generate(context)
|
|
58
|
+
keyboard = context.keyboard_layout # was @keyboard_layout
|
|
59
|
+
frequency = context.frequency_data # was @frequency_data
|
|
60
|
+
# ... pure function of (context.word, context.dictionary, keyboard, frequency)
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
`Suggestions::Context` gains accessors for the data the strategies
|
|
66
|
+
need. A `ContextBuilder` (new) is responsible for loading the
|
|
67
|
+
language-specific data lazily, keyed by `language_code` and
|
|
68
|
+
`dictionary.language_code`. The builder is the only place that
|
|
69
|
+
touches `Cache::FrequencyCache` and `Keyboard::Registry`.
|
|
70
|
+
|
|
71
|
+
## Phases
|
|
72
|
+
|
|
73
|
+
### Phase A — Move keyboard_layout to Context
|
|
74
|
+
|
|
75
|
+
`KeyboardProximityStrategy` and `EditDistanceStrategy` both currently
|
|
76
|
+
hold `@keyboard_layout`. Move to Context. Strategies read
|
|
77
|
+
`context.keyboard_layout`.
|
|
78
|
+
|
|
79
|
+
### Phase B — Move frequency_data to Context
|
|
80
|
+
|
|
81
|
+
`EditDistanceStrategy` currently calls `load_frequency_data` in
|
|
82
|
+
`initialize`. Move the loader into a `Suggestions::ContextBuilder`
|
|
83
|
+
that produces a Context with `frequency_data` populated.
|
|
84
|
+
|
|
85
|
+
### Phase C — Drop strategy configuration coupling
|
|
86
|
+
|
|
87
|
+
Stop passing `language_code` to strategy constructors. The language
|
|
88
|
+
is already on `context.dictionary.language_code`. Strategies read
|
|
89
|
+
language from context, not from their own configuration.
|
|
90
|
+
|
|
91
|
+
### Phase D — Verify strategies are pure
|
|
92
|
+
|
|
93
|
+
Add a property test:
|
|
94
|
+
|
|
95
|
+
```ruby
|
|
96
|
+
it "strategy.generate(ctx) returns the same SuggestionSet across calls" do
|
|
97
|
+
# ... generate twice with same context, expect identical results
|
|
98
|
+
end
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## Acceptance criteria
|
|
102
|
+
|
|
103
|
+
- [ ] No strategy reads `Cache::FrequencyCache` or
|
|
104
|
+
`Keyboard::Registry` directly.
|
|
105
|
+
- [ ] No strategy accepts `language_code` or `keyboard_layout` in
|
|
106
|
+
its constructor.
|
|
107
|
+
- [ ] `Suggestions::Context` carries all per-call state.
|
|
108
|
+
- [ ] Strategies pass a purity property test (same context → same
|
|
109
|
+
result, no global-state-dependent variation).
|
|
110
|
+
- [ ] Existing strategy specs continue to pass (or are updated to
|
|
111
|
+
build context with the new state).
|
|
112
|
+
- [ ] Generator (orchestrator) constructs the ContextBuilder once
|
|
113
|
+
per pipeline run, not once per strategy.
|
|
114
|
+
|
|
115
|
+
## Why this matters
|
|
116
|
+
|
|
117
|
+
- **Per-document configuration.** A multi-language document needs
|
|
118
|
+
different strategies per segment. Today, the strategy instances
|
|
119
|
+
are bound to one language at construction — `MultiLanguageChecker`
|
|
120
|
+
has to rebuild the pipeline per segment.
|
|
121
|
+
- **Testing.** Stateful strategies that hit disk in their
|
|
122
|
+
constructor are slow to test and require network/disk fixtures.
|
|
123
|
+
Pure-function strategies only need a Context value object.
|
|
124
|
+
- **Parallelism.** `ParallelChecker` can't safely share strategy
|
|
125
|
+
instances across threads today because the strategies hold mutable
|
|
126
|
+
state. Pure functions are trivially thread-safe.
|
|
127
|
+
|
|
128
|
+
## Dependencies
|
|
129
|
+
|
|
130
|
+
- **Blocked by:** none.
|
|
131
|
+
- **Blocks:** TODO 52 (multi-language document checking) becomes
|
|
132
|
+
much simpler once strategies are pure.
|
|
133
|
+
|
|
134
|
+
## Out of scope
|
|
135
|
+
|
|
136
|
+
- Changing the suggestion algorithm itself.
|
|
137
|
+
- Removing the `BaseStrategy` inheritance hierarchy.
|
|
138
|
+
- Renaming or restructuring the existing strategy classes.
|