text-metrics 0.0.1 → 1.0.0.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d2c50a6c558c741b22dd13e0c7327c0453ca619bc5dc535364c7af81e282a00d
-  data.tar.gz: 761532ff88057efc471f1c0b1f8c1395f527d7ce1d2490e41a982e85994b22f9
+  metadata.gz: d045a3d3bb921b33db65e5f0ae8cb2211541732760e5a059c17ab3bf36335b86
+  data.tar.gz: 8ad9fb90b40800eeafe15e2bffbc48b69b226ac7f71f77429efbbec95d0be2ab
 SHA512:
-  metadata.gz: 5327430d90c3a17bdc4822112700fcba99bb5f6fa5df7a22063b534048d87bfd59eb82b9fcd0363defa659c5e39a5e3109a005b94d55c35e23b2801bbd547532
-  data.tar.gz: 6dde381cb4592e1158aced2f8c90bae5fd585951e1f02450da197b98c0eaa1741903d4cc0310402fb5dba2447e9908187782327c57c5855d000e8ded84239a63
+  metadata.gz: aff19ee3172ad857e3762964fbe13866fffa1f8a91402d138b8b50ce08324c537ae865bc02419b0177fc2d4af5d71cbfd16a4a44ca18fe843e1e90914df2cc39
+  data.tar.gz: dd3cb65ccd6e5f27d4715e367e409f7995dcac7e2e4b77c63f43e074e121ca32b5e584dd2b3167585f501fafe6db296c0cf0202b778a42f058283fad24e0370c

data/CHANGELOG.md

@@ -1,3 +1,37 @@
 # Change log
 
 ## master
+
+## 1.0.0.beta1
+
+First public (beta) release. The API was reworked for stability before tagging 1.0 — see
+[`UPGRADING.md`](UPGRADING.md) for migration details. **Breaking changes:**
+
+- `TextMetrics.new` now takes the text as a positional argument and returns the
+  language-specific analyzer directly: `TextMetrics.new("text", language: :en_us)`.
+  The `TextMetrics::TextMetrics` wrapper and its `#text_metrics_processor` accessor are gone.
+- `language` is now a symbol (`:en_us`, `:fr`); strings are still accepted. An unknown
+  language raises `TextMetrics::Error` instead of a `NoMethodError`.
+- `#all` was renamed to `#to_h`, and it is now the single source of truth for the metric
+  set — every individual reader and `#to_h` are derived from the same list, so they can't drift.
+- Levenshtein moved off the analyzer to the module: `TextMetrics.distance(a, b)` (raw integer)
+  and `TextMetrics.similarity(a, b)` (0–100 score). This replaces
+  `levenshtein_distance_from(other, normalize:)`, whose return value changed meaning with the flag.
+- Punctuation metrics renamed to the singular: `punctuation_count`,
+  `words_per_punctuation_average`, `punctuation_per_sentence_average`.
+- `poly_syllabes_count` and the tokenizers are now private implementation details.
+- Analyzed text is whitespace-normalized once and exposed via `#text`.
+- English syllable counting now uses the CMU Pronouncing Dictionary as the source of truth
+  (falling back to `text-hyphen` only for out-of-vocabulary words), which is more accurate than
+  the previous hyphenation-only approach. English syllable counts and the readability scores
+  derived from them may shift slightly versus `0.x`.
+- Readability scores are now computed from full-precision ratios (rounding only the final
+  result) instead of from the display-rounded averages — this removes errors of several points
+  that the intermediate rounding could introduce.
+- Readability scores are no longer clamped: a Flesch score can now exceed 100 or go negative,
+  as the formulas define. (Previously clamped to 0–100 / 0–18 / 0–20.)
+- French Flesch Reading Ease now uses the correct Kandel-Moles base constant `207` (was `206.835`).
+- `flesch_kincaid_grade` now uses the standard US-grade formula for every language; the previous
+  unsourced French coefficients are gone (Flesch-Kincaid Grade has no validated French adaptation).
+- `letters_per_word_average` and the Coleman-Liau index now count alphabetic letters only, not
+  digits and punctuation, per the Coleman-Liau definition.

data/README.md

@@ -6,9 +6,23 @@
 
 Text Metrics is a Ruby library for text analysis. It is inspired from Textstat library in Python and the Ruby port of [Textstat](https://github.com/kupolak/textstat).
 
-In addition to basic metrics it also provides readability tests like Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index.
+In addition to basic metrics it also provides readability tests like Flesch Reading Ease, Flesch-Kincaid Grade Level, SMOG, Coleman-Liau Index and LIX. The main languages supported are English and French.
 
-At this point the main language supported are English and French.
+## Accurate, dictionary-based syllable counting
+
+Readability scores (Flesch, Flesch-Kincaid, SMOG) are only as trustworthy as the syllable counts they are built on — and the hyphenation heuristics most libraries rely on get a lot of words wrong. Text Metrics counts syllables from real pronunciation dictionaries instead:
+
+- **English** uses the [CMU Pronouncing Dictionary](https://github.com/cmusphinx/cmudict): pronunciation-derived syllable counts for ~126,000 words, falling back to hyphenation only for out-of-vocabulary words. Benchmarked against CMUdict as ground truth, the common hyphenation-only approach returns the **wrong** syllable count for **~46% of dictionary words** — Text Metrics uses the reference count directly.
+- **French** uses counts derived from the [Lexique](http://www.lexique.org/) database, with a vowel heuristic patched by an exceptions list for the words it gets wrong.
+
+This makes syllable-dependent metrics meaningfully more accurate than hyphenation-only implementations, especially on longer and less common words.
+
+### Formula notes
+
+- Scores are computed from full-precision ratios and returned **unclamped** — a Flesch Reading Ease score can legitimately exceed 100 (very easy) or go below 0 (very difficult).
+- **French Flesch Reading Ease** uses the Kandel-Moles (1958) adaptation: `207 − 1.015 × (words/sentences) − 73.6 × (syllables/words)`. Because its base is 207, very easy French text can score slightly above 100.
+- **Flesch-Kincaid Grade** maps to a US school grade and uses the same formula for all languages (there is no validated French adaptation).
+- **Coleman-Liau** counts alphabetic letters only (not digits or punctuation), per its definition.
 
 ## Features
 
@@ -47,23 +61,43 @@ gem "text-metrics", github: "plume-app/text-metrics", branch: "main"
 ## Usage
 
 ```ruby
-@text_analyser = TextMetrics.new(text: "This gem analyses all kind of texts.")
-# get all metrics at once:
-
-@text_analyser.all
-# { words_count: 7, characters_count: 30, sentences_count: 1, syllables_count: 9, syllables_per_word_average: 1.3, letters_per_word_average: 4.29, words_per_sentence_average: 7.0, characters_per_sentence_average: 30.0, flesch_reading_ease: 89.75, flesch_kincaid_grade: 2.5 }
-
-# or get each metric separately:
-@text_analyser.words_count # => 7
-@text_analyser.characters_count # => 30
-@text_analyser.sentences_count # => 1
-@text_analyser.syllables_count # => 9
-@text_analyser.syllables_per_word_average # => 1.3
-@text_analyser.letters_per_word_average # => 4.29
-@text_analyser.words_per_sentence_average # => 7.0
-@text_analyser.characters_per_sentence_average # => 30.0
-@text_analyser.flesch_reading_ease # => 89.75
-@text_analyser.flesch_kincaid_grade # => 2.5
+metrics = TextMetrics.new("This gem analyses all kinds of text.")
+
+# Get every metric at once as a Hash:
+metrics.to_h
+# {
+#   words_count: 7, characters_count: 30, sentences_count: 1, syllables_count: 11,
+#   punctuation_count: 1, syllables_per_word_average: 1.6, letters_per_word_average: 4.29,
+#   words_per_sentence_average: 7.0, characters_per_sentence_average: 30.0,
+#   words_per_punctuation_average: 7.0, punctuation_per_sentence_average: 1.0,
+#   flesch_reading_ease: 64.37, flesch_kincaid_grade: 6.0, lix: 21.29,
+#   smog_index: 0.0, coleman_liau_index: 5.2
+# }
+
+# Or read each metric on its own:
+metrics.words_count                     # => 7
+metrics.characters_count                # => 30
+metrics.flesch_reading_ease             # => 64.37
+metrics.flesch_kincaid_grade            # => 6.0
+```
+
+### Languages
+
+The default language is American English (`:en_us`). Pass `language:` to analyse French:
+
+```ruby
+TextMetrics.new("Bonjour le monde.", language: :fr)
+```
+
+An unknown language raises `TextMetrics::Error`.
+
+### Comparing two texts
+
+Levenshtein comparison is between two texts, so it lives on the module itself:
+
+```ruby
+TextMetrics.distance("kitten", "sitting")   # => 3      raw edit distance
+TextMetrics.similarity("kitten", "sitting") # => 57.14  0–100 score (100.0 == identical)
 ```
 
 ## Contributing
@@ -75,6 +109,8 @@ Bug reports and pull requests are welcome on GitHub at [https://github.com/plume
 This gem was inspired by [Textstat](https://github.com/kupolak/textstat) and [Textstat](https://github.com/textstat/textstat) in Python.
 This gem is generated via [`newgem` template](https://github.com/palkan/newgem) by [@palkan](https://github.com/palkan).
 
+English syllable counts come from the [CMU Pronouncing Dictionary](https://github.com/cmusphinx/cmudict) (unrestricted use), with [text-hyphen](https://github.com/halostatue/text-hyphen) as a fallback. French syllable counts are derived from [Lexique](http://www.lexique.org/).
+
 ## License
 
 The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/UPGRADING.md

@@ -0,0 +1,73 @@
+# Upgrading
+
+## To 1.0 (from the pre-release `0.x` internal API)
+
+1.0 is the first public release. The internal `0.x` API was reworked once, deliberately, so
+the public surface can stay stable from here on. Everything below is a one-time migration.
+
+### Building an analyzer
+
+`text` is now positional, and `TextMetrics.new` returns the analyzer itself — there is no
+longer a wrapper object with a `text_metrics_processor` accessor.
+
+```ruby
+# Before
+analyser = TextMetrics.new(text: "Some text", language: "fr")
+analyser.text_metrics_processor.words_count
+
+# After
+metrics = TextMetrics.new("Some text", language: :fr)
+metrics.words_count
+```
+
+### Languages are symbols, and unknown ones raise
+
+```ruby
+# Before — unknown language blew up with NoMethodError on nil
+TextMetrics.new(text: "x", language: "es")
+
+# After — symbols (strings still accepted), unknown languages raise TextMetrics::Error
+TextMetrics.new("x", language: :es) # => raises TextMetrics::Error
+```
+
+### `#all` is now `#to_h`
+
+```ruby
+# Before
+metrics.all
+
+# After
+metrics.to_h
+```
+
+`#to_h` and the individual metric readers are now generated from a single list
+(`TextMetrics::Processors::Base::METRICS`), so they can never report different sets again.
+
+### Levenshtein moved to the module
+
+The single method with a `normalize:` flag (which returned a raw distance or a 0–100 score
+depending on the flag) is replaced by two intention-revealing module methods:
+
+```ruby
+# Before
+metrics.levenshtein_distance_from("other", normalize: false) # raw distance
+metrics.levenshtein_distance_from("other")                   # 0–100 score
+
+# After
+TextMetrics.distance("text", "other")   # => Integer, raw edit distance
+TextMetrics.similarity("text", "other") # => Float, 0–100 score (100.0 == identical)
+```
+
+### Renamed metrics
+
+The punctuation metrics dropped the plural; some helpers are now private:
+
+| Before                             | After                            |
+| ---------------------------------- | -------------------------------- |
+| `punctuations_count`               | `punctuation_count`              |
+| `words_per_punctuations_average`   | `words_per_punctuation_average`  |
+| `punctuations_per_sentence_average`| `punctuation_per_sentence_average` |
+| `poly_syllabes_count` (public)     | private implementation detail    |
+
+The word/sentence/punctuation tokenizers are now private as well — use the count and average
+metrics or `#to_h`.

data/lib/text-metrics.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+# Bundler auto-requires the hyphenated gem name (`require "text-metrics"`) by default.
+# This shim points it at the real entry point so `gem "text-metrics"` works without an
+# explicit `require:` option.
+require "text_metrics"