text-metrics 1.0.0.beta1 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d045a3d3bb921b33db65e5f0ae8cb2211541732760e5a059c17ab3bf36335b86
-  data.tar.gz: 8ad9fb90b40800eeafe15e2bffbc48b69b226ac7f71f77429efbbec95d0be2ab
+  metadata.gz: 0aa8550e92948515979c3ddf605d9a591992f7fd855097c3a942bd4bdaa2e490
+  data.tar.gz: 540fd11f846a899d8a0266e53e303c879c5655fe18d674165b5a7a480332b245
 SHA512:
-  metadata.gz: aff19ee3172ad857e3762964fbe13866fffa1f8a91402d138b8b50ce08324c537ae865bc02419b0177fc2d4af5d71cbfd16a4a44ca18fe843e1e90914df2cc39
-  data.tar.gz: dd3cb65ccd6e5f27d4715e367e409f7995dcac7e2e4b77c63f43e074e121ca32b5e584dd2b3167585f501fafe6db296c0cf0202b778a42f058283fad24e0370c
+  metadata.gz: ca14e801a86eee04c3633b53a5eb77d403308fe6698a111664c6eac82409977d7e697019611ed1e21dcb43f4ec7cb275048d2f9fad6e279d9613a0a73b41f982
+  data.tar.gz: 47218fd86626d8656217ee13cdd8d67f2eb2950a88ee18aed23bd42d73e2c4d5a1f20ff44b2650f7fb308b0dd4184682bfb5891f35f5fe9680e53878713f4ff8

data/CHANGELOG.md

@@ -1,6 +1,13 @@
 # Change log
 
-## master
+## main
+
+## 1.0.0
+- First public release.
+- see `UPGRADING.md` for migration details from the pre-release `0.x` API.
+
+## ## 1.0.0.beta2
+- Add `gunning_fog_index` to the analyzer metrics and `#to_h` output.
 
 ## 1.0.0.beta1
 

data/README.md

@@ -4,54 +4,57 @@
 
 # Text Metrics
 
-Text Metrics is a Ruby library for text analysis. It is inspired from Textstat library in Python and the Ruby port of [Textstat](https://github.com/kupolak/textstat).
+Text Metrics is a Ruby library for analysing text. It was inspired by the Python Textstat library and the Ruby port of [Textstat](https://github.com/kupolak/textstat).
 
-In addition to basic metrics it also provides readability tests like Flesch Reading Ease, Flesch-Kincaid Grade Level, SMOG, Coleman-Liau Index and LIX. The main languages supported are English and French.
+It gives you the everyday counts you need — words, characters, sentences and syllables — plus readability scores such as Flesch Reading Ease, Flesch-Kincaid Grade Level, SMOG, Gunning Fog, Coleman-Liau Index and LIX. English and French are supported.
+
+It is battle-tested in production on millions of student writings at [Plume](https://plume-app.co).
 
 ## Accurate, dictionary-based syllable counting
 
-Readability scores (Flesch, Flesch-Kincaid, SMOG) are only as trustworthy as the syllable counts they are built on — and the hyphenation heuristics most libraries rely on get a lot of words wrong. Text Metrics counts syllables from real pronunciation dictionaries instead:
+Readability scores such as Flesch, Flesch-Kincaid and SMOG depend heavily on syllable counts. Many libraries estimate those counts with hyphenation rules, which are fast but often wrong. Text Metrics starts from real pronunciation data instead:
 
-- **English** uses the [CMU Pronouncing Dictionary](https://github.com/cmusphinx/cmudict): pronunciation-derived syllable counts for ~126,000 words, falling back to hyphenation only for out-of-vocabulary words. Benchmarked against CMUdict as ground truth, the common hyphenation-only approach returns the **wrong** syllable count for **~46% of dictionary words** — Text Metrics uses the reference count directly.
-- **French** uses counts derived from the [Lexique](http://www.lexique.org/) database, with a vowel heuristic patched by an exceptions list for the words it gets wrong.
+- **English** uses the [CMU Pronouncing Dictionary](https://github.com/cmusphinx/cmudict), which provides pronunciation-derived syllable counts for about 126,000 words. Hyphenation is only used when a word is not in the dictionary. Against CMUdict as ground truth, a hyphenation-only approach gets the syllable count wrong for about **46% of dictionary words**.
+- **French** uses counts derived from the [Lexique](http://www.lexique.org/) database, plus a vowel heuristic and an exceptions list for the words the heuristic misses.
 
-This makes syllable-dependent metrics meaningfully more accurate than hyphenation-only implementations, especially on longer and less common words.
+That makes syllable-sensitive scores more reliable, especially for longer or less common words.
 
 ### Formula notes
 
-- Scores are computed from full-precision ratios and returned **unclamped** — a Flesch Reading Ease score can legitimately exceed 100 (very easy) or go below 0 (very difficult).
-- **French Flesch Reading Ease** uses the Kandel-Moles (1958) adaptation: `207 − 1.015 × (words/sentences) − 73.6 × (syllables/words)`. Because its base is 207, very easy French text can score slightly above 100.
-- **Flesch-Kincaid Grade** maps to a US school grade and uses the same formula for all languages (there is no validated French adaptation).
+- Scores are computed from full-precision ratios and returned **unclamped**. A Flesch Reading Ease score can legitimately be above 100 for very easy text, or below 0 for very difficult text.
+- **French Flesch Reading Ease** uses the Kandel-Moles (1958) adaptation: `207 − 1.015 × (words/sentences) − 73.6 × (syllables/words)`. Because the formula starts from 207, very easy French text can score slightly above 100.
+- **Flesch-Kincaid Grade** maps to a US school grade. It uses the same formula for every language because there is no validated French adaptation.
+- **Gunning Fog** uses words with three or more syllables as complex words, matching the same syllable counts used by SMOG.
 - **Coleman-Liau** counts alphabetic letters only (not digits or punctuation), per its definition.
 
 ## Features
 
 _Basic metrics:_
 
-- [x] words count
-- [x] characters count
-- [x] sentences count
-- [x] syllables per word average
-- [x] letters per word average
-- [x] sentence length average
-- [x] sentence length (characters) average
+- [x] word count
+- [x] character count
+- [x] sentence count
+- [x] average syllables per word
+- [x] average letters per word
+- [x] average words per sentence
+- [x] average characters per sentence
 
 _Readability tests:_
 
 - [x] Flesch Reading Ease
 - [x] Flesch-Kincaid Grade Level
 - [x] Smog Index
+- [x] Gunning Fog Index
 - [x] Coleman-Liau Index
 - [x] Lix Index
-- [ ] Gunning Fog Index
 
 ## Installation
 
-No official release yet, but you can install it from GitHub:
+Text Metrics is not published to RubyGems yet. For now, install it from GitHub:
 
 ```ruby
 # Gemfile
-gem "text-metrics", github: "plume-app/text-metrics", branch: "main"
+gem "text-metrics", "~> 1.0.0"
 ```
 
 ### Supported Ruby versions
@@ -63,37 +66,38 @@ gem "text-metrics", github: "plume-app/text-metrics", branch: "main"
 ```ruby
 metrics = TextMetrics.new("This gem analyses all kinds of text.")
 
-# Get every metric at once as a Hash:
+# Get every metric at once:
 metrics.to_h
 # {
-#   words_count: 7, characters_count: 30, sentences_count: 1, syllables_count: 11,
-#   punctuation_count: 1, syllables_per_word_average: 1.6, letters_per_word_average: 4.29,
+#   words_count: 7, characters_count: 30, sentences_count: 1, syllables_count: 10,
+#   punctuation_count: 1, syllables_per_word_average: 1.4, letters_per_word_average: 4.14,
 #   words_per_sentence_average: 7.0, characters_per_sentence_average: 30.0,
 #   words_per_punctuation_average: 7.0, punctuation_per_sentence_average: 1.0,
-#   flesch_reading_ease: 64.37, flesch_kincaid_grade: 6.0, lix: 21.29,
-#   smog_index: 0.0, coleman_liau_index: 5.2
+#   flesch_reading_ease: 78.87, flesch_kincaid_grade: 4.0, lix: 21.29,
+#   smog_index: 0.0, gunning_fog_index: 8.5, coleman_liau_index: 4.33
 # }
 
-# Or read each metric on its own:
+# Or ask for a single metric:
 metrics.words_count                     # => 7
 metrics.characters_count                # => 30
-metrics.flesch_reading_ease             # => 64.37
-metrics.flesch_kincaid_grade            # => 6.0
+metrics.flesch_reading_ease             # => 78.87
+metrics.flesch_kincaid_grade            # => 4.0
+metrics.gunning_fog_index                # => 8.5
 ```
 
 ### Languages
 
-The default language is American English (`:en_us`). Pass `language:` to analyse French:
+American English (`:en_us`) is the default. To analyse French text, pass `language: :fr`:
 
 ```ruby
 TextMetrics.new("Bonjour le monde.", language: :fr)
 ```
 
-An unknown language raises `TextMetrics::Error`.
+Unsupported languages raise `TextMetrics::Error`.
 
 ### Comparing two texts
 
-Levenshtein comparison is between two texts, so it lives on the module itself:
+Levenshtein distance compares two strings, so it is exposed on the `TextMetrics` module:
 
 ```ruby
 TextMetrics.distance("kitten", "sitting")   # => 3      raw edit distance
@@ -102,12 +106,12 @@ TextMetrics.similarity("kitten", "sitting") # => 57.14  0–100 score (100.0 ==
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at [https://github.com/plume-app/text-metrics](https://github.com/plume-app/text-metrics).
+Bug reports and pull requests are welcome on [GitHub](https://github.com/plume-app/text-metrics).
 
 ## Credits
 
-This gem was inspired by [Textstat](https://github.com/kupolak/textstat) and [Textstat](https://github.com/textstat/textstat) in Python.
-This gem is generated via [`newgem` template](https://github.com/palkan/newgem) by [@palkan](https://github.com/palkan).
+This gem was inspired by [Textstat](https://github.com/textstat/textstat) in Python and the Ruby [Textstat](https://github.com/kupolak/textstat) port.
+It was generated from the [`newgem` template](https://github.com/palkan/newgem) by [@palkan](https://github.com/palkan).
 
 English syllable counts come from the [CMU Pronouncing Dictionary](https://github.com/cmusphinx/cmudict) (unrestricted use), with [text-hyphen](https://github.com/halostatue/text-hyphen) as a fallback. French syllable counts are derived from [Lexique](http://www.lexique.org/).
 

data/lib/text_metrics/processors/base.rb

@@ -25,6 +25,7 @@ module TextMetrics
         flesch_kincaid_grade
         lix
         smog_index
+        gunning_fog_index
         coleman_liau_index
       ].freeze
 
@@ -120,6 +121,12 @@ module TextMetrics
         0.0
       end
 
+      def gunning_fog_index
+        return 0.0 if words_count.zero?
+
+        (0.4 * (average_words_per_sentence + 100.0 * count_polysyllabic_words / words_count)).round(1)
+      end
+
       def coleman_liau_index
         return 0.0 if words_count.zero?
 

data/lib/text_metrics/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module TextMetrics # :nodoc:
-  VERSION = "1.0.0.beta1"
+  VERSION = "1.0.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: text-metrics
 version: !ruby/object:Gem::Version
-  version: 1.0.0.beta1
+  version: 1.0.0
 platform: ruby
 authors:
 - Adrien POLY