pyests 0.3.0__tar.gz → 0.4.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. {pyests-0.3.0 → pyests-0.4.0}/PKG-INFO +108 -38
  2. {pyests-0.3.0 → pyests-0.4.0}/README.es.md +107 -37
  3. {pyests-0.3.0 → pyests-0.4.0}/README.md +107 -37
  4. {pyests-0.3.0 → pyests-0.4.0}/ests/__init__.py +12 -2
  5. {pyests-0.3.0 → pyests-0.4.0}/ests/basic_stats.py +23 -39
  6. {pyests-0.3.0 → pyests-0.4.0}/ests/cohesion_stats.py +50 -99
  7. {pyests-0.3.0 → pyests-0.4.0}/ests/components.py +226 -74
  8. {pyests-0.3.0 → pyests-0.4.0}/ests/constants.py +476 -91
  9. {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/collocations.py +8 -14
  10. {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/compare.py +54 -113
  11. {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/dispersion.py +4 -8
  12. {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/keyness.py +24 -40
  13. {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/kwic.py +9 -13
  14. {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/stylometry.py +31 -62
  15. pyests-0.4.0/ests/datasets/__init__.py +5 -0
  16. {pyests-0.3.0 → pyests-0.4.0}/ests/datasets/dataset.py +3 -7
  17. {pyests-0.3.0 → pyests-0.4.0}/ests/datasets/freq_dict.py +26 -59
  18. {pyests-0.3.0 → pyests-0.4.0}/ests/datasets/spanish_literature.py +12 -16
  19. pyests-0.4.0/ests/datasets/spanish_sonnets.py +301 -0
  20. {pyests-0.3.0 → pyests-0.4.0}/ests/diversity_stats.py +61 -144
  21. pyests-0.4.0/ests/exceptions.py +88 -0
  22. {pyests-0.3.0 → pyests-0.4.0}/ests/extractors.py +12 -25
  23. {pyests-0.3.0 → pyests-0.4.0}/ests/lexical_stats.py +20 -36
  24. {pyests-0.3.0 → pyests-0.4.0}/ests/morph_stats.py +23 -50
  25. pyests-0.4.0/ests/phon_stats.py +584 -0
  26. {pyests-0.3.0 → pyests-0.4.0}/ests/readability_stats.py +41 -79
  27. pyests-0.4.0/ests/style_stats.py +649 -0
  28. {pyests-0.3.0 → pyests-0.4.0}/ests/syllables.py +48 -55
  29. {pyests-0.3.0 → pyests-0.4.0}/ests/syntax_stats.py +51 -97
  30. {pyests-0.3.0 → pyests-0.4.0}/ests/utils.py +101 -121
  31. pyests-0.4.0/ests/verse_stats.py +1226 -0
  32. {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/__init__.py +4 -0
  33. {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/corpus.py +11 -20
  34. {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/fingerprinting.py +9 -15
  35. pyests-0.4.0/ests/visualizers/highlight.py +1063 -0
  36. {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/sentences.py +5 -8
  37. {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/stylometry.py +8 -13
  38. {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/vocabulary.py +2 -2
  39. {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/word_tree.py +7 -14
  40. {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/zipf.py +5 -7
  41. {pyests-0.3.0 → pyests-0.4.0}/pyproject.toml +4 -1
  42. pyests-0.3.0/ests/datasets/__init__.py +0 -4
  43. pyests-0.3.0/ests/exceptions.py +0 -93
  44. {pyests-0.3.0 → pyests-0.4.0}/.gitignore +0 -0
  45. {pyests-0.3.0 → pyests-0.4.0}/LICENSE.txt +0 -0
  46. {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/__init__.py +0 -0
  47. {pyests-0.3.0 → pyests-0.4.0}/ests/py.typed +0 -0
  48. {pyests-0.3.0 → pyests-0.4.0}/ests/resources/README.md +0 -0
  49. {pyests-0.3.0 → pyests-0.4.0}/ests/resources/connectors.tsv +0 -0
  50. {pyests-0.3.0 → pyests-0.4.0}/ests/resources/google_books_top10000.txt +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: pyests
3
- Version: 0.3.0
3
+ Version: 0.4.0
4
4
  Summary: Spanish Texts Statistics
5
5
  Project-URL: Homepage, https://github.com/SergeyShk/esTS
6
6
  Project-URL: Repository, https://github.com/SergeyShk/esTS
@@ -68,9 +68,9 @@ Description-Content-Type: text/markdown
68
68
 
69
69
  ---
70
70
 
71
- **esTS** computes for Spanish texts what usually requires assembling several separate tools: basic statistics, readability, lexical diversity, lexical sophistication, morphology, syntax and cohesion - by published formulas with the coefficients and the scales of their authors, and by the parts of speech and the features of Universal Dependencies.
71
+ **esTS** computes statistics of Spanish texts: basic statistics, readability, lexical diversity, lexical sophistication, style, phonostatistics, morphology, syntax and cohesion - by published formulas with the coefficients and the scales of their authors, and by the parts of speech and the features of Universal Dependencies.
72
72
 
73
- The library works both with raw strings and with `Doc` objects of [spaCy](https://github.com/explosion/spaCy): sentences, words and character N-grams are extracted by rules, syllables and stress follow from the orthography, and only the morphological, the syntactic, the cohesion and the lexical sophistication statistics, the profile of the function words and the comparison of corpora need a trained model.
73
+ The library works both with raw strings and with `Doc` objects of [spaCy](https://github.com/explosion/spaCy); most statistics need no trained model (see Installation).
74
74
 
75
75
  * **[Object extraction](https://sergeyshk.github.io/esTS/extractors/sentences/)** - configurable sentence, word and character N-gram tokenizers that know the inverted marks, the dialogue dash and the abbreviations of Spanish
76
76
  * **[Syllables and stress](https://sergeyshk.github.io/esTS/syllables/)** - rule-based syllabification and the stressed syllable derived from the spelling, with no dictionary
@@ -79,15 +79,16 @@ The library works both with raw strings and with `Doc` objects of [spaCy](https:
79
79
  * **[Lexical diversity metrics](https://sergeyshk.github.io/esTS/stats/diversity_stats/)** - TTR and its variations, MATTR, MSTTR, MTLD, HD-D, Simpson's and Yule's indices, entropy, Zipf's and Heaps' laws
80
80
  * **[Morphological statistics](https://sergeyshk.github.io/esTS/stats/morph_stats/)** - parts of speech and fifteen grammatical features of Universal Dependencies, with the markers of Spanish: the moods, the non-finite forms, `ser` against `estar`, the adverbs in `-mente`
81
81
  * **[Corpus measures](https://sergeyshk.github.io/esTS/corpus/keyness/)** - keywords against a reference corpus, collocations, the dispersion of a word over the parts of a text, a KWIC concordance and the stylometry of authorship: Burrows's Delta with its variants, Zeta, the Mendenhall curve, the profile of the function words; the comparison of two corpora by 132 features of a text with effect sizes
82
- * **[Visualizers](https://sergeyshk.github.io/esTS/visualizers/zipf/)** - Zipf's law, literature fingerprinting, a word tree, lexical dispersion and keywords, a network of collocations, a dendrogram, PCA and MDS by Delta, vocabulary growth, sentence lengths
83
- * **[Datasets](https://sergeyshk.github.io/esTS/datasets/spanishliterature/)** - Spanish-language literature in the public domain: 150 works by 33 authors from Spain, Latin America and the Philippines in prose, poems, drama and publicism, with the genre, the years and the country; a frequency dictionary of 83,785 lemmas by Google Books Ngram with ipm, range and dispersion
82
+ * **[Visualizers](https://sergeyshk.github.io/esTS/visualizers/zipf/)** - Zipf's law, literature fingerprinting, a word tree, lexical dispersion and keywords, a network of collocations, a dendrogram, PCA and MDS by Delta, vocabulary growth, sentence lengths, and the highlighting of the fragments the statistics count, such as long sentences and passives
83
+ * **[Datasets](https://sergeyshk.github.io/esTS/datasets/spanishliterature/)** - Spanish-language literature in the public domain: 150 works by 33 authors in four genres; 4,259 sonnets of the 15th-20th centuries with the metrical pattern and the rhyme of every line; a frequency dictionary of 83,785 lemmas by Google Books Ngram
84
84
  * **[spaCy components](https://sergeyshk.github.io/esTS/components/)** - every statistics class as a component of a pipeline, the statistics attached to the `Doc` in one pass
85
85
  * **[Cohesion statistics](https://sergeyshk.github.io/esTS/stats/cohesion_stats/)** - the overlap of nouns, arguments and content words between sentences, givenness and temporal cohesion in the manner of Coh-Metrix, with the density of 255 Spanish discourse markers
86
86
  * **[Lexical sophistication statistics](https://sergeyshk.github.io/esTS/stats/lexical_stats/)** - how rare the words of a text are in the language: the frequency, range and dispersion of the lemmas by a dictionary of Google Books Ngram, the frequency bands top-1000 to 10000, surprisal, perplexity and lexical density
87
+ * **[Style metrics](https://sergeyshk.github.io/esTS/stats/style_stats/)** - the SEO indicators of Advego and Text.ru (nausea, water content, spam score, naturalness by Zipf's law, keyword density) and the markers of the officialese style by the Spanish guides to plain language: verbal nouns, compound prepositions, parenthetical expressions and clichés
88
+ * **[Phonostatistics](https://sergeyshk.github.io/esTS/stats/phon_stats/)** - the shares of the classes of sounds, consonant clusters, hiatuses, open syllables, hardness and the indices of alliteration and assonance, over the sounds of a rule-based transcription
89
+ * **[Verse statistics](https://sergeyshk.github.io/esTS/stats/verse_stats/)** - the scansion of Spanish verse by its syllabic meter: the metrical syllables, the meter of a poem, the stress profile and the types of the endecasílabo, the rhyme in full and by assonance with its scheme, the strophes and the form of a poem
87
90
  * **[Syntactic statistics](https://sergeyshk.github.io/esTS/stats/syntax_stats/)** - the dependency tree by distances, depth, clauses and coordination, with the constructions of the administrative style: the passive with `ser` and with `se`, the participial and the gerund clauses, the chains of `de`, the split predicates
88
91
 
89
- Style, phonostatistics, metre and rhyme come in 0.4.
90
-
91
92
  ## Installation
92
93
 
93
94
  Requires Python 3.11 or newer.
@@ -102,13 +103,13 @@ Or with [uv](https://docs.astral.sh/uv/):
102
103
  uv add pyests
103
104
  ```
104
105
 
105
- The distribution on PyPI is `pyests`, the package it installs is `ests`. The basic statistics, the readability and the lexical diversity metrics need no spaCy model; the morphological, the syntactic, the cohesion and the lexical sophistication statistics of a string do, and so do the profile of the function words, the features of a text and the comparison of corpora, and parsing a text yourself to pass the `Doc` instead of a string:
106
+ The distribution on PyPI is `pyests`, the package it installs is `ests`. The basic statistics, readability, lexical diversity, the style metrics except the verbal nouns, phonostatistics and verse statistics need no spaCy model. The morphological, syntactic, cohesion and lexical sophistication statistics of a string need one, and so do the verbal nouns, the profile of the function words, the features of a text, the comparison of corpora and parsing a `Doc` yourself:
106
107
 
107
108
  ```bash
108
109
  python -m spacy download es_core_news_sm
109
110
  ```
110
111
 
111
- The statistics by the frequency dictionary need it downloaded once with `FreqDict().download()`. The datasets go to the directory `ests_data` next to the installed package; another one is passed in `data_dir` or set in the environment variable `ESTS_DATA_DIR` before the package is imported.
112
+ The statistics by the frequency dictionary need it downloaded once with `FreqDict().download()`. The datasets go to the directory `ests_data` next to the installed package, or to the one passed in `data_dir` or set in `ESTS_DATA_DIR` before the package is imported.
112
113
 
113
114
  ## Quick start
114
115
 
@@ -149,7 +150,7 @@ The statistics by the frequency dictionary need it downloaded once with `FreqDic
149
150
 
150
151
  ### Object extraction
151
152
 
152
- The library allows creating your own tools for sentence, word and character N-gram extraction from a text, which can be further employed for counting statistics. The sentence splitter knows the inverted marks, the dash of a line of dialogue with the remark of the narrator, the abbreviations and the initials of Spanish; the word tokenizer keeps clitics, ordinals and numbers written the Spanish way together, and lemmas come from [simplemma](https://github.com/adbar/simplemma), which needs no model.
153
+ Configurable tools extract sentences, words and character N-grams from a text for the statistics. The sentence splitter knows the inverted marks, the dialogue dash and the abbreviations of Spanish; the word tokenizer keeps clitics, ordinals and Spanish numbers whole, and lemmas come from [simplemma](https://github.com/adbar/simplemma), which needs no model.
153
154
 
154
155
  ```python
155
156
  >>> from ests import CharNgramsExtractor, SentsExtractor, WordsExtractor
@@ -172,7 +173,7 @@ More in the [documentation](https://sergeyshk.github.io/esTS/extractors/sentence
172
173
 
173
174
  <br>
174
175
 
175
- Spanish spelling encodes both the syllable boundaries and the stress, so the library needs no dictionary: diphthongs, hiatuses and triphthongs, the silent `u` of `qu` and `gu`, the vocalic `y`, the `h` inside a diphthong and the consonant clusters give the syllables; the written accent, or the ending of the word when there is none, gives the stressed syllable. Adverbs in `-mente` and hyphenated compounds carry two stresses.
176
+ The syllables and the stress follow from the Spanish spelling, with no dictionary: the written accent, or the ending of the word when there is none, gives the stressed syllable. Adverbs in `-mente` and hyphenated compounds carry two stresses.
176
177
 
177
178
  ```python
178
179
  >>> from ests.syllables import stress_type, syllabify, word_stress, word_stresses
@@ -217,7 +218,7 @@ The library allows extracting the following statistics from a text:
217
218
  * the distribution of words by the number of letters
218
219
  * the distribution of words by the number of syllables
219
220
 
220
- A complex word has three or more syllables and a long word seven or more letters, as the Spanish readability formulas count them. Any statistic can be printed in a readable form:
221
+ A complex word has three or more syllables and a long word seven or more letters. Any statistic can be printed in a readable form:
221
222
 
222
223
  ```python
223
224
  >>> from ests import BasicStats
@@ -260,9 +261,9 @@ The library allows counting the following readability metrics:
260
261
  * LIX readability measure
261
262
  * RIX readability measure
262
263
 
263
- An interpretation layer works on top of the formulas: the band of a scale for the reading ease and for Legibilidad µ, a consensus grade as the median of the grade formulas, the school stage and the reader age of Spain, and reading time by the norms of Spanish-speaking readers.
264
+ On top of the formulas: the band of a scale for the reading ease and for Legibilidad µ, a consensus grade (the median of the grade formulas), the school stage and the reader age in Spain, and reading time.
264
265
 
265
- The coefficients of the Flesch reading ease are selected by the `preset` argument: by default the *fórmula de perspicuidad* of Szigriszt-Pazos with the INFLESZ scale validated on texts for patients (`general`); the coefficients of Fernández Huerta with the bands of their author are available as `classic`.
266
+ The coefficients of the Flesch reading ease are selected by `preset`: by default the *fórmula de perspicuidad* of Szigriszt-Pazos with the INFLESZ scale (`general`), or Fernández Huerta with the bands of its author (`classic`).
266
267
 
267
268
  ```python
268
269
  >>> from pprint import pprint
@@ -322,7 +323,7 @@ The library allows counting 32 lexical diversity metrics, among them:
322
323
  * Shannon entropy, evenness and perplexity
323
324
  * the slope of Zipf's law, the Zipf-Mandelbrot fit and the exponent of Heaps' law
324
325
 
325
- Any metric can be computed over windows of equal length, which is the standard way to compare texts of different lengths: the mean over the windows comes with a confidence interval.
326
+ Any metric can be computed over windows of equal length, to compare texts of different lengths, with a confidence interval of the mean.
326
327
 
327
328
  ```python
328
329
  >>> from ests import DiversityStats
@@ -352,7 +353,7 @@ More in the [documentation](https://sergeyshk.github.io/esTS/stats/diversity_sta
352
353
 
353
354
  <br>
354
355
 
355
- The library annotates a text with the parts of speech and the grammatical features of Universal Dependencies, as the Spanish models of spaCy give them, and counts them:
356
+ The parts of speech and the grammatical features of Universal Dependencies, as the Spanish models of spaCy give them:
356
357
 
357
358
  * the part of speech and fifteen features: case, definiteness, degree, gender, mood, numeral type, number, person, polarity, politeness, possessive, pronoun type, reflexive, tense and verb form
358
359
  * the distribution of the words by the values of any feature, and the parse of the text word by word
@@ -376,7 +377,7 @@ The library annotates a text with the parts of speech and the grammatical featur
376
377
  0.5
377
378
  ```
378
379
 
379
- The statistics need a spaCy model: a text is parsed with `es_core_news_sm`, and any other pipeline can be passed in `nlp`.
380
+ A text is parsed with `es_core_news_sm`; another pipeline can be passed in `nlp`.
380
381
 
381
382
  More in the [documentation](https://sergeyshk.github.io/esTS/stats/morph_stats/).
382
383
 
@@ -387,7 +388,7 @@ More in the [documentation](https://sergeyshk.github.io/esTS/stats/morph_stats/)
387
388
 
388
389
  <br>
389
390
 
390
- The library measures the dependency tree of Universal Dependencies and the constructions that the Spanish guides to clear language warn about:
391
+ The dependency tree of Universal Dependencies and the constructions that the Spanish guides to clear language warn about:
391
392
 
392
393
  * the complexity of the tree: dependency distances, depth, leaves and subtrees, valency of the finite verbs, coordination chains, clauses and subordinate clauses, modifiers per noun
393
394
  * the constructions: the passive with `ser` and with `se`, the participial and the gerund clauses, the chains of `de`, the split predicates, the impersonal `se`, the words of negation, the ratio of nouns to verbs
@@ -412,7 +413,7 @@ The library measures the dependency tree of Universal Dependencies and the const
412
413
  2.05
413
414
  ```
414
415
 
415
- The statistics need a parse: a text is parsed with `es_core_news_sm`, and any other pipeline can be passed in `nlp`.
416
+ A text is parsed with `es_core_news_sm`; another pipeline with a parser can be passed in `nlp`.
416
417
 
417
418
  More in the [documentation](https://sergeyshk.github.io/esTS/stats/syntax_stats/).
418
419
 
@@ -423,7 +424,7 @@ More in the [documentation](https://sergeyshk.github.io/esTS/stats/syntax_stats/
423
424
 
424
425
  <br>
425
426
 
426
- The library measures referential cohesion in the manner of Coh-Metrix and of its Spanish adaptation Coh-Metrix-Esp:
427
+ Referential cohesion in the manner of Coh-Metrix and its Spanish adaptation Coh-Metrix-Esp:
427
428
 
428
429
  * the overlap of nouns, of arguments and of content words between adjacent sentences and between all pairs of sentences, binary and proportional
429
430
  * givenness: pronouns, demonstratives and the content words whose lemma was already used
@@ -450,7 +451,7 @@ The library measures referential cohesion in the manner of Coh-Metrix and of its
450
451
  (86.96, 43.48)
451
452
  ```
452
453
 
453
- The statistics need the annotation: a text is parsed with `es_core_news_sm`, and any other pipeline can be passed in `nlp`.
454
+ A text is parsed with `es_core_news_sm`; another pipeline can be passed in `nlp`.
454
455
 
455
456
  More in the [documentation](https://sergeyshk.github.io/esTS/stats/cohesion_stats/).
456
457
 
@@ -461,7 +462,7 @@ More in the [documentation](https://sergeyshk.github.io/esTS/stats/cohesion_stat
461
462
 
462
463
  <br>
463
464
 
464
- How rare the words of a text are in the language, in the manner of TAALES: the mean frequency, range and dispersion of the lemmas by the frequency dictionary of Google Books Ngram, the shares of the words of the frequency bands top-1000, 2000, 5000 and 10000, the surprisal and the perplexity by the unigram model of the dictionary, the lexical density. The bands and the density work out of the box; the statistics by the dictionary need it downloaded once.
465
+ How rare the words of a text are in the language, in the manner of TAALES: the mean frequency, range and dispersion of the lemmas by the frequency dictionary of Google Books Ngram, the shares of the frequency bands top-1000, 2000, 5000 and 10000, surprisal and perplexity, the lexical density. The statistics by the dictionary need it downloaded once.
465
466
 
466
467
  ```python
467
468
  >>> from ests import LexicalStats
@@ -477,6 +478,71 @@ More in the [documentation](https://sergeyshk.github.io/esTS/stats/lexical_stats
477
478
 
478
479
  </details>
479
480
 
481
+ <details>
482
+ <summary><b>Style metrics</b></summary>
483
+
484
+ <br>
485
+
486
+ The SEO indicators of Advego and Text.ru - nausea, water content, spam score, naturalness by Zipf's law, keyword density - and the markers of the officialese style that the Spanish guides to plain language warn about: the nouns derived from a verb, the compound prepositions of the administrative style, the parenthetical expressions and the clichés.
487
+
488
+ ```python
489
+ >>> from ests import StyleStats
490
+
491
+ >>> ss = StyleStats("Se procedió a la revisión del expediente en el marco del plan a la mayor brevedad.")
492
+ >>> ss.compound_prepositions, ss.cliches, ss.verbal_nouns
493
+ (6.25, 12.5, 20.0)
494
+ ```
495
+
496
+ More in the [documentation](https://sergeyshk.github.io/esTS/stats/style_stats/).
497
+
498
+ </details>
499
+
500
+ <details>
501
+ <summary><b>Phonostatistics</b></summary>
502
+
503
+ <br>
504
+
505
+ The shares of the classes of sounds, the consonant clusters, the hiatuses, the open syllables, the hardness and the indices of alliteration and assonance, counted over the sounds of a rule-based transcription, not over the letters: `h` is silent, `ll` and `rr` are one sound.
506
+
507
+ ```python
508
+ >>> from ests import PhonStats
509
+
510
+ >>> ps = PhonStats("Los suspiros se escapan de su boca de fresa")
511
+ >>> round(ps.p_voiceless, 3), round(ps.hardness, 3), ps.sounds[-1]
512
+ (0.371, 0.684, ('f', 'r', 'e', 's', 'a'))
513
+ ```
514
+
515
+ More in the [documentation](https://sergeyshk.github.io/esTS/stats/phon_stats/).
516
+
517
+ </details>
518
+
519
+ <details>
520
+ <summary><b>Verse statistics</b></summary>
521
+
522
+ <br>
523
+
524
+ The scansion of Spanish verse by its syllabic meter: the metrical syllables of a line with the synalepha and the law of the final stress, fitted to the meter of the poem; the meter, the stress profile and the types of the endecasílabo; the rhyme in full (consonante) and by assonance (asonante) with its scheme, the strophes and the form of a poem. No model and no dictionary are needed.
525
+
526
+ ```python
527
+ >>> from ests import VerseStats
528
+
529
+ >>> text = """Cuando me paro a contemplar mi estado
530
+ ... y a ver los pasos por do me han traído,
531
+ ... hallo, según por do anduve perdido,
532
+ ... que a mayor mal pudiera haber llegado."""
533
+ >>> vs = VerseStats(text)
534
+ >>> vs.meter, vs.patterns[0], vs.c_rhythms
535
+ ('endecasílabo', '---+---+-+-', {'4-8-10': 2, '4-7-10': 1, '3-6-10': 1})
536
+ >>> vs.syllables[0]
537
+ ('cuan', 'do', 'me', 'pa', 'ro‿a', 'con', 'tem', 'plar', 'mi‿es', 'ta', 'do')
538
+ >>> vs.rhyme_schemes, vs.strophes
539
+ (('ABBA',), ('cuarteto',))
540
+ ```
541
+
542
+ More in the [documentation](https://sergeyshk.github.io/esTS/stats/verse_stats/).
543
+
544
+ </details>
545
+
480
546
  <details>
481
547
  <summary><b>spaCy components</b></summary>
482
548
 
@@ -500,7 +566,7 @@ Every statistics class is also a component of a pipeline, so a text is annotated
500
566
  (12, ('DET', 'NOUN'), 2.0)
501
567
  ```
502
568
 
503
- The factories are `ests_basic`, `ests_readability`, `ests_diversity`, `ests_morph`, `ests_syntax`, `ests_cohesion` and `ests_lexical`; the name of the pipe is free and is what the extension is called.
569
+ The factories are `ests_basic`, `ests_readability`, `ests_diversity`, `ests_morph`, `ests_syntax`, `ests_cohesion`, `ests_lexical`, `ests_style`, `ests_phon` and `ests_verse`; the name of the pipe is the name of the extension.
504
570
 
505
571
  More in the [documentation](https://sergeyshk.github.io/esTS/components/).
506
572
 
@@ -511,14 +577,14 @@ More in the [documentation](https://sergeyshk.github.io/esTS/components/).
511
577
 
512
578
  <br>
513
579
 
514
- The library compares corpora and describes the use of a word, with the measures of corpus linguistics:
580
+ The measures of corpus linguistics that compare corpora and describe the use of a word:
515
581
 
516
582
  * keywords of a target corpus against a reference one or against the frequency dictionary of Google Books Ngram: the log-likelihood with its p-value, Log Ratio, chi-square, %DIFF, BIC, ELL and the odds ratio
517
- * collocations by logDice, MI, MI³, t-score, Dice, log-likelihood, NPMI and minimum sensitivity, checked against NLTK
583
+ * collocations by logDice, MI, MI³, t-score, Dice, log-likelihood, NPMI and minimum sensitivity
518
584
  * the dispersion of a word over the parts of a text: DP of Gries, normalized DP, Juilland's D, Carroll's D2, Rosengren's S and the Kullback-Leibler divergence
519
585
  * a KWIC concordance by word form or by lemma
520
586
  * stylometry: the distances between texts by Burrows's Delta and its variants, with the attribution of a text to reference authors, the markers of preferred and avoided words by Zeta, Kilgarriff's chi-square, the Mendenhall curve and the profile of the function words
521
- * the comparison of two corpora by 132 features of a text over windows of about the same size: Cliff's delta, Cohen's d and the AUC of every feature, the Mann-Whitney test with Holm's correction and a bootstrap interval of the difference of the medians that resamples whole texts
587
+ * the comparison of two corpora by 132 features of a text over windows of about the same size: Cliff's delta, Cohen's d and the AUC of every feature, the Mann-Whitney test with Holm's correction and a bootstrap interval of the difference of the medians
522
588
 
523
589
  ```python
524
590
  >>> from ests import WordsExtractor
@@ -555,10 +621,11 @@ More in the [documentation](https://sergeyshk.github.io/esTS/corpus/keyness/).
555
621
 
556
622
  <br>
557
623
 
558
- * [spanish_literature](https://sergeyshk.github.io/esTS/datasets/spanishliterature/) - Spanish-language literature in the public domain: 150 works by 33 authors from Spain, Latin America and the Philippines, from Cervantes to the 1920s, in prose, poems, drama and publicism; 65 million characters
559
- * [freq_dict](https://sergeyshk.github.io/esTS/datasets/freqdict/) - a frequency dictionary of 83,785 Spanish lemmas from the books of Google Books Ngram of 1980-2019 (63 billion words): ipm, range and Juilland's D over the years, the number of books and the part of speech (CC BY 3.0)
624
+ * [spanish_literature](https://sergeyshk.github.io/esTS/datasets/spanishliterature/) - Spanish-language literature in the public domain: 150 works by 33 authors from Spain, Latin America and the Philippines, from Cervantes to the 1920s, in prose, poems, drama and publicism
625
+ * [spanish_sonnets](https://sergeyshk.github.io/esTS/datasets/spanishsonnets/) - Spanish sonnets of the Diachronic Spanish Sonnet Corpus (DISCO) in the public domain: 4,259 sonnets by 1,167 authors of the 15th-20th centuries, every line with its metrical pattern and the label of its rhyme, the automatic annotation of DISCO (CC BY 4.0)
626
+ * [freq_dict](https://sergeyshk.github.io/esTS/datasets/freqdict/) - a frequency dictionary of 83,785 Spanish lemmas from the books of Google Books Ngram of 1980-2019: ipm, range and Juilland's D over the years, the number of books and the part of speech (CC BY 3.0)
560
627
 
561
- The texts are cut to the text of the author, without title pages, notes of the transcribers and the editors, tables of contents and footnotes, and come with the genre, the author, the title, the years of the first publication and the country; the records can be filtered by any of them and by the length of the text.
628
+ The texts are cut to the text of the author and come with the genre, the author, the title, the years of the first publication and the country; the records can be filtered by any of them and by the length of the text.
562
629
 
563
630
  ```python
564
631
  >>> from ests.datasets import SpanishLiterature
@@ -573,7 +640,7 @@ La de Bringas 1884 413730
573
640
  Tormento 1884 477286
574
641
  ```
575
642
 
576
- The archive (19 MB) is downloaded once by `download()` into the data directory and verified against its SHA-256 checksum; before the download `get_texts()` and `get_records()` raise `DatasetNotFoundError` with a hint.
643
+ The archive (19 MB) is downloaded once by `download()` into the data directory; before that `get_texts()` and `get_records()` raise `DatasetNotFoundError`.
577
644
 
578
645
  More in the [documentation](https://sergeyshk.github.io/esTS/datasets/spanishliterature/).
579
646
 
@@ -585,13 +652,13 @@ More in the [documentation](https://sergeyshk.github.io/esTS/datasets/spanishlit
585
652
  <br>
586
653
 
587
654
  * [Zipf's law](https://sergeyshk.github.io/esTS/visualizers/zipf/) with the theoretical curve and the Zipf-Mandelbrot fit
588
- * [Literature fingerprinting](https://sergeyshk.github.io/esTS/visualizers/fingerprinting/) (Literature Fingerprinting)
589
- * [Word tree](https://sergeyshk.github.io/esTS/visualizers/word_tree/) (Word Tree)
655
+ * [Literature fingerprinting](https://sergeyshk.github.io/esTS/visualizers/fingerprinting/)
656
+ * [Word tree](https://sergeyshk.github.io/esTS/visualizers/word_tree/)
590
657
  * [Corpus plots](https://sergeyshk.github.io/esTS/visualizers/corpus/): lexical dispersion, a chart of keywords, a network of collocations
591
658
  * [Stylometric plots](https://sergeyshk.github.io/esTS/visualizers/stylometry/): a dendrogram, the principal components and the multidimensional scaling by Delta, the Mendenhall curves
592
659
  * [Vocabulary growth and frequency spectrum](https://sergeyshk.github.io/esTS/visualizers/vocabulary/), [sentence lengths](https://sergeyshk.github.io/esTS/visualizers/sentences/) with a moving average
593
660
 
594
- The matplotlib plots take the axes `ax` and return `Axes`, so they can be laid out on one figure; the network of collocations and the word tree are graphs of graphviz, whose executables render them. Cosine Delta separates three novels by Galdós from three by Unamuno:
661
+ The matplotlib plots take the axes `ax` and return `Axes`; the network of collocations and the word tree are rendered by the executables of Graphviz. Cosine Delta separates three novels by Galdós from three by Unamuno:
595
662
 
596
663
  ```python
597
664
  import matplotlib.pyplot as plt
@@ -651,7 +718,7 @@ Run `make help` for the full list of commands.
651
718
 
652
719
  The documentation is bilingual: English pages are `docs/*.md`, Spanish ones are `docs/*.es.md` next to them ([mkdocs-static-i18n](https://github.com/ultrabug/mkdocs-static-i18n)); when editing a page, update both versions.
653
720
 
654
- The installed version is `ests.__version__`. All exceptions inherit `ests.EstsError` and one of the built-in classes (`SourceError`, `ParameterError` and `DataFileError` - `ValueError`, `SourceTypeError` - `TypeError`, `UnknownStatError` - `KeyError`, `DatasetNotFoundError` - `OSError`, `DownloadError` - `RuntimeError`), so `except ValueError` keeps working. The library prints nothing on its own: its messages go to the `ests` logger (`logging.getLogger("ests")`) and are silent by default.
721
+ The installed version is `ests.__version__`. All exceptions inherit `ests.EstsError` and a built-in class (`ValueError`, `TypeError`, `KeyError`, `OSError` or `RuntimeError`), so `except ValueError` keeps working. The library prints nothing on its own: its messages go to the `ests` logger and are silent by default. More in the [documentation](https://sergeyshk.github.io/esTS/exceptions/).
655
722
 
656
723
  Before submitting changes, install the hooks that run the linters on commit and the tests on push:
657
724
 
@@ -661,7 +728,7 @@ uv run pre-commit install
661
728
 
662
729
  ## Contributing
663
730
 
664
- Bug reports, ideas and pull requests are welcome - [issues](https://github.com/SergeyShk/esTS/issues) are open. The workflow, the checks to run before submitting a pull request and how to shape the changes are described in [CONTRIBUTING.md](https://github.com/SergeyShk/esTS/blob/master/CONTRIBUTING.md); the rules of conduct are in the [code of conduct](https://github.com/SergeyShk/esTS/blob/master/CODE_OF_CONDUCT.md).
731
+ Bug reports, ideas and pull requests are welcome in the [issues](https://github.com/SergeyShk/esTS/issues). The workflow and the checks to run before a pull request are in [CONTRIBUTING.md](https://github.com/SergeyShk/esTS/blob/master/CONTRIBUTING.md); the rules of conduct are in the [code of conduct](https://github.com/SergeyShk/esTS/blob/master/CODE_OF_CONDUCT.md).
665
732
 
666
733
  <details>
667
734
  <summary><b>Project structure</b></summary>
@@ -675,17 +742,20 @@ Bug reports, ideas and pull requests are welcome - [issues](https://github.com/S
675
742
  * lexical_stats.py - lexical sophistication statistics
676
743
  * components.py - components of a spaCy pipeline
677
744
  * corpus - measures of corpus linguistics: keywords, collocations, dispersion, concordance, stylometry, comparison of corpora
678
- * datasets - datasets: Spanish-language literature, the frequency dictionary
745
+ * datasets - datasets: Spanish-language literature, Spanish sonnets, the frequency dictionary
679
746
  * constants.py - constants of the Spanish language and of the metrics
680
747
  * diversity_stats.py - lexical diversity metrics
681
748
  * exceptions.py - library exceptions
682
749
  * extractors.py - tools for object extraction from a text
683
750
  * morph_stats.py - morphological statistics
684
751
  * readability_stats.py - readability metrics
752
+ * style_stats.py - style metrics
753
+ * phon_stats.py - phonostatistics
685
754
  * syntax_stats.py - syntactic statistics
755
+ * verse_stats.py - verse statistics
686
756
  * syllables.py - syllabification and stress
687
757
  * utils.py - helper tools
688
- * visualizers - plots: Zipf's law, fingerprinting, word tree, corpus and stylometric plots, vocabulary growth, sentence lengths
758
+ * visualizers - plots: Zipf's law, fingerprinting, word tree, corpus and stylometric plots, vocabulary growth, sentence lengths, text highlighting
689
759
  * **scripts** - scripts that build the archives of the datasets
690
760
  * **tests** - tests mirroring the package structure
691
761
 
@@ -701,7 +771,7 @@ Bug reports, ideas and pull requests are welcome - [issues](https://github.com/S
701
771
 
702
772
  ## Citation
703
773
 
704
- Please use the following BibTeX entry for citing **esTS** if you use it in your research or software. Citations are helpful for the continued development and maintenance of this library. The same metadata is in [CITATION.cff](https://github.com/SergeyShk/esTS/blob/master/CITATION.cff) - GitHub shows it under the "Cite this repository" button. The Concept DOI [10.5281/zenodo.22924655](https://doi.org/10.5281/zenodo.22924655) on Zenodo points at every version of the library; the DOI of a single version is on the page of its release.
774
+ Please use the following BibTeX entry for citing **esTS** if you use it in your research or software. The same metadata is in [CITATION.cff](https://github.com/SergeyShk/esTS/blob/master/CITATION.cff) ("Cite this repository" on GitHub). The Concept DOI [10.5281/zenodo.22924655](https://doi.org/10.5281/zenodo.22924655) points at every version of the library; the DOI of a single version is on the page of its release.
705
775
 
706
776
  ```bibtex
707
777
  @software{esTS,