pyests 0.3.0__tar.gz → 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {pyests-0.3.0 → pyests-0.4.0}/PKG-INFO +108 -38
- {pyests-0.3.0 → pyests-0.4.0}/README.es.md +107 -37
- {pyests-0.3.0 → pyests-0.4.0}/README.md +107 -37
- {pyests-0.3.0 → pyests-0.4.0}/ests/__init__.py +12 -2
- {pyests-0.3.0 → pyests-0.4.0}/ests/basic_stats.py +23 -39
- {pyests-0.3.0 → pyests-0.4.0}/ests/cohesion_stats.py +50 -99
- {pyests-0.3.0 → pyests-0.4.0}/ests/components.py +226 -74
- {pyests-0.3.0 → pyests-0.4.0}/ests/constants.py +476 -91
- {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/collocations.py +8 -14
- {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/compare.py +54 -113
- {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/dispersion.py +4 -8
- {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/keyness.py +24 -40
- {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/kwic.py +9 -13
- {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/stylometry.py +31 -62
- pyests-0.4.0/ests/datasets/__init__.py +5 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/datasets/dataset.py +3 -7
- {pyests-0.3.0 → pyests-0.4.0}/ests/datasets/freq_dict.py +26 -59
- {pyests-0.3.0 → pyests-0.4.0}/ests/datasets/spanish_literature.py +12 -16
- pyests-0.4.0/ests/datasets/spanish_sonnets.py +301 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/diversity_stats.py +61 -144
- pyests-0.4.0/ests/exceptions.py +88 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/extractors.py +12 -25
- {pyests-0.3.0 → pyests-0.4.0}/ests/lexical_stats.py +20 -36
- {pyests-0.3.0 → pyests-0.4.0}/ests/morph_stats.py +23 -50
- pyests-0.4.0/ests/phon_stats.py +584 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/readability_stats.py +41 -79
- pyests-0.4.0/ests/style_stats.py +649 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/syllables.py +48 -55
- {pyests-0.3.0 → pyests-0.4.0}/ests/syntax_stats.py +51 -97
- {pyests-0.3.0 → pyests-0.4.0}/ests/utils.py +101 -121
- pyests-0.4.0/ests/verse_stats.py +1226 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/__init__.py +4 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/corpus.py +11 -20
- {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/fingerprinting.py +9 -15
- pyests-0.4.0/ests/visualizers/highlight.py +1063 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/sentences.py +5 -8
- {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/stylometry.py +8 -13
- {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/vocabulary.py +2 -2
- {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/word_tree.py +7 -14
- {pyests-0.3.0 → pyests-0.4.0}/ests/visualizers/zipf.py +5 -7
- {pyests-0.3.0 → pyests-0.4.0}/pyproject.toml +4 -1
- pyests-0.3.0/ests/datasets/__init__.py +0 -4
- pyests-0.3.0/ests/exceptions.py +0 -93
- {pyests-0.3.0 → pyests-0.4.0}/.gitignore +0 -0
- {pyests-0.3.0 → pyests-0.4.0}/LICENSE.txt +0 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/corpus/__init__.py +0 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/py.typed +0 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/resources/README.md +0 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/resources/connectors.tsv +0 -0
- {pyests-0.3.0 → pyests-0.4.0}/ests/resources/google_books_top10000.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: pyests
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.4.0
|
|
4
4
|
Summary: Spanish Texts Statistics
|
|
5
5
|
Project-URL: Homepage, https://github.com/SergeyShk/esTS
|
|
6
6
|
Project-URL: Repository, https://github.com/SergeyShk/esTS
|
|
@@ -68,9 +68,9 @@ Description-Content-Type: text/markdown
|
|
|
68
68
|
|
|
69
69
|
---
|
|
70
70
|
|
|
71
|
-
**esTS** computes
|
|
71
|
+
**esTS** computes statistics of Spanish texts: basic statistics, readability, lexical diversity, lexical sophistication, style, phonostatistics, morphology, syntax and cohesion - by published formulas with the coefficients and the scales of their authors, and by the parts of speech and the features of Universal Dependencies.
|
|
72
72
|
|
|
73
|
-
The library works both with raw strings and with `Doc` objects of [spaCy](https://github.com/explosion/spaCy)
|
|
73
|
+
The library works both with raw strings and with `Doc` objects of [spaCy](https://github.com/explosion/spaCy); most statistics need no trained model (see Installation).
|
|
74
74
|
|
|
75
75
|
* **[Object extraction](https://sergeyshk.github.io/esTS/extractors/sentences/)** - configurable sentence, word and character N-gram tokenizers that know the inverted marks, the dialogue dash and the abbreviations of Spanish
|
|
76
76
|
* **[Syllables and stress](https://sergeyshk.github.io/esTS/syllables/)** - rule-based syllabification and the stressed syllable derived from the spelling, with no dictionary
|
|
@@ -79,15 +79,16 @@ The library works both with raw strings and with `Doc` objects of [spaCy](https:
|
|
|
79
79
|
* **[Lexical diversity metrics](https://sergeyshk.github.io/esTS/stats/diversity_stats/)** - TTR and its variations, MATTR, MSTTR, MTLD, HD-D, Simpson's and Yule's indices, entropy, Zipf's and Heaps' laws
|
|
80
80
|
* **[Morphological statistics](https://sergeyshk.github.io/esTS/stats/morph_stats/)** - parts of speech and fifteen grammatical features of Universal Dependencies, with the markers of Spanish: the moods, the non-finite forms, `ser` against `estar`, the adverbs in `-mente`
|
|
81
81
|
* **[Corpus measures](https://sergeyshk.github.io/esTS/corpus/keyness/)** - keywords against a reference corpus, collocations, the dispersion of a word over the parts of a text, a KWIC concordance and the stylometry of authorship: Burrows's Delta with its variants, Zeta, the Mendenhall curve, the profile of the function words; the comparison of two corpora by 132 features of a text with effect sizes
|
|
82
|
-
* **[Visualizers](https://sergeyshk.github.io/esTS/visualizers/zipf/)** - Zipf's law, literature fingerprinting, a word tree, lexical dispersion and keywords, a network of collocations, a dendrogram, PCA and MDS by Delta, vocabulary growth, sentence lengths
|
|
83
|
-
* **[Datasets](https://sergeyshk.github.io/esTS/datasets/spanishliterature/)** - Spanish-language literature in the public domain: 150 works by 33 authors
|
|
82
|
+
* **[Visualizers](https://sergeyshk.github.io/esTS/visualizers/zipf/)** - Zipf's law, literature fingerprinting, a word tree, lexical dispersion and keywords, a network of collocations, a dendrogram, PCA and MDS by Delta, vocabulary growth, sentence lengths, and the highlighting of the fragments the statistics count, such as long sentences and passives
|
|
83
|
+
* **[Datasets](https://sergeyshk.github.io/esTS/datasets/spanishliterature/)** - Spanish-language literature in the public domain: 150 works by 33 authors in four genres; 4,259 sonnets of the 15th-20th centuries with the metrical pattern and the rhyme of every line; a frequency dictionary of 83,785 lemmas by Google Books Ngram
|
|
84
84
|
* **[spaCy components](https://sergeyshk.github.io/esTS/components/)** - every statistics class as a component of a pipeline, the statistics attached to the `Doc` in one pass
|
|
85
85
|
* **[Cohesion statistics](https://sergeyshk.github.io/esTS/stats/cohesion_stats/)** - the overlap of nouns, arguments and content words between sentences, givenness and temporal cohesion in the manner of Coh-Metrix, with the density of 255 Spanish discourse markers
|
|
86
86
|
* **[Lexical sophistication statistics](https://sergeyshk.github.io/esTS/stats/lexical_stats/)** - how rare the words of a text are in the language: the frequency, range and dispersion of the lemmas by a dictionary of Google Books Ngram, the frequency bands top-1000 to 10000, surprisal, perplexity and lexical density
|
|
87
|
+
* **[Style metrics](https://sergeyshk.github.io/esTS/stats/style_stats/)** - the SEO indicators of Advego and Text.ru (nausea, water content, spam score, naturalness by Zipf's law, keyword density) and the markers of the officialese style by the Spanish guides to plain language: verbal nouns, compound prepositions, parenthetical expressions and clichés
|
|
88
|
+
* **[Phonostatistics](https://sergeyshk.github.io/esTS/stats/phon_stats/)** - the shares of the classes of sounds, consonant clusters, hiatuses, open syllables, hardness and the indices of alliteration and assonance, over the sounds of a rule-based transcription
|
|
89
|
+
* **[Verse statistics](https://sergeyshk.github.io/esTS/stats/verse_stats/)** - the scansion of Spanish verse by its syllabic meter: the metrical syllables, the meter of a poem, the stress profile and the types of the endecasílabo, the rhyme in full and by assonance with its scheme, the strophes and the form of a poem
|
|
87
90
|
* **[Syntactic statistics](https://sergeyshk.github.io/esTS/stats/syntax_stats/)** - the dependency tree by distances, depth, clauses and coordination, with the constructions of the administrative style: the passive with `ser` and with `se`, the participial and the gerund clauses, the chains of `de`, the split predicates
|
|
88
91
|
|
|
89
|
-
Style, phonostatistics, metre and rhyme come in 0.4.
|
|
90
|
-
|
|
91
92
|
## Installation
|
|
92
93
|
|
|
93
94
|
Requires Python 3.11 or newer.
|
|
@@ -102,13 +103,13 @@ Or with [uv](https://docs.astral.sh/uv/):
|
|
|
102
103
|
uv add pyests
|
|
103
104
|
```
|
|
104
105
|
|
|
105
|
-
The distribution on PyPI is `pyests`, the package it installs is `ests`. The basic statistics, the
|
|
106
|
+
The distribution on PyPI is `pyests`, the package it installs is `ests`. The basic statistics, readability, lexical diversity, the style metrics except the verbal nouns, phonostatistics and verse statistics need no spaCy model. The morphological, syntactic, cohesion and lexical sophistication statistics of a string need one, and so do the verbal nouns, the profile of the function words, the features of a text, the comparison of corpora and parsing a `Doc` yourself:
|
|
106
107
|
|
|
107
108
|
```bash
|
|
108
109
|
python -m spacy download es_core_news_sm
|
|
109
110
|
```
|
|
110
111
|
|
|
111
|
-
The statistics by the frequency dictionary need it downloaded once with `FreqDict().download()`. The datasets go to the directory `ests_data` next to the installed package
|
|
112
|
+
The statistics by the frequency dictionary need it downloaded once with `FreqDict().download()`. The datasets go to the directory `ests_data` next to the installed package, or to the one passed in `data_dir` or set in `ESTS_DATA_DIR` before the package is imported.
|
|
112
113
|
|
|
113
114
|
## Quick start
|
|
114
115
|
|
|
@@ -149,7 +150,7 @@ The statistics by the frequency dictionary need it downloaded once with `FreqDic
|
|
|
149
150
|
|
|
150
151
|
### Object extraction
|
|
151
152
|
|
|
152
|
-
|
|
153
|
+
Configurable tools extract sentences, words and character N-grams from a text for the statistics. The sentence splitter knows the inverted marks, the dialogue dash and the abbreviations of Spanish; the word tokenizer keeps clitics, ordinals and Spanish numbers whole, and lemmas come from [simplemma](https://github.com/adbar/simplemma), which needs no model.
|
|
153
154
|
|
|
154
155
|
```python
|
|
155
156
|
>>> from ests import CharNgramsExtractor, SentsExtractor, WordsExtractor
|
|
@@ -172,7 +173,7 @@ More in the [documentation](https://sergeyshk.github.io/esTS/extractors/sentence
|
|
|
172
173
|
|
|
173
174
|
<br>
|
|
174
175
|
|
|
175
|
-
|
|
176
|
+
The syllables and the stress follow from the Spanish spelling, with no dictionary: the written accent, or the ending of the word when there is none, gives the stressed syllable. Adverbs in `-mente` and hyphenated compounds carry two stresses.
|
|
176
177
|
|
|
177
178
|
```python
|
|
178
179
|
>>> from ests.syllables import stress_type, syllabify, word_stress, word_stresses
|
|
@@ -217,7 +218,7 @@ The library allows extracting the following statistics from a text:
|
|
|
217
218
|
* the distribution of words by the number of letters
|
|
218
219
|
* the distribution of words by the number of syllables
|
|
219
220
|
|
|
220
|
-
A complex word has three or more syllables and a long word seven or more letters
|
|
221
|
+
A complex word has three or more syllables and a long word seven or more letters. Any statistic can be printed in a readable form:
|
|
221
222
|
|
|
222
223
|
```python
|
|
223
224
|
>>> from ests import BasicStats
|
|
@@ -260,9 +261,9 @@ The library allows counting the following readability metrics:
|
|
|
260
261
|
* LIX readability measure
|
|
261
262
|
* RIX readability measure
|
|
262
263
|
|
|
263
|
-
|
|
264
|
+
On top of the formulas: the band of a scale for the reading ease and for Legibilidad µ, a consensus grade (the median of the grade formulas), the school stage and the reader age in Spain, and reading time.
|
|
264
265
|
|
|
265
|
-
The coefficients of the Flesch reading ease are selected by
|
|
266
|
+
The coefficients of the Flesch reading ease are selected by `preset`: by default the *fórmula de perspicuidad* of Szigriszt-Pazos with the INFLESZ scale (`general`), or Fernández Huerta with the bands of its author (`classic`).
|
|
266
267
|
|
|
267
268
|
```python
|
|
268
269
|
>>> from pprint import pprint
|
|
@@ -322,7 +323,7 @@ The library allows counting 32 lexical diversity metrics, among them:
|
|
|
322
323
|
* Shannon entropy, evenness and perplexity
|
|
323
324
|
* the slope of Zipf's law, the Zipf-Mandelbrot fit and the exponent of Heaps' law
|
|
324
325
|
|
|
325
|
-
Any metric can be computed over windows of equal length,
|
|
326
|
+
Any metric can be computed over windows of equal length, to compare texts of different lengths, with a confidence interval of the mean.
|
|
326
327
|
|
|
327
328
|
```python
|
|
328
329
|
>>> from ests import DiversityStats
|
|
@@ -352,7 +353,7 @@ More in the [documentation](https://sergeyshk.github.io/esTS/stats/diversity_sta
|
|
|
352
353
|
|
|
353
354
|
<br>
|
|
354
355
|
|
|
355
|
-
The
|
|
356
|
+
The parts of speech and the grammatical features of Universal Dependencies, as the Spanish models of spaCy give them:
|
|
356
357
|
|
|
357
358
|
* the part of speech and fifteen features: case, definiteness, degree, gender, mood, numeral type, number, person, polarity, politeness, possessive, pronoun type, reflexive, tense and verb form
|
|
358
359
|
* the distribution of the words by the values of any feature, and the parse of the text word by word
|
|
@@ -376,7 +377,7 @@ The library annotates a text with the parts of speech and the grammatical featur
|
|
|
376
377
|
0.5
|
|
377
378
|
```
|
|
378
379
|
|
|
379
|
-
|
|
380
|
+
A text is parsed with `es_core_news_sm`; another pipeline can be passed in `nlp`.
|
|
380
381
|
|
|
381
382
|
More in the [documentation](https://sergeyshk.github.io/esTS/stats/morph_stats/).
|
|
382
383
|
|
|
@@ -387,7 +388,7 @@ More in the [documentation](https://sergeyshk.github.io/esTS/stats/morph_stats/)
|
|
|
387
388
|
|
|
388
389
|
<br>
|
|
389
390
|
|
|
390
|
-
The
|
|
391
|
+
The dependency tree of Universal Dependencies and the constructions that the Spanish guides to clear language warn about:
|
|
391
392
|
|
|
392
393
|
* the complexity of the tree: dependency distances, depth, leaves and subtrees, valency of the finite verbs, coordination chains, clauses and subordinate clauses, modifiers per noun
|
|
393
394
|
* the constructions: the passive with `ser` and with `se`, the participial and the gerund clauses, the chains of `de`, the split predicates, the impersonal `se`, the words of negation, the ratio of nouns to verbs
|
|
@@ -412,7 +413,7 @@ The library measures the dependency tree of Universal Dependencies and the const
|
|
|
412
413
|
2.05
|
|
413
414
|
```
|
|
414
415
|
|
|
415
|
-
|
|
416
|
+
A text is parsed with `es_core_news_sm`; another pipeline with a parser can be passed in `nlp`.
|
|
416
417
|
|
|
417
418
|
More in the [documentation](https://sergeyshk.github.io/esTS/stats/syntax_stats/).
|
|
418
419
|
|
|
@@ -423,7 +424,7 @@ More in the [documentation](https://sergeyshk.github.io/esTS/stats/syntax_stats/
|
|
|
423
424
|
|
|
424
425
|
<br>
|
|
425
426
|
|
|
426
|
-
|
|
427
|
+
Referential cohesion in the manner of Coh-Metrix and its Spanish adaptation Coh-Metrix-Esp:
|
|
427
428
|
|
|
428
429
|
* the overlap of nouns, of arguments and of content words between adjacent sentences and between all pairs of sentences, binary and proportional
|
|
429
430
|
* givenness: pronouns, demonstratives and the content words whose lemma was already used
|
|
@@ -450,7 +451,7 @@ The library measures referential cohesion in the manner of Coh-Metrix and of its
|
|
|
450
451
|
(86.96, 43.48)
|
|
451
452
|
```
|
|
452
453
|
|
|
453
|
-
|
|
454
|
+
A text is parsed with `es_core_news_sm`; another pipeline can be passed in `nlp`.
|
|
454
455
|
|
|
455
456
|
More in the [documentation](https://sergeyshk.github.io/esTS/stats/cohesion_stats/).
|
|
456
457
|
|
|
@@ -461,7 +462,7 @@ More in the [documentation](https://sergeyshk.github.io/esTS/stats/cohesion_stat
|
|
|
461
462
|
|
|
462
463
|
<br>
|
|
463
464
|
|
|
464
|
-
How rare the words of a text are in the language, in the manner of TAALES: the mean frequency, range and dispersion of the lemmas by the frequency dictionary of Google Books Ngram, the shares of the
|
|
465
|
+
How rare the words of a text are in the language, in the manner of TAALES: the mean frequency, range and dispersion of the lemmas by the frequency dictionary of Google Books Ngram, the shares of the frequency bands top-1000, 2000, 5000 and 10000, surprisal and perplexity, the lexical density. The statistics by the dictionary need it downloaded once.
|
|
465
466
|
|
|
466
467
|
```python
|
|
467
468
|
>>> from ests import LexicalStats
|
|
@@ -477,6 +478,71 @@ More in the [documentation](https://sergeyshk.github.io/esTS/stats/lexical_stats
|
|
|
477
478
|
|
|
478
479
|
</details>
|
|
479
480
|
|
|
481
|
+
<details>
|
|
482
|
+
<summary><b>Style metrics</b></summary>
|
|
483
|
+
|
|
484
|
+
<br>
|
|
485
|
+
|
|
486
|
+
The SEO indicators of Advego and Text.ru - nausea, water content, spam score, naturalness by Zipf's law, keyword density - and the markers of the officialese style that the Spanish guides to plain language warn about: the nouns derived from a verb, the compound prepositions of the administrative style, the parenthetical expressions and the clichés.
|
|
487
|
+
|
|
488
|
+
```python
|
|
489
|
+
>>> from ests import StyleStats
|
|
490
|
+
|
|
491
|
+
>>> ss = StyleStats("Se procedió a la revisión del expediente en el marco del plan a la mayor brevedad.")
|
|
492
|
+
>>> ss.compound_prepositions, ss.cliches, ss.verbal_nouns
|
|
493
|
+
(6.25, 12.5, 20.0)
|
|
494
|
+
```
|
|
495
|
+
|
|
496
|
+
More in the [documentation](https://sergeyshk.github.io/esTS/stats/style_stats/).
|
|
497
|
+
|
|
498
|
+
</details>
|
|
499
|
+
|
|
500
|
+
<details>
|
|
501
|
+
<summary><b>Phonostatistics</b></summary>
|
|
502
|
+
|
|
503
|
+
<br>
|
|
504
|
+
|
|
505
|
+
The shares of the classes of sounds, the consonant clusters, the hiatuses, the open syllables, the hardness and the indices of alliteration and assonance, counted over the sounds of a rule-based transcription, not over the letters: `h` is silent, `ll` and `rr` are one sound.
|
|
506
|
+
|
|
507
|
+
```python
|
|
508
|
+
>>> from ests import PhonStats
|
|
509
|
+
|
|
510
|
+
>>> ps = PhonStats("Los suspiros se escapan de su boca de fresa")
|
|
511
|
+
>>> round(ps.p_voiceless, 3), round(ps.hardness, 3), ps.sounds[-1]
|
|
512
|
+
(0.371, 0.684, ('f', 'r', 'e', 's', 'a'))
|
|
513
|
+
```
|
|
514
|
+
|
|
515
|
+
More in the [documentation](https://sergeyshk.github.io/esTS/stats/phon_stats/).
|
|
516
|
+
|
|
517
|
+
</details>
|
|
518
|
+
|
|
519
|
+
<details>
|
|
520
|
+
<summary><b>Verse statistics</b></summary>
|
|
521
|
+
|
|
522
|
+
<br>
|
|
523
|
+
|
|
524
|
+
The scansion of Spanish verse by its syllabic meter: the metrical syllables of a line with the synalepha and the law of the final stress, fitted to the meter of the poem; the meter, the stress profile and the types of the endecasílabo; the rhyme in full (consonante) and by assonance (asonante) with its scheme, the strophes and the form of a poem. No model and no dictionary are needed.
|
|
525
|
+
|
|
526
|
+
```python
|
|
527
|
+
>>> from ests import VerseStats
|
|
528
|
+
|
|
529
|
+
>>> text = """Cuando me paro a contemplar mi estado
|
|
530
|
+
... y a ver los pasos por do me han traído,
|
|
531
|
+
... hallo, según por do anduve perdido,
|
|
532
|
+
... que a mayor mal pudiera haber llegado."""
|
|
533
|
+
>>> vs = VerseStats(text)
|
|
534
|
+
>>> vs.meter, vs.patterns[0], vs.c_rhythms
|
|
535
|
+
('endecasílabo', '---+---+-+-', {'4-8-10': 2, '4-7-10': 1, '3-6-10': 1})
|
|
536
|
+
>>> vs.syllables[0]
|
|
537
|
+
('cuan', 'do', 'me', 'pa', 'ro‿a', 'con', 'tem', 'plar', 'mi‿es', 'ta', 'do')
|
|
538
|
+
>>> vs.rhyme_schemes, vs.strophes
|
|
539
|
+
(('ABBA',), ('cuarteto',))
|
|
540
|
+
```
|
|
541
|
+
|
|
542
|
+
More in the [documentation](https://sergeyshk.github.io/esTS/stats/verse_stats/).
|
|
543
|
+
|
|
544
|
+
</details>
|
|
545
|
+
|
|
480
546
|
<details>
|
|
481
547
|
<summary><b>spaCy components</b></summary>
|
|
482
548
|
|
|
@@ -500,7 +566,7 @@ Every statistics class is also a component of a pipeline, so a text is annotated
|
|
|
500
566
|
(12, ('DET', 'NOUN'), 2.0)
|
|
501
567
|
```
|
|
502
568
|
|
|
503
|
-
The factories are `ests_basic`, `ests_readability`, `ests_diversity`, `ests_morph`, `ests_syntax`, `ests_cohesion` and `
|
|
569
|
+
The factories are `ests_basic`, `ests_readability`, `ests_diversity`, `ests_morph`, `ests_syntax`, `ests_cohesion`, `ests_lexical`, `ests_style`, `ests_phon` and `ests_verse`; the name of the pipe is the name of the extension.
|
|
504
570
|
|
|
505
571
|
More in the [documentation](https://sergeyshk.github.io/esTS/components/).
|
|
506
572
|
|
|
@@ -511,14 +577,14 @@ More in the [documentation](https://sergeyshk.github.io/esTS/components/).
|
|
|
511
577
|
|
|
512
578
|
<br>
|
|
513
579
|
|
|
514
|
-
The
|
|
580
|
+
The measures of corpus linguistics that compare corpora and describe the use of a word:
|
|
515
581
|
|
|
516
582
|
* keywords of a target corpus against a reference one or against the frequency dictionary of Google Books Ngram: the log-likelihood with its p-value, Log Ratio, chi-square, %DIFF, BIC, ELL and the odds ratio
|
|
517
|
-
* collocations by logDice, MI, MI³, t-score, Dice, log-likelihood, NPMI and minimum sensitivity
|
|
583
|
+
* collocations by logDice, MI, MI³, t-score, Dice, log-likelihood, NPMI and minimum sensitivity
|
|
518
584
|
* the dispersion of a word over the parts of a text: DP of Gries, normalized DP, Juilland's D, Carroll's D2, Rosengren's S and the Kullback-Leibler divergence
|
|
519
585
|
* a KWIC concordance by word form or by lemma
|
|
520
586
|
* stylometry: the distances between texts by Burrows's Delta and its variants, with the attribution of a text to reference authors, the markers of preferred and avoided words by Zeta, Kilgarriff's chi-square, the Mendenhall curve and the profile of the function words
|
|
521
|
-
* the comparison of two corpora by 132 features of a text over windows of about the same size: Cliff's delta, Cohen's d and the AUC of every feature, the Mann-Whitney test with Holm's correction and a bootstrap interval of the difference of the medians
|
|
587
|
+
* the comparison of two corpora by 132 features of a text over windows of about the same size: Cliff's delta, Cohen's d and the AUC of every feature, the Mann-Whitney test with Holm's correction and a bootstrap interval of the difference of the medians
|
|
522
588
|
|
|
523
589
|
```python
|
|
524
590
|
>>> from ests import WordsExtractor
|
|
@@ -555,10 +621,11 @@ More in the [documentation](https://sergeyshk.github.io/esTS/corpus/keyness/).
|
|
|
555
621
|
|
|
556
622
|
<br>
|
|
557
623
|
|
|
558
|
-
* [spanish_literature](https://sergeyshk.github.io/esTS/datasets/spanishliterature/) - Spanish-language literature in the public domain: 150 works by 33 authors from Spain, Latin America and the Philippines, from Cervantes to the 1920s, in prose, poems, drama and publicism
|
|
559
|
-
* [
|
|
624
|
+
* [spanish_literature](https://sergeyshk.github.io/esTS/datasets/spanishliterature/) - Spanish-language literature in the public domain: 150 works by 33 authors from Spain, Latin America and the Philippines, from Cervantes to the 1920s, in prose, poems, drama and publicism
|
|
625
|
+
* [spanish_sonnets](https://sergeyshk.github.io/esTS/datasets/spanishsonnets/) - Spanish sonnets of the Diachronic Spanish Sonnet Corpus (DISCO) in the public domain: 4,259 sonnets by 1,167 authors of the 15th-20th centuries, every line with its metrical pattern and the label of its rhyme, the automatic annotation of DISCO (CC BY 4.0)
|
|
626
|
+
* [freq_dict](https://sergeyshk.github.io/esTS/datasets/freqdict/) - a frequency dictionary of 83,785 Spanish lemmas from the books of Google Books Ngram of 1980-2019: ipm, range and Juilland's D over the years, the number of books and the part of speech (CC BY 3.0)
|
|
560
627
|
|
|
561
|
-
The texts are cut to the text of the author
|
|
628
|
+
The texts are cut to the text of the author and come with the genre, the author, the title, the years of the first publication and the country; the records can be filtered by any of them and by the length of the text.
|
|
562
629
|
|
|
563
630
|
```python
|
|
564
631
|
>>> from ests.datasets import SpanishLiterature
|
|
@@ -573,7 +640,7 @@ La de Bringas 1884 413730
|
|
|
573
640
|
Tormento 1884 477286
|
|
574
641
|
```
|
|
575
642
|
|
|
576
|
-
The archive (19 MB) is downloaded once by `download()` into the data directory
|
|
643
|
+
The archive (19 MB) is downloaded once by `download()` into the data directory; before that `get_texts()` and `get_records()` raise `DatasetNotFoundError`.
|
|
577
644
|
|
|
578
645
|
More in the [documentation](https://sergeyshk.github.io/esTS/datasets/spanishliterature/).
|
|
579
646
|
|
|
@@ -585,13 +652,13 @@ More in the [documentation](https://sergeyshk.github.io/esTS/datasets/spanishlit
|
|
|
585
652
|
<br>
|
|
586
653
|
|
|
587
654
|
* [Zipf's law](https://sergeyshk.github.io/esTS/visualizers/zipf/) with the theoretical curve and the Zipf-Mandelbrot fit
|
|
588
|
-
* [Literature fingerprinting](https://sergeyshk.github.io/esTS/visualizers/fingerprinting/)
|
|
589
|
-
* [Word tree](https://sergeyshk.github.io/esTS/visualizers/word_tree/)
|
|
655
|
+
* [Literature fingerprinting](https://sergeyshk.github.io/esTS/visualizers/fingerprinting/)
|
|
656
|
+
* [Word tree](https://sergeyshk.github.io/esTS/visualizers/word_tree/)
|
|
590
657
|
* [Corpus plots](https://sergeyshk.github.io/esTS/visualizers/corpus/): lexical dispersion, a chart of keywords, a network of collocations
|
|
591
658
|
* [Stylometric plots](https://sergeyshk.github.io/esTS/visualizers/stylometry/): a dendrogram, the principal components and the multidimensional scaling by Delta, the Mendenhall curves
|
|
592
659
|
* [Vocabulary growth and frequency spectrum](https://sergeyshk.github.io/esTS/visualizers/vocabulary/), [sentence lengths](https://sergeyshk.github.io/esTS/visualizers/sentences/) with a moving average
|
|
593
660
|
|
|
594
|
-
The matplotlib plots take the axes `ax` and return `Axes
|
|
661
|
+
The matplotlib plots take the axes `ax` and return `Axes`; the network of collocations and the word tree are rendered by the executables of Graphviz. Cosine Delta separates three novels by Galdós from three by Unamuno:
|
|
595
662
|
|
|
596
663
|
```python
|
|
597
664
|
import matplotlib.pyplot as plt
|
|
@@ -651,7 +718,7 @@ Run `make help` for the full list of commands.
|
|
|
651
718
|
|
|
652
719
|
The documentation is bilingual: English pages are `docs/*.md`, Spanish ones are `docs/*.es.md` next to them ([mkdocs-static-i18n](https://github.com/ultrabug/mkdocs-static-i18n)); when editing a page, update both versions.
|
|
653
720
|
|
|
654
|
-
The installed version is `ests.__version__`. All exceptions inherit `ests.EstsError` and
|
|
721
|
+
The installed version is `ests.__version__`. All exceptions inherit `ests.EstsError` and a built-in class (`ValueError`, `TypeError`, `KeyError`, `OSError` or `RuntimeError`), so `except ValueError` keeps working. The library prints nothing on its own: its messages go to the `ests` logger and are silent by default. More in the [documentation](https://sergeyshk.github.io/esTS/exceptions/).
|
|
655
722
|
|
|
656
723
|
Before submitting changes, install the hooks that run the linters on commit and the tests on push:
|
|
657
724
|
|
|
@@ -661,7 +728,7 @@ uv run pre-commit install
|
|
|
661
728
|
|
|
662
729
|
## Contributing
|
|
663
730
|
|
|
664
|
-
Bug reports, ideas and pull requests are welcome
|
|
731
|
+
Bug reports, ideas and pull requests are welcome in the [issues](https://github.com/SergeyShk/esTS/issues). The workflow and the checks to run before a pull request are in [CONTRIBUTING.md](https://github.com/SergeyShk/esTS/blob/master/CONTRIBUTING.md); the rules of conduct are in the [code of conduct](https://github.com/SergeyShk/esTS/blob/master/CODE_OF_CONDUCT.md).
|
|
665
732
|
|
|
666
733
|
<details>
|
|
667
734
|
<summary><b>Project structure</b></summary>
|
|
@@ -675,17 +742,20 @@ Bug reports, ideas and pull requests are welcome - [issues](https://github.com/S
|
|
|
675
742
|
* lexical_stats.py - lexical sophistication statistics
|
|
676
743
|
* components.py - components of a spaCy pipeline
|
|
677
744
|
* corpus - measures of corpus linguistics: keywords, collocations, dispersion, concordance, stylometry, comparison of corpora
|
|
678
|
-
* datasets - datasets: Spanish-language literature, the frequency dictionary
|
|
745
|
+
* datasets - datasets: Spanish-language literature, Spanish sonnets, the frequency dictionary
|
|
679
746
|
* constants.py - constants of the Spanish language and of the metrics
|
|
680
747
|
* diversity_stats.py - lexical diversity metrics
|
|
681
748
|
* exceptions.py - library exceptions
|
|
682
749
|
* extractors.py - tools for object extraction from a text
|
|
683
750
|
* morph_stats.py - morphological statistics
|
|
684
751
|
* readability_stats.py - readability metrics
|
|
752
|
+
* style_stats.py - style metrics
|
|
753
|
+
* phon_stats.py - phonostatistics
|
|
685
754
|
* syntax_stats.py - syntactic statistics
|
|
755
|
+
* verse_stats.py - verse statistics
|
|
686
756
|
* syllables.py - syllabification and stress
|
|
687
757
|
* utils.py - helper tools
|
|
688
|
-
* visualizers - plots: Zipf's law, fingerprinting, word tree, corpus and stylometric plots, vocabulary growth, sentence lengths
|
|
758
|
+
* visualizers - plots: Zipf's law, fingerprinting, word tree, corpus and stylometric plots, vocabulary growth, sentence lengths, text highlighting
|
|
689
759
|
* **scripts** - scripts that build the archives of the datasets
|
|
690
760
|
* **tests** - tests mirroring the package structure
|
|
691
761
|
|
|
@@ -701,7 +771,7 @@ Bug reports, ideas and pull requests are welcome - [issues](https://github.com/S
|
|
|
701
771
|
|
|
702
772
|
## Citation
|
|
703
773
|
|
|
704
|
-
Please use the following BibTeX entry for citing **esTS** if you use it in your research or software.
|
|
774
|
+
Please use the following BibTeX entry for citing **esTS** if you use it in your research or software. The same metadata is in [CITATION.cff](https://github.com/SergeyShk/esTS/blob/master/CITATION.cff) ("Cite this repository" on GitHub). The Concept DOI [10.5281/zenodo.22924655](https://doi.org/10.5281/zenodo.22924655) points at every version of the library; the DOI of a single version is on the page of its release.
|
|
705
775
|
|
|
706
776
|
```bibtex
|
|
707
777
|
@software{esTS,
|