slopometer 0.0.2__tar.gz → 0.0.4__tar.gz

slopometer-0.0.4/PKG-INFO

@@ -0,0 +1,84 @@
+Metadata-Version: 2.4
+Name: slopometer
+Version: 0.0.4
+Summary: Score prose for simplicity and precision against plain-English reference rules
+Author-email: Jeremy Howard <github@jhoward.fastmail.fm>
+License-Expression: Apache-2.0
+Project-URL: Repository, https://github.com/AnswerDotAI/slopometer
+Project-URL: Documentation, https://AnswerDotAI.github.io/slopometer/
+Keywords: nbdev
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastcore
+Requires-Dist: spacy
+Requires-Dist: mdhtml>=0.1.40
+Requires-Dist: httpx2
+Requires-Dist: warmpy>=0.1.1
+Requires-Dist: rgapi>=0.1.27
+Dynamic: license-file
+
+# slopometer
+
+
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
+
+`slopometer` measures prose against a range of reference-prose rules. The rules merge GOV.UK/GDS house style with the discipline of ASD-STE100, including 26 numbered tells, banned-word lists, and a reference register. It also has rules covering a range of best practices for clear technical communication, including paragraph length, referant tracking, and restatement.
+
+`slopometer` is for anyone shipping READMEs, docstrings, PR text, or agent responses. It catches slop mechanically, in milliseconds, with no LLM in the loop. Every finding cites a rule, quotes a span, and carries a weight.
+
+## Install
+
+    pip install slopometer
+
+The first score on a machine downloads spaCy’s `en_core_web_md` model (about 40MB, once) into `~/.cache/slopometer` and loads it by path. The model never enters a virtual environment. One copy serves every project. Environment syncs cannot remove it.
+
+## Use
+
+``` python
+from slopometer.score import score_text, score_path
+```
+
+The default minimum is 150 scored words. Shorter inputs return `too short to meter` with their word count and no numeric score. Set `min_words=0` to meter short examples:
+
+``` python
+score_text("This section describes our approach. It isn't just a linter - it's a comprehensive paradigm for quality.", min_words=0)
+```
+
+    density 194.1 (weight 33 on 17 prose words), worst 10
+    1: [10] notxbuty (tell 16, not-X-but-Y): "isn't just a"
+    1: [10] banned: 'comprehensive' -> 'complete'
+    1: [10] banned: 'paradigm'
+    1: [3] throat_clearing (tell 13, throat-clearing): 'This section describes'
+
+Density is weighted findings per 100 words across paragraphs, headings, and list items. Heading and list markers do not count as words. Code blocks and other content excluded from scoring do not enter the denominator or count toward the minimum.
+
+Each em dash adds 4 to the total weight. Semicolons joining independent clauses also add 4. Single and double hyphens add no weight on their own. Semicolons in lists of noun phrases or inside parentheses do not score. A comma and “and” before a clause with its own subject add 3. A sentence with three or more sections adds 1 for each section beyond two. A section ends at a comma, semicolon, colon, dash, spaced hyphen, or opening parenthesis. List items stay in one section.
+
+[`score_text`](https://AnswerDotAI.github.io/slopometer/score.html#score_text), [`score_path`](https://AnswerDotAI.github.io/slopometer/score.html#score_path), [`score_paths`](https://AnswerDotAI.github.io/slopometer/score.html#score_paths), and [`score_many`](https://AnswerDotAI.github.io/slopometer/score.html#score_many) accept `min_words`, defaulting to 150. A short result has `too_short=True`, an empty findings list, and `None` for `density`, `total`, and `worst`.
+
+[`score_path`](https://AnswerDotAI.github.io/slopometer/score.html#score_path) scores Markdown files, the Markdown cells of an `.ipynb` notebook, or the module docstring of a `.py` file. Notebook cells are joined with blank lines and scored as one document; code cells, outputs, and raw cells are excluded. File reports carry `lineno|hash|` addresses, prefixed with the cell ID for notebooks. For a `.py` file, addresses name lines of that file. `Result.location(finding)` returns the source location. [`score_paths`](https://AnswerDotAI.github.io/slopometer/score.html#score_paths) scores several files and directories in parallel, one result per file. A directory contributes its `.md`, `.qmd`, `.ipynb` and `.py` files. The command line accepts the same paths, or Markdown on stdin:
+
+    slopometer README.md
+    slopometer nbs/00_core.ipynb
+    slopometer pkg/module.py
+    slopometer nbs/ README.md
+    git log -1 --format=%B | slopometer --min-words 0
+    slopometer draft.md --threshold 10
+    slopometer draft.md --pangram
+
+`--threshold` returns exit code 1 when any file’s score exceeds the limit. Several paths or a directory give one report per file, highest density first. JSON output is a list with one entry per file. Each entry includes its `path`, which is null for stdin. Inputs below `--min-words` print the short-input message and exit successfully without scoring. JSON output marks them with `too_short: true`, includes `min_words`, and uses null scores. Every JSON finding includes a `location` object with one-based `line`, `address`, `end_line` and `end_address`. The end fields locate the finding’s last character. Notebook locations add `cell_id` and zero-based `cell_index`. Finding `start` and `end` offsets refer to the combined Markdown, not notebook JSON.
+
+`--pangram`, [`pangram_text`](https://AnswerDotAI.github.io/slopometer/pangram.html#pangram_text), [`pangram_path`](https://AnswerDotAI.github.io/slopometer/pangram.html#pangram_path) and [`pangram_paths`](https://AnswerDotAI.github.io/slopometer/pangram.html#pangram_paths) score with [Pangram](https://www.pangram.com)’s AI detector instead of the rules. Each finding is a window of text that Pangram scored. Density runs from 0 (human) to 100 (AI). `--threshold` compares against the same scale. Each request sends the text to Pangram’s servers and costs about \$0.05 per 100 words. The API key comes from `PANGRAM_API_KEY`.
+
+The command runs warm through `warmpy`. The first input long enough to score loads the model in a background process. Later calls answer in milliseconds. After thirty idle minutes the process exits.
+
+## What it checks, and what it cannot
+
+The rules live in notebooks that teach each family beside its code. Each rule states its tell, shows a violating example, and shows the plain rewrite. The [lexicon notebook](https://AnswerDotAI.github.io/slopometer/lexicon.html) holds the word and phrase rules. The [syntax notebook](https://AnswerDotAI.github.io/slopometer/syntax.html) builds sentence rules on spaCy’s parse. The [para notebook](https://AnswerDotAI.github.io/slopometer/para.html) covers paragraph and document rules and the limits of word-vector heuristics. The [score notebook](https://AnswerDotAI.github.io/slopometer/score.html) assembles the pipeline. A drift test asserts that every `write_docs` tell maps to a rule or to an explicit unscoreable registry. The meter and the style guide cannot drift apart silently.
+
+Vocabulary novelty is not scored. Identifying unexplained terminology requires audience and document context.
+
+A rule ships only when its false-positive rate on clean reference prose is near zero. The meter scores the style guide’s own clean passage at exactly 0.0. The score notebook measures the blind spot instead of hiding it: mechanically chopped prose passes every surface rule while staying opaque. Agent review (`check_docs`) and Pangram scoring (`--pangram`) cover that residue. Vale, write-good, and proselint solve neighboring problems. The [lexicon notebook](https://AnswerDotAI.github.io/slopometer/lexicon.html) records what came from them. It also credits the [GOV.UK words-to-avoid list](https://www.gov.uk/guidance/style-guide/a-to-z-of-gov-uk-style#words-to-avoid) (OGL v3).

slopometer-0.0.4/README.md

@@ -0,0 +1,62 @@
+# slopometer
+
+
+<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->
+
+`slopometer` measures prose against a range of reference-prose rules. The rules merge GOV.UK/GDS house style with the discipline of ASD-STE100, including 26 numbered tells, banned-word lists, and a reference register. It also has rules covering a range of best practices for clear technical communication, including paragraph length, referant tracking, and restatement.
+
+`slopometer` is for anyone shipping READMEs, docstrings, PR text, or agent responses. It catches slop mechanically, in milliseconds, with no LLM in the loop. Every finding cites a rule, quotes a span, and carries a weight.
+
+## Install
+
+    pip install slopometer
+
+The first score on a machine downloads spaCy’s `en_core_web_md` model (about 40MB, once) into `~/.cache/slopometer` and loads it by path. The model never enters a virtual environment. One copy serves every project. Environment syncs cannot remove it.
+
+## Use
+
+``` python
+from slopometer.score import score_text, score_path
+```
+
+The default minimum is 150 scored words. Shorter inputs return `too short to meter` with their word count and no numeric score. Set `min_words=0` to meter short examples:
+
+``` python
+score_text("This section describes our approach. It isn't just a linter - it's a comprehensive paradigm for quality.", min_words=0)
+```
+
+    density 194.1 (weight 33 on 17 prose words), worst 10
+    1: [10] notxbuty (tell 16, not-X-but-Y): "isn't just a"
+    1: [10] banned: 'comprehensive' -> 'complete'
+    1: [10] banned: 'paradigm'
+    1: [3] throat_clearing (tell 13, throat-clearing): 'This section describes'
+
+Density is weighted findings per 100 words across paragraphs, headings, and list items. Heading and list markers do not count as words. Code blocks and other content excluded from scoring do not enter the denominator or count toward the minimum.
+
+Each em dash adds 4 to the total weight. Semicolons joining independent clauses also add 4. Single and double hyphens add no weight on their own. Semicolons in lists of noun phrases or inside parentheses do not score. A comma and “and” before a clause with its own subject add 3. A sentence with three or more sections adds 1 for each section beyond two. A section ends at a comma, semicolon, colon, dash, spaced hyphen, or opening parenthesis. List items stay in one section.
+
+[`score_text`](https://AnswerDotAI.github.io/slopometer/score.html#score_text), [`score_path`](https://AnswerDotAI.github.io/slopometer/score.html#score_path), [`score_paths`](https://AnswerDotAI.github.io/slopometer/score.html#score_paths), and [`score_many`](https://AnswerDotAI.github.io/slopometer/score.html#score_many) accept `min_words`, defaulting to 150. A short result has `too_short=True`, an empty findings list, and `None` for `density`, `total`, and `worst`.
+
+[`score_path`](https://AnswerDotAI.github.io/slopometer/score.html#score_path) scores Markdown files, the Markdown cells of an `.ipynb` notebook, or the module docstring of a `.py` file. Notebook cells are joined with blank lines and scored as one document; code cells, outputs, and raw cells are excluded. File reports carry `lineno|hash|` addresses, prefixed with the cell ID for notebooks. For a `.py` file, addresses name lines of that file. `Result.location(finding)` returns the source location. [`score_paths`](https://AnswerDotAI.github.io/slopometer/score.html#score_paths) scores several files and directories in parallel, one result per file. A directory contributes its `.md`, `.qmd`, `.ipynb` and `.py` files. The command line accepts the same paths, or Markdown on stdin:
+
+    slopometer README.md
+    slopometer nbs/00_core.ipynb
+    slopometer pkg/module.py
+    slopometer nbs/ README.md
+    git log -1 --format=%B | slopometer --min-words 0
+    slopometer draft.md --threshold 10
+    slopometer draft.md --pangram
+
+`--threshold` returns exit code 1 when any file’s score exceeds the limit. Several paths or a directory give one report per file, highest density first. JSON output is a list with one entry per file. Each entry includes its `path`, which is null for stdin. Inputs below `--min-words` print the short-input message and exit successfully without scoring. JSON output marks them with `too_short: true`, includes `min_words`, and uses null scores. Every JSON finding includes a `location` object with one-based `line`, `address`, `end_line` and `end_address`. The end fields locate the finding’s last character. Notebook locations add `cell_id` and zero-based `cell_index`. Finding `start` and `end` offsets refer to the combined Markdown, not notebook JSON.
+
+`--pangram`, [`pangram_text`](https://AnswerDotAI.github.io/slopometer/pangram.html#pangram_text), [`pangram_path`](https://AnswerDotAI.github.io/slopometer/pangram.html#pangram_path) and [`pangram_paths`](https://AnswerDotAI.github.io/slopometer/pangram.html#pangram_paths) score with [Pangram](https://www.pangram.com)’s AI detector instead of the rules. Each finding is a window of text that Pangram scored. Density runs from 0 (human) to 100 (AI). `--threshold` compares against the same scale. Each request sends the text to Pangram’s servers and costs about \$0.05 per 100 words. The API key comes from `PANGRAM_API_KEY`.
+
+The command runs warm through `warmpy`. The first input long enough to score loads the model in a background process. Later calls answer in milliseconds. After thirty idle minutes the process exits.
+
+## What it checks, and what it cannot
+
+The rules live in notebooks that teach each family beside its code. Each rule states its tell, shows a violating example, and shows the plain rewrite. The [lexicon notebook](https://AnswerDotAI.github.io/slopometer/lexicon.html) holds the word and phrase rules. The [syntax notebook](https://AnswerDotAI.github.io/slopometer/syntax.html) builds sentence rules on spaCy’s parse. The [para notebook](https://AnswerDotAI.github.io/slopometer/para.html) covers paragraph and document rules and the limits of word-vector heuristics. The [score notebook](https://AnswerDotAI.github.io/slopometer/score.html) assembles the pipeline. A drift test asserts that every `write_docs` tell maps to a rule or to an explicit unscoreable registry. The meter and the style guide cannot drift apart silently.
+
+Vocabulary novelty is not scored. Identifying unexplained terminology requires audience and document context.
+
+A rule ships only when its false-positive rate on clean reference prose is near zero. The meter scores the style guide’s own clean passage at exactly 0.0. The score notebook measures the blind spot instead of hiding it: mechanically chopped prose passes every surface rule while staying opaque. Agent review (`check_docs`) and Pangram scoring (`--pangram`) cover that residue. Vale, write-good, and proselint solve neighboring problems. The [lexicon notebook](https://AnswerDotAI.github.io/slopometer/lexicon.html) records what came from them. It also credits the [GOV.UK words-to-avoid list](https://www.gov.uk/guidance/style-guide/a-to-z-of-gov-uk-style#words-to-avoid) (OGL v3).

{slopometer-0.0.2 → slopometer-0.0.4}/pyproject.toml

@@ -1,5 +1,5 @@
 [build-system]
-requires = ["setuptools>=64"]
+requires = ["setuptools>=77"]
 build-backend = "setuptools.build_meta"
 
 [project]
@@ -8,14 +8,14 @@ dynamic = ["version"]
 description = "Score prose for simplicity and precision against plain-English reference rules"
 readme = "README.md"
 requires-python = ">=3.10"
-license = {text = "Apache-2.0"}
+license = "Apache-2.0"
 authors = [{name = "Jeremy Howard", email = "github@jhoward.fastmail.fm"}]
 keywords = ['nbdev']
 classifiers = [
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3 :: Only",
 ]
-dependencies = ["fastcore", "spacy", "mdhtml", "warmpy>=0.1.0"]
+dependencies = ["fastcore", "spacy", "mdhtml>=0.1.40", "httpx2", "warmpy>=0.1.1", "rgapi>=0.1.27"]
 
 [project.urls]
 Repository = "https://github.com/AnswerDotAI/slopometer"

{slopometer-0.0.2 → slopometer-0.0.4}/slopometer/__init__.py

@@ -7,9 +7,10 @@ Modules:
 - `slopometer.corpus`: Build the labeled corpus: weak positives from scored replies, mixed human negatives
 - `slopometer.features`: Continuous measurements for learned scoring
 - `slopometer.lexicon`: Word and phrase rules: banned vocabulary, hedges, fillers, and splices
-- `slopometer.para`: Paragraph rules: restatement, elegant variation, forced symmetry, and coinage
+- `slopometer.pangram`: Score prose with Pangram's AI detector, as an alternative to the rules
+- `slopometer.para`: Paragraph rules: restatement, elegant variation, forced symmetry, and length
 - `slopometer.score`: Run every rule, weigh the findings, and report worst first
 - `slopometer.segment`: Markdown becomes typed blocks that keep their file positions
 - `slopometer.syntax`: The spaCy rules: sentences, clauses, passives, and the patterns anchored to them"""
 
-__version__ = "0.0.2"
+__version__ = "0.0.4"

{slopometer-0.0.2 → slopometer-0.0.4}/slopometer/_modidx.py

@@ -5,7 +5,8 @@ d = { 'settings': { 'branch': 'main',
                 'doc_host': 'https://AnswerDotAI.github.io',
                 'git_url': 'https://github.com/AnswerDotAI/slopometer',
                 'lib_path': 'slopometer'},
-  'syms': { 'slopometer.cli': {'slopometer.cli.main': ('cli.html#main', 'slopometer/cli.py')},
+  'syms': { 'slopometer.cli': { 'slopometer.cli._as_json': ('cli.html#_as_json', 'slopometer/cli.py'),
+                                'slopometer.cli.main': ('cli.html#main', 'slopometer/cli.py')},
             'slopometer.core': { 'slopometer.core.Finding': ('core.html#finding', 'slopometer/core.py'),
                                  'slopometer.core.Finding.__init__': ('core.html#finding.__init__', 'slopometer/core.py'),
                                  'slopometer.core.Finding.__repr__': ('core.html#finding.__repr__', 'slopometer/core.py'),
@@ -34,9 +35,27 @@ d = { 'settings': { 'branch': 'main',
                                     'slopometer.lexicon.find_emphasis': ('lexicon.html#find_emphasis', 'slopometer/lexicon.py'),
                                     'slopometer.lexicon.find_splice': ('lexicon.html#find_splice', 'slopometer/lexicon.py'),
                                     'slopometer.lexicon.lex_rule': ('lexicon.html#lex_rule', 'slopometer/lexicon.py')},
+            'slopometer.pangram': { 'slopometer.pangram.PangramFinding': ('pangram.html#pangramfinding', 'slopometer/pangram.py'),
+                                    'slopometer.pangram.PangramFinding.__init__': ( 'pangram.html#pangramfinding.__init__',
+                                                                                    'slopometer/pangram.py'),
+                                    'slopometer.pangram.PangramFinding.__repr__': ( 'pangram.html#pangramfinding.__repr__',
+                                                                                    'slopometer/pangram.py'),
+                                    'slopometer.pangram.PangramResult': ('pangram.html#pangramresult', 'slopometer/pangram.py'),
+                                    'slopometer.pangram.PangramResult.__init__': ( 'pangram.html#pangramresult.__init__',
+                                                                                   'slopometer/pangram.py'),
+                                    'slopometer.pangram.PangramResult._header': ( 'pangram.html#pangramresult._header',
+                                                                                  'slopometer/pangram.py'),
+                                    'slopometer.pangram.PangramResult.density': ( 'pangram.html#pangramresult.density',
+                                                                                  'slopometer/pangram.py'),
+                                    'slopometer.pangram._pangram_input': ('pangram.html#_pangram_input', 'slopometer/pangram.py'),
+                                    'slopometer.pangram._pangram_result': ('pangram.html#_pangram_result', 'slopometer/pangram.py'),
+                                    'slopometer.pangram._predict': ('pangram.html#_predict', 'slopometer/pangram.py'),
+                                    'slopometer.pangram.pangram_path': ('pangram.html#pangram_path', 'slopometer/pangram.py'),
+                                    'slopometer.pangram.pangram_paths': ('pangram.html#pangram_paths', 'slopometer/pangram.py'),
+                                    'slopometer.pangram.pangram_text': ('pangram.html#pangram_text', 'slopometer/pangram.py')},
             'slopometer.para': { 'slopometer.para._content': ('para.html#_content', 'slopometer/para.py'),
+                                 'slopometer.para._same_word': ('para.html#_same_word', 'slopometer/para.py'),
                                  'slopometer.para.find_bullet_mold': ('para.html#find_bullet_mold', 'slopometer/para.py'),
-                                 'slopometer.para.find_coinage': ('para.html#find_coinage', 'slopometer/para.py'),
                                  'slopometer.para.find_heading_echo': ('para.html#find_heading_echo', 'slopometer/para.py'),
                                  'slopometer.para.find_para_cap': ('para.html#find_para_cap', 'slopometer/para.py'),
                                  'slopometer.para.find_triad': ('para.html#find_triad', 'slopometer/para.py'),
@@ -46,28 +65,45 @@ d = { 'settings': { 'branch': 'main',
                                   'slopometer.score.Result.__init__': ('score.html#result.__init__', 'slopometer/score.py'),
                                   'slopometer.score.Result.__repr__': ('score.html#result.__repr__', 'slopometer/score.py'),
                                   'slopometer.score.Result._addr': ('score.html#result._addr', 'slopometer/score.py'),
+                                  'slopometer.score.Result._header': ('score.html#result._header', 'slopometer/score.py'),
+                                  'slopometer.score.Result._pos': ('score.html#result._pos', 'slopometer/score.py'),
                                   'slopometer.score.Result.density': ('score.html#result.density', 'slopometer/score.py'),
+                                  'slopometer.score.Result.location': ('score.html#result.location', 'slopometer/score.py'),
+                                  'slopometer.score.Result.too_short': ('score.html#result.too_short', 'slopometer/score.py'),
                                   'slopometer.score.Result.total': ('score.html#result.total', 'slopometer/score.py'),
                                   'slopometer.score.Result.worst': ('score.html#result.worst', 'slopometer/score.py'),
+                                  'slopometer.score._doc_paths': ('score.html#_doc_paths', 'slopometer/score.py'),
                                   'slopometer.score._rebase': ('score.html#_rebase', 'slopometer/score.py'),
+                                  'slopometer.score._score_file': ('score.html#_score_file', 'slopometer/score.py'),
                                   'slopometer.score.find_overstructure': ('score.html#find_overstructure', 'slopometer/score.py'),
                                   'slopometer.score.run_rules': ('score.html#run_rules', 'slopometer/score.py'),
                                   'slopometer.score.score_many': ('score.html#score_many', 'slopometer/score.py'),
                                   'slopometer.score.score_path': ('score.html#score_path', 'slopometer/score.py'),
+                                  'slopometer.score.score_paths': ('score.html#score_paths', 'slopometer/score.py'),
                                   'slopometer.score.score_text': ('score.html#score_text', 'slopometer/score.py')},
             'slopometer.segment': { 'slopometer.segment.Block': ('segment.html#block', 'slopometer/segment.py'),
                                     'slopometer.segment.Block.__init__': ('segment.html#block.__init__', 'slopometer/segment.py'),
                                     'slopometer.segment.Block.__repr__': ('segment.html#block.__repr__', 'slopometer/segment.py'),
                                     'slopometer.segment.n_headings': ('segment.html#n_headings', 'slopometer/segment.py'),
                                     'slopometer.segment.prose_words': ('segment.html#prose_words', 'slopometer/segment.py'),
+                                    'slopometer.segment.scored_words': ('segment.html#scored_words', 'slopometer/segment.py'),
                                     'slopometer.segment.scrub': ('segment.html#scrub', 'slopometer/segment.py'),
                                     'slopometer.segment.segment': ('segment.html#segment', 'slopometer/segment.py')},
             'slopometer.syntax': { 'slopometer.syntax._clausal': ('syntax.html#_clausal', 'slopometer/syntax.py'),
+                                   'slopometer.syntax._inert_subj': ('syntax.html#_inert_subj', 'slopometer/syntax.py'),
+                                   'slopometer.syntax._is_break': ('syntax.html#_is_break', 'slopometer/syntax.py'),
+                                   'slopometer.syntax._list_commas': ('syntax.html#_list_commas', 'slopometer/syntax.py'),
                                    'slopometer.syntax._n_clauses': ('syntax.html#_n_clauses', 'slopometer/syntax.py'),
+                                   'slopometer.syntax._n_sections': ('syntax.html#_n_sections', 'slopometer/syntax.py'),
                                    'slopometer.syntax._own_subj': ('syntax.html#_own_subj', 'slopometer/syntax.py'),
+                                   'slopometer.syntax._subject_last': ('syntax.html#_subject_last', 'slopometer/syntax.py'),
                                    'slopometer.syntax._tok_find': ('syntax.html#_tok_find', 'slopometer/syntax.py'),
+                                   'slopometer.syntax._verbless': ('syntax.html#_verbless', 'slopometer/syntax.py'),
+                                   'slopometer.syntax.find_and_splice': ('syntax.html#find_and_splice', 'slopometer/syntax.py'),
                                    'slopometer.syntax.find_artifact_agent': ('syntax.html#find_artifact_agent', 'slopometer/syntax.py'),
                                    'slopometer.syntax.find_clauses': ('syntax.html#find_clauses', 'slopometer/syntax.py'),
+                                   'slopometer.syntax.find_inert_act': ('syntax.html#find_inert_act', 'slopometer/syntax.py'),
+                                   'slopometer.syntax.find_inert_mind': ('syntax.html#find_inert_mind', 'slopometer/syntax.py'),
                                    'slopometer.syntax.find_intensifiers': ('syntax.html#find_intensifiers', 'slopometer/syntax.py'),
                                    'slopometer.syntax.find_nominalization': ('syntax.html#find_nominalization', 'slopometer/syntax.py'),
                                    'slopometer.syntax.find_noun_cluster': ('syntax.html#find_noun_cluster', 'slopometer/syntax.py'),
@@ -75,6 +111,7 @@ d = { 'settings': { 'branch': 'main',
                                    'slopometer.syntax.find_recipient_subject': ( 'syntax.html#find_recipient_subject',
                                                                                  'slopometer/syntax.py'),
                                    'slopometer.syntax.find_rhetorical': ('syntax.html#find_rhetorical', 'slopometer/syntax.py'),
+                                   'slopometer.syntax.find_sections': ('syntax.html#find_sections', 'slopometer/syntax.py'),
                                    'slopometer.syntax.find_semi_splice': ('syntax.html#find_semi_splice', 'slopometer/syntax.py'),
                                    'slopometer.syntax.find_sent_len': ('syntax.html#find_sent_len', 'slopometer/syntax.py'),
                                    'slopometer.syntax.find_verb_banned': ('syntax.html#find_verb_banned', 'slopometer/syntax.py'),

slopometer-0.0.4/slopometer/cli.py

@@ -0,0 +1,41 @@
+"""The command line: score a file or stdin, warm
+
+`slopometer <file>` prints the worst-first report, `slopometer` alone reads stdin, and `--threshold` turns the density into an exit code for CI and hooks. Several files or directories give one report per file, highest density first. `--pangram` scores with Pangram's AI detector instead of the rules. Loading the model takes about a second and scoring takes milliseconds, so the command runs through [warmpy](https://github.com/AnswerDotAI/warmpy): the first call starts a background process that loads the model, later calls reuse it, and it exits after thirty idle minutes. The Claude Code stop hook runs this same command with the response text on stdin.
+
+Docs: https://AnswerDotAI.github.io/slopometer/cli.html.md"""
+
+# AUTOGENERATED! DO NOT EDIT! File to edit: ../nbs/06_cli.ipynb.
+
+# %% auto #0
+__all__ = ['main']
+
+# %% ../nbs/06_cli.ipynb #01e3be12
+import sys
+from pathlib import Path
+from warmpy import warm_parse
+
+# %% ../nbs/06_cli.ipynb #d21704e3
+def _as_json(res):
+    "A `Result` as a JSON-ready dict, with each finding's source location"
+    findings = [dict(vars(f), location=res.location(f)) for f in res.findings]
+    return dict(path=res.path and str(res.path), density=res.density, worst=res.worst, words=res.words,
+        too_short=res.too_short, min_words=res.min_words, findings=findings)
+
+@warm_parse
+def main(
+    *paths:str, # Markdown, notebook, or `.py` files, or directories of them; stdin when omitted
+    threshold:float=None, # Exit code 1 when any density exceeds this, which runs from 0 to 100 with `pangram`
+    json:bool=False, # Emit JSON instead of reports: a list with one entry per file
+    min_words:int=150, # Minimum scored words; 0 disables the cutoff
+    pangram:bool=False, # Score with Pangram's AI detector instead of the rules; needs `PANGRAM_API_KEY`
+):
+    "Score Markdown, notebook, or module docstring prose against the aai reference-prose rules, or with Pangram's AI detector"
+    from json import dumps
+    from slopometer.score import score_paths, score_text
+    from slopometer.pangram import pangram_paths, pangram_text
+    sp,st = (pangram_paths,pangram_text) if pangram else (score_paths,score_text)
+    rs = sp(paths, min_words=min_words) if paths else [st(sys.stdin.read(), min_words=min_words)]
+    rs = sorted(rs, key=lambda r: (r.density is not None, r.density or 0), reverse=True)
+    if json: print(dumps([_as_json(r) for r in rs]))
+    else: print('\n\n'.join(map(str, rs)))
+    if threshold is not None and any(r.density is not None and r.density > threshold for r in rs): return 1

{slopometer-0.0.2 → slopometer-0.0.4}/slopometer/core.py

@@ -14,7 +14,7 @@ from fastcore.utils import *
 
 # %% ../nbs/00_core.ipynb #5fecc627
 tells = {
-    1: 'splices',              2: 'contract by aside',       3: 'emphasis devices',
+    1: 'splices',              2: 'key rule in an aside',    3: 'emphasis devices',
     4: 'elegant variation',    5: 'flourish over identifier', 6: 'consequence glue',
     7: 'hedging',              8: 'noting fillers',          9: 'restatement',
     10: 'justification rider', 11: 'decorative verbs',       12: 'audience misjudged',
@@ -50,7 +50,7 @@ def rule(
     name, # Rule name, unique within slopometer
     tell, # `write_docs` tell number the rule detects
     weight, # Tier the rule's findings carry
-    level, # The unit the detector reads: 'phrase', 'sentence', 'para', or 'doc'
+    level, # The unit the detector reads: 'phrase', 'markdown', 'sentence', 'para', or 'doc'
 ):
     "Register a detector, a function from its unit's text to the list of `Finding`s in it"
     def _f(f):
@@ -71,10 +71,9 @@ def find_decoration(txt):
 
 # %% ../nbs/00_core.ipynb #1af3670a
 unscoreable = {
-    2: 'knowing which fact is the contract takes judgment',
+    2: 'knowing which fact is the key rule takes judgment',
     5: "telling a flourish from a real identifier takes the project's vocabulary",
     9: 'lead-sentence restatement defeats averaged sentence vectors, and only the heading echo is detectable (measured in the para notebook)',
     10: 'a justification rider reads like a stated consequence, and only the domain says which it is',
-    11: 'verb texture beyond the banned-verb lexicon is a whiteboard judgment',
-    12: 'matching prose to its audience takes the audience, and the coinage rule covers only the definable half',
+    12: 'matching prose and terminology to the reader requires audience and document context',
     26: 'depth is judged against what the reader already knows',}

{slopometer-0.0.2 → slopometer-0.0.4}/slopometer/corpus.py

@@ -12,7 +12,7 @@ __all__ = ['GH_REPOS', 'CL_SKIP', 'label_overlap', 'label_density', 'reply_final
            'build_corpus']
 
 # %% ../nbs/08_corpus.ipynb #a2a2e052
-import base64, json, httpx
+import base64, json, httpx2
 from collections import Counter
 from datetime import datetime, timedelta
 from fastcore.utils import *
@@ -52,10 +52,10 @@ def _mirror_prompts(days, path_re):
     from aidialog.dlgskill import find_msgs
     from rgapi.skill import fd
     root = index().root
-    cutoff = datetime.now() - timedelta(days=days)
+    cutoff = (datetime.now() - timedelta(days=days)).timestamp()
     for p in fd(root, ext='ipynb', path_re=path_re):
-        if p.mtime <= cutoff: continue
-        yield from ((p, m) for m in find_msgs(msg_type='prompt', context=0, dlg=str(root/p)))
+        if p.stat().st_mtime <= cutoff: continue
+        yield from ((p.relative_to(root), m) for m in find_msgs(msg_type='prompt', context=0, dlg=p))
 
 def mirror_replies(
     days=21, # How far back to walk the mirror
@@ -89,7 +89,7 @@ async def readme_paras(
     gql = GhGql()
     names = [o['name'] async for o in gql.paged(gql.organization(login=org).repositories, 'name createdAt isFork')
         if o['createdAt'][:10] < before and not o['isFork']]
-    names = sorted(set(names) & {p.rstrip('/') for p in ls(root, dirs=True, files=False)} - set(exclude))
+    names = sorted(set(names) & {p.name for p in ls(root, dirs=True, files=False)} - set(exclude))
     txts = await gql.batch(gql.repo(f'{org}/{n}').object(expression='HEAD:README.md')('... on Blob { text }') for n in names)
     return [r for n, t in zip(names, txts) if t for r in doc_paras(t['text'], 'readme', f'{org}/{n}', min_words)]
 
@@ -159,11 +159,11 @@ async def gh_readme_paras(repos=GH_REPOS, date='2022-06-01', min_words=50):
 def _wiki_get(lang, tries=2, **params):
     for i in range(tries):
         try:
-            r = httpx.get(f'https://{lang}.wikipedia.org/w/api.php', follow_redirects=True, timeout=30,
+            r = httpx2.get(f'https://{lang}.wikipedia.org/w/api.php', follow_redirects=True, timeout=30,
                 headers={'User-Agent': 'slopometer/0.1 (https://github.com/AnswerDotAI/slopometer)'},
                 params=dict(action='query', prop='extracts', explaintext=1, exintro=1, format='json', **params))
             return r.json()['query']['pages'].values()
-        except httpx.TimeoutException:
+        except httpx2.TimeoutException:
             if i == tries-1: raise
 
 def wiki_paras(n=40, lang='en', min_words=50):

{slopometer-0.0.2 → slopometer-0.0.4}/slopometer/features.py

@@ -1,6 +1,8 @@
 """Continuous measurements for learned scoring
 
-The rules keep only what crosses a bar, and a learned scorer wants the bar removed. This module measures prose as continuous numbers: every rule's findings as a rate, the sentence and paragraph distributions the thresholds truncate, the parts-of-speech profile, specifics density, referential load, and burstiness. Three of these measures failed earlier as threshold rules, and the para and score notebooks record why. As features they need no threshold, because a learned combiner such as a random forest weighs them against labeled examples. The score stays rule-based and explainable. The feature vector exists for calibration experiments, and anything the experiments validate can graduate into a rule.
+`features` measures prose for experiments with learned scoring. It returns rule-finding rates, sentence and paragraph statistics, grammatical counts, and measures of concrete detail and references.
+
+Some measurements failed as standalone rules in the `para` and `score` notebooks. A model trained on labelled examples could still use them in combination. These experiments do not change slopometer's rule-based score.
 
 Docs: https://AnswerDotAI.github.io/slopometer/features.html.md"""
 

{slopometer-0.0.2 → slopometer-0.0.4}/slopometer/lexicon.py

@@ -1,6 +1,6 @@
 """Word and phrase rules: banned vocabulary, hedges, fillers, and splices
 
-The phrase-level rules: banned vocabulary with plain replacements, hedge phrases, noting fillers, filler transitions, splice punctuation, consequence glue, and markdown emphasis. Each is a regex over a block's scrubbed text, needs no linguistic analysis, and runs in microseconds.
+These rules match words, phrases, punctuation, and Markdown emphasis in scrubbed text. They cover banned vocabulary, hedges, fillers, splices, and consequence joins. Findings include plain replacements where available. The rules use regexes without linguistic analysis.
 
 Docs: https://AnswerDotAI.github.io/slopometer/lexicon.html.md"""
 
@@ -12,10 +12,10 @@ __all__ = ['banned', 'find_banned', 'hedges', 'noting', 'transitions', 'find_hed
 
 # %% ../nbs/02_lexicon.ipynb #23bc602c
 from fastcore.utils import *
+from mdhtml import inlines
 from .core import *
 from .segment import *
 
-
 # %% ../nbs/02_lexicon.ipynb #449d550b
 def lex_rule(
     name, # Rule name, as in `core.rule`
@@ -64,13 +64,13 @@ find_noting = lex_rule('noting', tell=8, weight=SMELL, lex=noting)
 find_transitions = lex_rule('transitions', tell=19, weight=SMELL, lex=transitions)
 
 # %% ../nbs/02_lexicon.ipynb #79549961
-_splice = re.compile(r'—|(?<=\w) -- (?=\w)|(?<=\w) - (?=\w)')
+_splice = re.compile(r'—')
 _conseq = re.compile(r', so\b')
 
-@rule('splice', tell=1, weight=KILL, level='phrase')
+@rule('splice', tell=1, weight=4, level='phrase')
 def find_splice(txt):
-    "Em dashes, two-hyphen dashes, and spaced-hyphen joins"
-    return [Finding('splice', 1, m.start(), m.end(), m.group(), KILL) for m in _splice.finditer(txt)]
+    "Em dashes"
+    return [Finding('splice', 1, m.start(), m.end(), m.group(), 4) for m in _splice.finditer(txt)]
 
 @rule('conseq', tell=6, weight=SMELL, level='phrase')
 def find_conseq(txt):
@@ -78,12 +78,10 @@ def find_conseq(txt):
     return [Finding('conseq', 6, m.start(), m.end(), m.group(), SMELL) for m in _conseq.finditer(txt)]
 
 # %% ../nbs/02_lexicon.ipynb #8daaf936
-_emph = re.compile(r'\*\*[^*\n]+\*\*|(?<![\w*])\*[^*\s][^*\n]*\*(?![\w*])|(?<![\w_])__[^_\n]+__(?![\w_])')
-
-@rule('emphasis', tell=3, weight=SMELL, level='phrase')
+@rule('emphasis', tell=3, weight=SMELL, level='markdown')
 def find_emphasis(txt):
-    "Bold and italic markers in running prose"
-    return [Finding('emphasis', 3, m.start(), m.end(), m.group(), SMELL) for m in _emph.finditer(txt)]
+    "Bold and italic text in Markdown `txt`"
+    return [Finding('emphasis', 3, n['start'], n['end'], txt[n['start']:n['end']], SMELL) for n in inlines(txt) if n['type'] in ('emph', 'strong')]
 
 # %% ../nbs/02_lexicon.ipynb #cdb0967f
 wordy = {'all of': 'all', 'additional': 'more', 'a number of': 'some', 'along the lines of': 'like',

slopometer-0.0.4/slopometer/pangram.py

@@ -0,0 +1,117 @@
+"""Score prose with Pangram's AI detector, as an alternative to the rules
+
+`pangram_text` and `pangram_path` send prose to [Pangram](https://www.pangram.com)'s AI-detection API instead of running the rules. They read the same Markdown as `score_text` and `score_path`. They return a `Result` with the same addresses, JSON fields and `min_words` cutoff. `slopometer --pangram` runs `pangram_path` from the command line.
+
+Pangram scores text in windows, which can run to hundreds of words. Each finding is one window. Its weight is the window's score multiplied by 10. A window that Pangram rates fully AI-written weighs 10, the same as a kill-tier rule finding. The density is the mean window score, weighted by word count, on a scale from 0 (human) to 100 (AI). Under `--pangram`, `--threshold` compares against this scale.
+
+Each request sends the text to Pangram's servers and costs about $0.05 per 100 words. The functions read the API key from `PANGRAM_API_KEY`.
+
+Docs: https://AnswerDotAI.github.io/slopometer/pangram.html.md"""
+
+# AUTOGENERATED! DO NOT EDIT! File to edit: ../nbs/05a_pangram.ipynb.
+
+# %% auto #0
+__all__ = ['PANGRAM_URL', 'PangramFinding', 'PangramResult', 'pangram_text', 'pangram_path', 'pangram_paths']
+
+# %% ../nbs/05a_pangram.ipynb #50ef2286
+import time, httpx2
+from bisect import bisect_right
+from fastcore.utils import *
+from .core import *
+from .segment import *
+from .score import Result, _score_file, _doc_paths
+
+# %% ../nbs/05a_pangram.ipynb #a265951c
+def _pangram_input(blocks):
+    "Text to send to Pangram for `blocks`, and each block's offset in it"
+    offs, n = [], 0
+    for b in blocks:
+        offs.append(n)
+        n += len(b.txt) + 1
+    return '\n'.join(b.txt for b in blocks), offs
+
+# %% ../nbs/05a_pangram.ipynb #a1383847
+PANGRAM_URL = 'https://text.external-api.pangram.com'
+
+def _predict(
+    text, # Text to classify
+    timeout=120, # Seconds to wait for the task to finish
+):
+    "Pangram's response for `text`, from a task posted to its API and polled until done"
+    with httpx2.Client(base_url=PANGRAM_URL, headers={'x-api-key': os.environ['PANGRAM_API_KEY']}) as cli:
+        tid = cli.post('/task', json=dict(text=text, model='default')).raise_for_status().json()['task_id']
+        for _ in range(timeout*2):
+            res = cli.get(f'/task/{tid}').raise_for_status().json()
+            if res['stage'] == 'STAGE_SUCCESS': return res
+            if res['stage'] == 'STAGE_FAILED': raise RuntimeError(f'Pangram task {tid} failed')
+            time.sleep(0.5)
+    raise TimeoutError(f'Pangram task {tid} did not finish in {timeout} seconds')
+
+# %% ../nbs/05a_pangram.ipynb #ddbc675b
+class PangramFinding(Finding):
+    def __init__(self,
+        start, # Character offset where the window starts
+        end, # Character offset one past the window's end
+        text, # The window's text
+        score, # Pangram's `ai_assistance_score`, from 0 (human) to 1 (AI)
+        label, # Pangram's label, such as 'AI-Generated' or 'Human Written'
+        confidence, # 'High', 'Medium', or 'Low'
+        words, # Words in the window
+    ):
+        super().__init__('pangram', None, start, end, text, round(10*score))
+        store_attr('score,label,confidence,words')
+    def __repr__(self):
+        ex = self.text if len(self.text) <= 80 else self.text[:77]+'...'
+        return f'[{self.weight}] pangram {self.score:.2f} {self.label}, {self.confidence} confidence, {self.words} words: {ex!r}'
+
+# %% ../nbs/05a_pangram.ipynb #392e91e1
+class PangramResult(Result):
+    def __init__(self,
+        txt, # The scored document
+        findings, # `PangramFinding`s, worst first
+        words, # Word count across all scored blocks
+        payload=None, # Pangram's response; None when no request was sent
+        min_words=150, # Minimum scored words; 0 disables the cutoff
+    ):
+        super().__init__(txt, findings, words, min_words=min_words)
+        self.payload = payload
+    def _header(self): return f'pangram score {self.density} on {self.words} prose words, worst {self.worst}'
+
+# %% ../nbs/05a_pangram.ipynb #acd5edf6
+@patch(as_prop=True)
+def density(self:PangramResult):
+    "Mean window score, weighted by word count and scaled to 0-100"
+    if self.too_short: return None
+    n = sum(f.words for f in self.findings)
+    return round(100*sum(f.score*f.words for f in self.findings)/max(n, 1), 1)
+
+# %% ../nbs/05a_pangram.ipynb #098e727a
+def _pangram_result(txt, payload, min_words=150):
+    "`PangramResult` for markdown `txt`, with the windows in Pangram's `payload` mapped back to document offsets"
+    blocks = segment(txt)
+    sent, offs = _pangram_input(blocks)
+    if payload['text'] != sent: raise ValueError('Pangram scored different text from the text sent, and its window offsets cannot be mapped')
+    def doc_off(i):
+        k = bisect_right(offs, i) - 1
+        return blocks[k].start + min(i-offs[k], len(blocks[k].txt))
+    fs = [PangramFinding(doc_off(w['start_index']), doc_off(w['end_index']-1)+1, w['text'], w['ai_assistance_score'],
+        w['label'], w['confidence'], w['word_count']) for w in payload['windows']]
+    return PangramResult(txt, sorted(fs, key=lambda f: (-f.score, f.start)), scored_words(blocks), payload, min_words)
+
+# %% ../nbs/05a_pangram.ipynb #24c81a92
+def pangram_text(txt, min_words=150):
+    "Score markdown `txt` with Pangram, sending no request below `min_words` scored words"
+    if min_words < 0: raise ValueError('min_words must be non-negative')
+    blocks = segment(txt)
+    words = scored_words(blocks)
+    if not words or words < min_words: return PangramResult(txt, [], words, min_words=min_words)
+    return _pangram_result(txt, _predict(_pangram_input(blocks)[0]), min_words)
+
+# %% ../nbs/05a_pangram.ipynb #db5d53f7
+def pangram_path(p, min_words=150):
+    "Score a Markdown file, an `.ipynb` file's Markdown cells, or a `.py` file's module docstring with Pangram, with exhash addresses"
+    return _score_file(p, pangram_text, min_words)
+
+def pangram_paths(paths, n_workers=8, min_words=150):
+    "Score files and directories with Pangram in parallel threads, one `PangramResult` per file"
+    return parallel(pangram_path, _doc_paths(paths), n_workers=n_workers, threadpool=True, min_words=min_words)