on-the-list 0.1.0__tar.gz

on_the_list-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Waiga Arya
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

on_the_list-0.1.0/PKG-INFO

@@ -0,0 +1,417 @@
+Metadata-Version: 2.4
+Name: on-the-list
+Version: 0.1.0
+Summary: Reads a cosmetic ingredient list and reports what the EU's official annexes say about the ingredients on it.
+Author: Waiga Arya
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Waiga/on-the-list
+Project-URL: Issues, https://github.com/Waiga/on-the-list/issues
+Keywords: cosmetics,inci,ingredients,cosing,labelling,cli
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Manufacturing
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# on-the-list
+
+Reads a cosmetic ingredient list and reports what the EU's official annexes say
+about the ingredients on it.
+
+Runs entirely on your machine. No account, no API key, no upload, no
+dependencies beyond Python itself. The annexes are shipped with the package;
+downloading a fresh copy is a separate command you have to type.
+
+What came back when it was pointed at 16,635 real published labels, including
+the checks it gets wrong and how often, is written up in
+[An ingredient list cannot tell you most of what you want to know](https://medium.com/@aryawaiga0/an-ingredient-list-cannot-tell-you-most-of-what-you-want-to-know-f3807f357837).
+
+```
+$ on-the-list examples/shade-stick.txt
+
+on-the-list 0.1.0 — examples/shade-stick.txt
+CosIng annexes II-VI, Commission last update 28/08/2026, 1913 distinct names,
+read from the copy shipped with this package
+
+10 ingredients read. 2 prohibited, 1 colourant order, 1 repeated entry.
+
+NAMES THAT MATCH AN ANNEX II ENTRY
+  Annex II is the list of substances prohibited in cosmetic products. A match
+  is reported here as a match. Whether it means anything about this product
+  is for someone with the formulation in front of them.
+
+  Butylphenyl Methylpropional matches Annex II, entry 1666, in the
+  Commission's list of substances prohibited in cosmetic products.
+      printed as: Butylphenyl Methylpropional
+      register name: BUTYLPHENYL METHYLPROPIONAL (read from the 'identified'
+      column)
+      annex entry: 2-(4-tert-butylbenzyl) propionaldehyde
+```
+
+## The thing to understand before anything else
+
+**An ingredient that appears in none of the annexes is not an error, and this
+tool never reports it as one.**
+
+Annexes II to VI are lists of *restricted* substances: prohibited, restricted,
+permitted colourants, permitted preservatives, permitted UV filters. They are
+not an inventory of valid INCI names. Aqua and Glycerin appear in none of them,
+because nothing restricts Aqua or Glycerin.
+
+So *"that is not a real INCI name"* is not a check this tool can make, and it
+does not attempt one. Absence from the annexes means **these annexes say
+nothing about it**, and the report says exactly that, in those words. There is
+no `unknown` and no `unrecognised` anywhere in the output.
+
+This is the most likely way to misread the tool, which is why it is the first
+thing on the page. The reasoning is set out in full in
+[`docs/superpowers/specs/2026-09-09-on-the-list-design.md`](docs/superpowers/specs/2026-09-09-on-the-list-design.md).
+
+## What it does not do
+
+It never says a product is compliant, safe, legal, permitted or clean. It
+compares names printed on a label with names printed in a published annex, and
+reports where they coincide. Whether that means anything about a particular
+product depends on its concentration, its product type, its route of exposure
+and its formulation, none of which an ingredient list states.
+
+That is not modesty. It is the only claim the evidence supports.
+
+## Install
+
+```bash
+pip install .
+```
+
+Python 3.11 or newer. Nothing else.
+
+## Use
+
+```bash
+on-the-list label.txt                      # whole pack text; the list is found in it
+cat product-page.txt | on-the-list -
+on-the-list --ingredients inci.txt         # you already have just the list
+on-the-list --ingredients inci.txt --pack-text pack.txt   # runs the warning check too
+on-the-list label.txt --format markdown    # to paste into an email
+on-the-list label.txt --format json        # for a pipeline
+on-the-list label.txt --skip colourant-order
+on-the-list --list-checks
+on-the-list register                       # which register is in use, and its hashes
+on-the-list update-register                # the only command that uses the network
+```
+
+Exit codes: `0` nothing found, `1` at least one finding, `2` could not run. A
+label whose ingredient list did not parse exits `2`, never `0`, and so does a
+run with every check switched off: a green CI job over a label that nothing was
+compared to is the one thing this tool exists not to do.
+
+## The four checks
+
+Each one can be switched off on its own with `--skip`, and the report always
+prints which ran and which did not, with a reason. A check that did not run must
+never read as a check that found nothing.
+
+| | |
+|---|---|
+| **prohibited** | An ingredient's name matches an entry in **Annex II**. Reported as a match with the entry number, split into entries that say something unconditional about the substance and entries whose own wording sets a condition. |
+| **colourant-order** | A `CI NNNNN` colourant is printed before a non-colourant. Article 19(1)(g) allows colourants in any order *after* the other ingredients, so this is a positional observation with no judgement in it. |
+| **repeated-entry** | The same name appears twice in the declared list. Needs no register at all. |
+| **warning-wording** | An annex attaches a `Contains …` statement to an ingredient on the list, and that statement is not in the pack text you supplied. Reported as a **gap between two documents**, never as a violation: the wording may be printed somewhere the supplied text does not cover, in another language, or the entry's condition may not apply. It needs pack text to search — a whole label file counts as that, `--ingredients` on its own does not — and it says which when it does not run. |
+
+Two things that are not checks and change no exit code:
+
+**What the annexes name.** Every entry in Annexes II to VI that names an
+ingredient on the list, with what that annex is — prohibited, restricted, a
+permitted colourant, preservative or UV filter — and the product types the entry
+is limited to. This is the tool's own title, and for a while it was the one
+thing the report did not print: a label containing Phenoxyethanol produced no
+finding, and the reader was told one of three ingredients was named somewhere
+and never told where.
+
+**Coverage.** How many ingredients no annex entry names, labelled *not
+restricted by these annexes*.
+
+### Four narrowings that cost coverage on purpose
+
+**The position check uses only bare `CI NNNNN` forms.** Named Annex IV
+substances are not used, because most do more than one job: Titanium Dioxide is
+an Annex IV colourant, an Annex VI UV filter and an opacifier, and where it
+sits in a list proves nothing about which of those it is doing. A CI number is
+unambiguous.
+
+**Only `Contains …` statements are searched for in pack text.** The annexes'
+wording column mixes label text with conditions that have nothing to do with a
+label — *"Purity criteria as set out in Commission Directive 95/45/EC (E 129)"*,
+*"Only nanomaterials having the following characteristics are allowed"*.
+Treating the whole column as label text would report a missing warning on every
+sunscreen containing zinc oxide. 38 of the 627 entries in Annexes III to VI
+yield a statement this tool will look for; the rest are shown to the reader and
+not searched for, and the report says so.
+
+**A match on part of a printed name is not reported against Annex II.**
+`CI 77288 / CHROMIUM` is a colourant printed beside its element name, and the
+fragment `CHROMIUM` matches the Annex II entry for chromium metal. Over 16,635
+real labels, fragment matches produced no correct Annex II finding at all, so
+they are shown under *considered and not counted* with the reason instead. A
+name with a bracketed aside removed — `Titanium Dioxide (nano)` → `Titanium
+Dioxide` — is still the name of the ingredient, and does match.
+
+**Nothing is built on a concentration.** A label does not state one. Every
+maximum-concentration column in the annexes is unused.
+
+## Reproducibility
+
+The register shipped in this package is the European Commission's own CosIng
+export, byte for byte. Every report names the version it used, and you can
+check it:
+
+```bash
+$ on-the-list register
+CosIng annexes II-VI, Commission last update 28/08/2026, 1913 distinct names,
+read from the copy shipped with this package
+
+CosIng, © European Union, 1995-2026. Annexes II to VI of Regulation (EC)
+No 1223/2009, retrieved from the European Commission CosIng API. Reused under
+Commission Decision 2011/833/EU (CC BY 4.0). Unmodified.
+
+Annex II — list of substances prohibited in cosmetic products
+  1758 entries, Commission last update 28/08/2026
+  sha256 b7105a05bf724bb10cebaac3a3813b6146c3153ae1d07097d35bb1f73cd7283b
+  https://api.tech.ec.europa.eu/cosing20/1.0/api/annexes/II/export-csv
+```
+
+```bash
+curl -s https://api.tech.ec.europa.eu/cosing20/1.0/api/annexes/II/export-csv | shasum -a 256
+```
+
+If that hash no longer matches, the Commission has republished the annex, and a
+result produced against the old one is a result about a document that no longer
+exists. `on-the-list update-register` downloads the current files; the report
+then names the new hashes instead.
+
+Full manifest, including the corpus used for the measurements below:
+[`docs/corpus-manifest.md`](docs/corpus-manifest.md).
+
+## Measured against real labels
+
+Unit tests pass on the inputs their author imagined, which proves very little.
+This was run over **16,635 real published cosmetic labels** from an Open Beauty
+Facts export — real packs, real messiness, none of it written by this project.
+The selection rule is every record in the export whose `ingredients_text` field
+is at least 50 characters. Nothing else is filtered or sampled.
+
+`tools/measure_corpus.py` is the script that produced every number here.
+
+| | |
+|---|---|
+| Records in the export | 64,237 |
+| Labels selected | 16,635 |
+| Labels whose list parsed | 16,535 |
+| Crashes | 0 |
+| Ingredients parsed | 298,423 |
+| **Annex II matches** | **2,408** on 2,037 labels |
+| — where the annex entry is unconditional | 1,066 on 942 labels |
+| — where the annex entry sets a condition | 1,342 on 1,167 labels |
+| Colourant position | 975 on 645 labels |
+| Repeated entries | 712 on 291 labels |
+| Warning wording not found | 1,170 on 1,027 labels |
+| Considered and not counted | 554 |
+| Labels with pack prose inside the ingredient panel | 4,288 (25.8%) |
+| Labels whose ingredient field carries a section heading | 1,631 (9.8%) |
+
+Every one of those comes out of `tools/measure_corpus.py`, including the
+per-entry and per-statement breakdowns quoted below. `--samples` draws the
+audit samples with the seed and size the manifest names.
+
+### How wrong each check is
+
+Hand-audited against the published annex text or the label's own list. These
+are the numbers, not a summary of them.
+
+**prohibited — 9.3% wrong on the unconditional group.** All 19 Annex II entries
+behind the 1,066 unconditional findings were read against the annex text.
+Eighteen are correct: Butylphenyl Methylpropional (651 findings), Zinc
+Pyrithione (105), Hydroxyisohexyl 3-Cyclohexene Carboxaldehyde (100),
+Pentasodium Pentetate (65), Ergocalciferol and Cholecalciferol (11), borates
+(7), 4-Methylbenzylidene Camphor (5), Formaldehyde (4), and a tail of one to
+three each. One is wrong, and it is the tool's largest single known false
+positive: **Annex II entry 1388 is `Octamethylcyclotetrasiloxane; D4`, and the
+Commission's own identified-ingredients column gives its INCI name as
+`CYCLOMETHICONE`**. Cyclomethicone names a *mixture* of cyclic siloxanes, so a
+label printing it has not said it contains D4. That produced 99 findings. There
+is no mechanical signal separating this from a correct match, so it is reported
+here rather than patched around.
+
+The 1,342 conditional findings are counted neither right nor wrong, because an
+ingredient list cannot settle them. They are dominated by entries whose
+condition no label states: petrolatum *"except if the full refining history is
+known"*, the furocoumarin entry that names ordinary citrus oils *"except for
+normal content in natural essences used"*, and the colourants prohibited only
+*"when used as a substance in hair dye products"*. Each is printed with its own
+wording so a reader can see the condition and decide.
+
+**colourant-order — 4 of 30 wrong.** Thirty findings drawn by
+`tools/measure_corpus.py --samples out.json` (seed 11, one row per label), each
+read against the label's own list. Two failures are pack prose that the parser
+kept — Spanish marketing copy after the colourants, and an OCR'd panel where
+the list has no commas at all. One is a make-up palette declaring several
+shades in one field. One is an OCR'd label whose text before `INGREDIENTS` is
+unreadable.
+
+The underlying weakness is deeper than that rate suggests, and it is stated in
+[`docs/limitations.md`](docs/limitations.md): the Regulation *permits*
+colourants after the other ingredients but does not *require* it, so a colourant
+in weight order is not necessarily out of place. The check reports where a name
+is printed and nothing more.
+
+**repeated-entry — 16 of 30 wrong.** Thirty findings from the same draw, read
+the same way. This is the weakest check in the tool and the number is not a
+typo.
+
+Eight of the sixteen are **multi-component packs**: a hair colour kit declaring
+the crème, the developer and the conditioner in one field; a 2-in-1 shampoo. An
+ingredient appearing once in each component is not a repeated ingredient. The
+other eight are fields that are not one product's ingredient list at all — an
+alphabetical glossary of every ingredient a brand uses, a food supplement, a
+certification mark parsed as an ingredient, and OCR damage that split one name
+into two.
+
+The tool detects the multi-component case when the pack prints a heading it can
+see — `Gel N°1:`, `MASQUE:`, a second `Ingredients:` — and then says so and does
+not run either order-dependent check. It flagged 1,631 of the 16,635 labels that
+way. It cannot detect it when the pack prints no heading, which is most of the
+time, and no heuristic tried here improved that materially without losing real
+findings.
+
+**On a single product's own ingredient list — which is what the tool is for —
+none of the sixteen failure modes applies.** All twelve sound findings in the
+same sample were single-product lists. The corpus number is still the honest one
+to publish, and it is 53%.
+
+**warning-wording — not measured.** Open Beauty Facts carries no field holding
+the text printed on a pack, so every finding in the corpus run is a gap by
+construction and the rate means nothing. What the run does establish is that the
+check fires on the right population: of the 1,170 findings, 692 are
+`Contains sodium fluoride` on fluoride toothpastes, 132
+`Contains sodium monofluorophosphate`, 127 `Contains hydrogen peroxide` on
+developers, 93 `Contains ammonia` on hair colour and 48
+`Contains Benzophenone-3` on sunscreens. Its accuracy is untested, and this
+README will say so until a corpus with real pack text exists.
+
+### What the measurement found that the tests did not
+
+Every one of these was a real defect, found only by meeting 16,635 real labels
+or by reading the annexes against their own text:
+
+- **Feeding `csv.reader` a list of lines destroys newlines inside quoted
+  fields.** The annexes put ten-line warning blocks in one cell.
+  `Contains selenium disulphide\nAvoid contact with eyes` silently became one
+  string. The extractor found 32 statements instead of 38, and 19 of the ones it
+  did find ran on into the sentence after them. No error, all tests green. The
+  reader uses `io.StringIO`.
+- **The identified-ingredients column is comma-separated and its values contain
+  commas.** A plain `split(",")` gives a different answer from the parser on 52
+  rows, and on 38 of them it leaves a one-character fragment: a substance called
+  `N`, from `N,N-DIETHYL-m-AMINOPHENOL`.
+- **Annex II entry 1725 is `Styrene/Acrylates copolymer (nano)` and lists the
+  ordinary INCI name beside it.** 421 labels printing the ordinary polymer
+  matched a prohibited entry that is not about it. They are now reported as
+  considered and not counted, with the reason.
+- **HTML entities were resolved after splitting, not before.** The `;` inside
+  `<` is a separator: one Russian pack produced three ingredients called
+  `&lt`, an American one produced three called `FD`, and each set was reported
+  as a repeated ingredient.
+- **`[+/- CI 77491, CI 77492]` closes its bracket after the colourants, not
+  after the marker.** A pattern that expected `[+/-]` as a unit matched none of
+  it, so a whole shade-range block was read as declared ingredients.
+- **Toothpastes print `Contient du fluorure de sodium (1450 ppm de fluor)`
+  immediately after the colourants with no break.** Read as an ingredient, it
+  was the commonest reason a colourant appeared to be printed in front of a
+  non-colourant.
+- **`Cl 77492` with a lowercase L is endemic in scanned panels**, and appears
+  beside a correctly read `CI` in the same list.
+- **`Styrene / Acrylates Copolymer` is one name, not two.** Treating a spaced
+  slash as a synonym separator left the fragment `Styrene`, which matches the
+  styrene monomer in Annex II, on 35 real labels. Requiring a space on both
+  sides of the slash left 21 of them; a polymer names its monomers with slashes,
+  and the rule now says so, which leaves none.
+- **Hydroquinone is prohibited by Annex II "with the exception of entry 14 in
+  Annex III"**, where it is allowed in professional nail products. Reporting
+  only the prohibition is a half-truth, so a match that also appears in another
+  annex now says which.
+- **Only 314 of Annex II's 1,758 rows carry an INCI name at all.** The rest are
+  chemical or CAS identifiers with no cosmetic-glossary equivalent. The
+  prohibited check can only see 18% of the prohibited list.
+- **A file handed to `--ingredients` usually starts with the word
+  "Ingredients:".** It was not stripped on that path, so the first ingredient
+  became `Ingredients: Formaldehyde`, matched nothing, and the run exited 0 —
+  while the same text through the whole-label path reported the match.
+- **`Chromium (CI 77288)` was reported as prohibited and `CI 77288 / CHROMIUM`
+  was not.** Same substance, two print orders, opposite answers. What survives a
+  bracket being removed is the name only if it is at least half the words.
+- **`Glycerin +/- 0.5%` opened a shade-range block.** Every declared ingredient
+  after a printed tolerance dropped out of both order-dependent checks, and the
+  report said only that the list had a "may contain" block.
+- **The position check was quadratic.** 20,000 colour index numbers took 64
+  seconds; the hostile-input tests never called the checks, only the parser.
+- **A colourant found only in Annex II was described as "listed in Annex IV".**
+  Four are: CI 12150, CI 20170, CI 27290 and CI 45425, each prohibited in hair
+  dye. The finding now names the entries the register actually holds.
+- **A malformed `--register` directory raised a traceback and exited 1** — the
+  code for "findings were reported" — and the message about the Commission
+  having changed the export format was unreachable.
+- **`--skip` on all four checks exited 0.** Nothing was compared, and the report
+  said "nothing found".
+
+The corpus itself is not redistributed here. Open Beauty Facts is ODbL 1.0,
+which is share-alike and incompatible with this repository's MIT licence. Only
+the measurements are published, with the export's hash so you can obtain the
+same file.
+
+## What it misses
+
+Stated plainly, because a checking tool that hides its blind spots is worse than
+none. The long version is in [`docs/limitations.md`](docs/limitations.md).
+
+- **It sees 18% of Annex II.** Most rows have no INCI name, so most prohibited
+  substances cannot be matched against a label at all.
+- **It cannot see concentration.** Most annex restrictions are limits, and a
+  limit is not something a name can breach.
+- **It cannot see product type.** A restriction on hair dye says nothing about a
+  face cream, and an ingredient list does not say which the product is.
+- **It matches exact names only, after folding.** No fuzzy matching, no
+  substring search, no edit distance. A misspelling, a supplier's trade name, or
+  a name the Commission writes differently will not match, and the tool will not
+  tell you it missed one. It does look a printed name up under a bracketed or
+  slash-separated part of itself — that is how `CI 77891` is found inside
+  `Titanium Dioxide (CI 77891)` — but **only the Annex II check refuses a match
+  found that way**; the other checks and the coverage count accept it.
+- **It reads five annexes.** Annex I (the safety report) and Annex VII are not
+  read, and neither is any national requirement, retailer standard, or rule
+  outside the EU.
+- **A pack panel inside the ingredient field derails everything.** A quarter of
+  the corpus had prose in the panel. The tool says so when it can tell.
+- **A "may contain" block is excluded from the position check** and is printed
+  across a whole shade range, so nothing positional can be said about it.
+- **Crowd-sourced label data can be wrong.** In the measurement above, a finding
+  may be about a mistyped record rather than about a pack.
+
+## Licence
+
+MIT, for the code. See `LICENSE`.
+
+The annex data in `on_the_list/data/` is European Commission material: CosIng,
+© European Union, 1995-2026, Annexes II to VI of Regulation (EC) No 1223/2009,
+retrieved from the CosIng API and redistributed unmodified. Reuse is governed by
+Commission Decision 2011/833/EU; the Commission's legal notice states that
+content it owns is licensed under CC BY 4.0.
+
+Nothing in this repository is legal or regulatory advice.