taxo 1.0.0__tar.gz

taxo-1.0.0/.gitignore

@@ -0,0 +1,27 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+*.egg
+.eggs/
+build/
+dist/
+.venv/
+
+# uv
+uv.lock
+
+# Packaging
+taxotagger.egg-info/
+
+# Test / coverage
+.pytest_cache/
+.coverage
+htmlcov/
+
+# Editors / OS
+.vscode/
+.idea/
+.DS_Store
+
+

taxo-1.0.0/CITATION.cff

@@ -0,0 +1,17 @@
+# This CITATION.cff file was generated with cffinit.
+# Visit https://bit.ly/cffinit to generate yours today!
+
+cff-version: 1.2.0
+title: taxo
+message: >-
+  If you use this software, please cite it using the
+  metadata from this file.
+type: software
+authors:
+  - given-names: Andreas
+    family-names: Hamm
+    email: andreas.hamm@dlr.de
+    affiliation: German Aerospace Center (DLR)
+    orcid: 'https://orcid.org/0000-0001-5854-851X'
+url: 'https://elib.dlr.de/190966/'
+license: MIT

taxo-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 German Aerospace Center (DLR)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

taxo-1.0.0/PKG-INFO

@@ -0,0 +1,177 @@
+Metadata-Version: 2.5
+Name: taxo
+Version: 1.0.0
+Summary: Tag documents with taxonomy terms using embeddings and a concept hierarchy
+Project-URL: Homepage, https://github.com/DLR-SC/taxo-tag
+Project-URL: Repository, https://github.com/DLR-SC/taxo-tag
+Author-email: Andreas Hamm <andreas.hamm@dlr.de>
+License: MIT
+License-File: LICENSE
+Keywords: document-tagging,embeddings,hierarchy,sentence-transformers,taxonomy
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: networkx>=3.4.2
+Requires-Dist: nltk>=3.9.1
+Requires-Dist: numpy>=2.2.5
+Requires-Dist: openpyxl>=3.1.5
+Requires-Dist: pandas>=2.2.3
+Requires-Dist: scipy>=1.14.1
+Requires-Dist: sentence-transformers>=4.1.0
+Requires-Dist: torch>=2.6.0
+Requires-Dist: transformers>=4.46.0
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# taxo
+
+Tag documents with terms from a concept hierarchy (taxonomy) using sentence
+embeddings.
+
+`taxo` reads in a hierarchy of concepts, embeds their descriptions in a shared
+vector space, and then finds the concepts most related to a document. It can
+also aggregate sentence-level evidence along hierarchical paths to produce a
+consistent set of top-level tags.
+
+For the main ideas used in `taxo`, see
+```
+@inproceedings{dlr190966,
+           title = {Strategy Comparison for Semantic Zero-Shot Taxonomy Filters},
+           month = {October},
+       booktitle = {4th International Open Search Symposium (OSSYM 2022)},
+          author = {Hamm, Andreas},
+            year = {2022},
+             url = {https://elib.dlr.de/190966/}
+}
+```
+
+## Installation
+
+On Python >= 3.11:
+
+```bash
+pip install taxo
+```
+
+This is a uv-style package, so `uv` works too:
+
+```bash
+uv add taxo
+```
+
+## Usage
+The package is both installed and imported as `taxo`.
+
+Given a taxonomy (by this we mean a hierarchy of concepts, ordered in the structure of a directed acyclic graph (DAG)) - presented in tabular form as described below - and the text of a document, this package can be used to tag the document with the best-fitting taxonomy concepts.
+
+The table used to specify the taxonomy is expected to be a Pandas dataframe. Thus, it can be imported from several tabular sources like csv files or Excel sheets. An example is the file `media.xlsx` included in this repository. It is based on the [IPTC media topics taxonomy](https://iptc.org/standards/media-topics/), a taxonomy frequently used for subject classification of news articles.
+
+The following short excerpt illustrates the structure:
+
+| key       | label_en                | description_en                                                                                                                  | label_de              | description_de                                                                                                                                                              | parents       |
+|-----------|-------------------------|---------------------------------------------------------------------------------------------------------------------------------|-----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| M00000000 | [TOP]                   |                                                                                                                                 | [TOP]                 |                                                                                                                                                                             | []            |
+| M05000000 | education               | All aspects of furthering knowledge, formally or informally                                                                     | Erziehung, Ausbildung | Alle Aspekte der Förderung von Wissen für den Menschen von der Geburt bis zum Tod.                                                                                          | ['M00000000'] |
+| M06000000 | environment             | All aspects of protection, damage, and condition of the ecosystem of the planet earth and its surroundings.                     | Umwelt                | Alle Aspekte des Schutzes, der Zerstörung und die Bedingungen des Ökosystems der Erde und ihrer Umgebung                                                                    | ['M00000000'] |
+| M07000000 | health                  | All aspects pertaining to the physical and mental welfare of living beings                                                      | Medizin, Gesundheit   | Alle Aspekte des körperlichen und geistigen Wohlbefindens von Menschen.                                                                                                     | ['M00000000'] |
+| M20000400 | school                  | A building or institution in which education is provided                                                                        | Schule                | Gebäude oder Einrichtung, wo verschiedenartige Ausbildung angeboten wird.                                                                                                   | ['M05000000'] |
+| M20000405 | college and university  | Institutions of higher learning that provide degrees in a course of study at the undergraduate, postgraduate or doctorate level | Hochschule            | Ausbildung oberhalb des zweiten Levels, meist erwartet von Fachhochschulen und Universitäten, die in einer professionellen Qualifikation oder einem akademischen Grad führt | ['M20000400'] |
+| M20000406 | college                 | Institution of higher learning, its specific scope is set differently in different countries                                    | Fachhochschule        | Institution der Lehre auf höherer Ebene, spezielle Form variiert von Land zu Land                                                                                           | ['M20000405'] |
+| M20000407 | university              | Institutions of higher learning capable of providing doctorate degrees                                                          | Universität           | Institution der Lehre auf höherer Ebene, geeignet um Doktoren- und Professorentitel etc. zu verleihen.                                                                      | ['M20000405'] |
+| M20000446 | diseases and conditions | Any harmful or destructive health condition affecting humans.                                                                   | Krankheit             | Jeder gesundheitsgefährdende oder zerstörerische Zustand                                                                                                                    | ['M07000000'] |
+
+
+The table has one row for every node (concept) of the taxonomy. Formally, there is one special node needed, the so called root or top node, which is the ancestor of all other nodes in the hierarchy. (See the node with key `M00000000` in the media example.) 
+
+Each row of the table has to specify:
+- the concept's unique `key`
+- language dependent labels for the concept
+- short language dependent descriptions of the concept
+- the keys of the hierarchical `parents` of the concept 
+
+The parents' keys have to be given as a string that evaluates to a list in Python (e.g., `"['concept1', 'concept2']"`). In the case of a hierarchy that is a tree, every concept has of course only one parent.
+
+Given a 2-letter ISO code `xy` for a language, the row headers for labels and descriptions are `label_xy` and `description_xy`, respectively. The table is expected to have English entries (`en`) as default, and more entries for the other languages needed. The code in this package is also meant to handle German (`de`) in addition to English; other languages need small obvious modifications.
+
+Tagging is based on semantic similarity considerations as calculated by a Sentence Transformer model. Any model provided by the sentence_transformers package can be used. The small model `all-MiniLM-L6-v2` works well for English (but not so well for other languages).
+
+The code fragment below shows the principles of usage, finally executing the tagging by calling the method `identify_concepts` which has several tuning parameters which can be adapted to specific constellations and use cases. 
+
+```python
+import networkx as nx
+import pandas as pd
+from sentence_transformers import SentenceTransformer
+
+import taxo
+
+# Build a taxonomy from a dataframe (one row per concept).
+df = pd.read_excel("media.xlsx")
+df.fillna("", inplace=True)
+
+tax = taxo.ConceptTaxonomy(
+    nx.DiGraph(),
+    root="M00000000",          # key of the formal top concept
+    transformer="all-MiniLM-L6-v2",
+    default_lang="en",
+)
+tax.from_df(df)
+
+# Tag a document.
+doc = taxo.TaxedDocument(tax, text)
+result = doc.identify_concepts(
+    top_k=10,               # concepts kept when checking similarity to text
+    min_similarity=0.25,    # minimum sentence-level similarity to be kept
+    title_weight=2,         # weight of the first sentence (the document title)
+    saturation=1.5,         # saturation parameter for score aggregation
+    min_agg=0.25,           # minimum aggregated score for a concept to be returned
+    top_percent=80,         # percentage kept when truncating the hierarchy on the top level
+    second_percent=80,      # percentage kept when truncating the hierarchy on the second level
+    max_baseconcepts=7,     # maximum number of returned (leaf) concepts
+    max_paths=2,            # number of paths to the root considered per concept
+    with_scores=True,       # include aggregated scores in the results
+)
+```
+
+Each entry of `result` is a hierarchical path from the root to a concept, as a
+list of `(name, key)` pairs (optionally with an aggregated score appended).
+
+See also the included Jupyter notebook `usage.ipynb`.
+
+## API
+
+### `ConceptTaxonomy`
+
+Holds the concept hierarchy and its embeddings.
+
+- `__init__(taxgraph, root, long_descr=True, transformer=None, default_lang="en", classifier=None, token=None)`
+- `from_df(df_concepts, rename_columns=None)` — add/update concepts from a dataframe
+- `to_df()` — export the hierarchy to a dataframe
+- `describe_concepts(lang=None, intro=None, long=True)` — prepare comparison sentences
+- `embed_concepts(model=None)` — compute the concept embeddings
+- `get_concepts_in_text(text, top_k=10, threshold=0.3, sim_func=None)` — similarity or zero-shot search
+- `get_parent_concepts(child, top_parent=None, lang=None)` — paths from a concept to the root
+- `show_concepts(concepts, lang=None, scores=None)` — format results for output
+
+### `TaxedDocument`
+
+Prepares a document for concept identification.
+
+- `set_parameters(top_k=10, threshold=0.3, weight_dev=None, kparm=1.0)`
+- `score_concepts_overall()` / `score_concepts_sentencewise()`
+- `aggregate_sentence_scores(min_agg_score=0)`
+- `find_best_paths(concept, n)`
+- `evaluate_hierarchy(max_paths=2)` / `restrict_hierarchy(level, percentage)`
+- `get_relevant_concepts(min_score=None, max_number=100)`
+- `identify_concepts(...)` — all of the above in one call
+
+
+## License
+
+MIT — DLR. See LICENSE file.

taxo-1.0.0/README.md

@@ -0,0 +1,145 @@
+# taxo
+
+Tag documents with terms from a concept hierarchy (taxonomy) using sentence
+embeddings.
+
+`taxo` reads in a hierarchy of concepts, embeds their descriptions in a shared
+vector space, and then finds the concepts most related to a document. It can
+also aggregate sentence-level evidence along hierarchical paths to produce a
+consistent set of top-level tags.
+
+For the main ideas used in `taxo`, see
+```
+@inproceedings{dlr190966,
+           title = {Strategy Comparison for Semantic Zero-Shot Taxonomy Filters},
+           month = {October},
+       booktitle = {4th International Open Search Symposium (OSSYM 2022)},
+          author = {Hamm, Andreas},
+            year = {2022},
+             url = {https://elib.dlr.de/190966/}
+}
+```
+
+## Installation
+
+On Python >= 3.11:
+
+```bash
+pip install taxo
+```
+
+This is a uv-style package, so `uv` works too:
+
+```bash
+uv add taxo
+```
+
+## Usage
+The package is both installed and imported as `taxo`.
+
+Given a taxonomy (by this we mean a hierarchy of concepts, ordered in the structure of a directed acyclic graph (DAG)) - presented in tabular form as described below - and the text of a document, this package can be used to tag the document with the best-fitting taxonomy concepts.
+
+The table used to specify the taxonomy is expected to be a Pandas dataframe. Thus, it can be imported from several tabular sources like csv files or Excel sheets. An example is the file `media.xlsx` included in this repository. It is based on the [IPTC media topics taxonomy](https://iptc.org/standards/media-topics/), a taxonomy frequently used for subject classification of news articles.
+
+The following short excerpt illustrates the structure:
+
+| key       | label_en                | description_en                                                                                                                  | label_de              | description_de                                                                                                                                                              | parents       |
+|-----------|-------------------------|---------------------------------------------------------------------------------------------------------------------------------|-----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| M00000000 | [TOP]                   |                                                                                                                                 | [TOP]                 |                                                                                                                                                                             | []            |
+| M05000000 | education               | All aspects of furthering knowledge, formally or informally                                                                     | Erziehung, Ausbildung | Alle Aspekte der Förderung von Wissen für den Menschen von der Geburt bis zum Tod.                                                                                          | ['M00000000'] |
+| M06000000 | environment             | All aspects of protection, damage, and condition of the ecosystem of the planet earth and its surroundings.                     | Umwelt                | Alle Aspekte des Schutzes, der Zerstörung und die Bedingungen des Ökosystems der Erde und ihrer Umgebung                                                                    | ['M00000000'] |
+| M07000000 | health                  | All aspects pertaining to the physical and mental welfare of living beings                                                      | Medizin, Gesundheit   | Alle Aspekte des körperlichen und geistigen Wohlbefindens von Menschen.                                                                                                     | ['M00000000'] |
+| M20000400 | school                  | A building or institution in which education is provided                                                                        | Schule                | Gebäude oder Einrichtung, wo verschiedenartige Ausbildung angeboten wird.                                                                                                   | ['M05000000'] |
+| M20000405 | college and university  | Institutions of higher learning that provide degrees in a course of study at the undergraduate, postgraduate or doctorate level | Hochschule            | Ausbildung oberhalb des zweiten Levels, meist erwartet von Fachhochschulen und Universitäten, die in einer professionellen Qualifikation oder einem akademischen Grad führt | ['M20000400'] |
+| M20000406 | college                 | Institution of higher learning, its specific scope is set differently in different countries                                    | Fachhochschule        | Institution der Lehre auf höherer Ebene, spezielle Form variiert von Land zu Land                                                                                           | ['M20000405'] |
+| M20000407 | university              | Institutions of higher learning capable of providing doctorate degrees                                                          | Universität           | Institution der Lehre auf höherer Ebene, geeignet um Doktoren- und Professorentitel etc. zu verleihen.                                                                      | ['M20000405'] |
+| M20000446 | diseases and conditions | Any harmful or destructive health condition affecting humans.                                                                   | Krankheit             | Jeder gesundheitsgefährdende oder zerstörerische Zustand                                                                                                                    | ['M07000000'] |
+
+
+The table has one row for every node (concept) of the taxonomy. Formally, there is one special node needed, the so called root or top node, which is the ancestor of all other nodes in the hierarchy. (See the node with key `M00000000` in the media example.) 
+
+Each row of the table has to specify:
+- the concept's unique `key`
+- language dependent labels for the concept
+- short language dependent descriptions of the concept
+- the keys of the hierarchical `parents` of the concept 
+
+The parents' keys have to be given as a string that evaluates to a list in Python (e.g., `"['concept1', 'concept2']"`). In the case of a hierarchy that is a tree, every concept has of course only one parent.
+
+Given a 2-letter ISO code `xy` for a language, the row headers for labels and descriptions are `label_xy` and `description_xy`, respectively. The table is expected to have English entries (`en`) as default, and more entries for the other languages needed. The code in this package is also meant to handle German (`de`) in addition to English; other languages need small obvious modifications.
+
+Tagging is based on semantic similarity considerations as calculated by a Sentence Transformer model. Any model provided by the sentence_transformers package can be used. The small model `all-MiniLM-L6-v2` works well for English (but not so well for other languages).
+
+The code fragment below shows the principles of usage, finally executing the tagging by calling the method `identify_concepts` which has several tuning parameters which can be adapted to specific constellations and use cases. 
+
+```python
+import networkx as nx
+import pandas as pd
+from sentence_transformers import SentenceTransformer
+
+import taxo
+
+# Build a taxonomy from a dataframe (one row per concept).
+df = pd.read_excel("media.xlsx")
+df.fillna("", inplace=True)
+
+tax = taxo.ConceptTaxonomy(
+    nx.DiGraph(),
+    root="M00000000",          # key of the formal top concept
+    transformer="all-MiniLM-L6-v2",
+    default_lang="en",
+)
+tax.from_df(df)
+
+# Tag a document.
+doc = taxo.TaxedDocument(tax, text)
+result = doc.identify_concepts(
+    top_k=10,               # concepts kept when checking similarity to text
+    min_similarity=0.25,    # minimum sentence-level similarity to be kept
+    title_weight=2,         # weight of the first sentence (the document title)
+    saturation=1.5,         # saturation parameter for score aggregation
+    min_agg=0.25,           # minimum aggregated score for a concept to be returned
+    top_percent=80,         # percentage kept when truncating the hierarchy on the top level
+    second_percent=80,      # percentage kept when truncating the hierarchy on the second level
+    max_baseconcepts=7,     # maximum number of returned (leaf) concepts
+    max_paths=2,            # number of paths to the root considered per concept
+    with_scores=True,       # include aggregated scores in the results
+)
+```
+
+Each entry of `result` is a hierarchical path from the root to a concept, as a
+list of `(name, key)` pairs (optionally with an aggregated score appended).
+
+See also the included Jupyter notebook `usage.ipynb`.
+
+## API
+
+### `ConceptTaxonomy`
+
+Holds the concept hierarchy and its embeddings.
+
+- `__init__(taxgraph, root, long_descr=True, transformer=None, default_lang="en", classifier=None, token=None)`
+- `from_df(df_concepts, rename_columns=None)` — add/update concepts from a dataframe
+- `to_df()` — export the hierarchy to a dataframe
+- `describe_concepts(lang=None, intro=None, long=True)` — prepare comparison sentences
+- `embed_concepts(model=None)` — compute the concept embeddings
+- `get_concepts_in_text(text, top_k=10, threshold=0.3, sim_func=None)` — similarity or zero-shot search
+- `get_parent_concepts(child, top_parent=None, lang=None)` — paths from a concept to the root
+- `show_concepts(concepts, lang=None, scores=None)` — format results for output
+
+### `TaxedDocument`
+
+Prepares a document for concept identification.
+
+- `set_parameters(top_k=10, threshold=0.3, weight_dev=None, kparm=1.0)`
+- `score_concepts_overall()` / `score_concepts_sentencewise()`
+- `aggregate_sentence_scores(min_agg_score=0)`
+- `find_best_paths(concept, n)`
+- `evaluate_hierarchy(max_paths=2)` / `restrict_hierarchy(level, percentage)`
+- `get_relevant_concepts(min_score=None, max_number=100)`
+- `identify_concepts(...)` — all of the above in one call
+
+
+## License
+
+MIT — DLR. See LICENSE file.

taxo-1.0.0/pyproject.toml

@@ -0,0 +1,59 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "taxo"
+version = "1.0.0"
+description = "Tag documents with taxonomy terms using embeddings and a concept hierarchy"
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "MIT" }
+authors = [
+    { name = "Andreas Hamm", email = "andreas.hamm@dlr.de" },
+]
+keywords = ["taxonomy", "document-tagging", "embeddings", "sentence-transformers", "hierarchy"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "networkx>=3.4.2",
+    "nltk>=3.9.1",
+    "numpy>=2.2.5",
+    "pandas>=2.2.3",
+    "sentence_transformers>=4.1.0",
+    "scipy>=1.14.1",
+    "openpyxl>=3.1.5",
+    "torch>=2.6.0",
+    "transformers>=4.46.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "build",
+    "twine",
+    "pytest",
+]
+
+[project.urls]
+Homepage = "https://github.com/DLR-SC/taxo-tag"
+Repository = "https://github.com/DLR-SC/taxo-tag"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/taxo"]
+
+[tool.hatch.build.targets.sdist]
+include = [
+    "src/",
+    "README.md",
+    "LICENSE",
+    "CITATION.cff",
+    "pyproject.toml",
+    "requirements.txt",
+    "usage.ipynb",
+]

taxo-1.0.0/requirements.txt

@@ -0,0 +1,9 @@
+networkx>=3.4.2
+nltk>=3.9.1
+numpy>=2.2.5
+pandas>=2.2.3
+sentence_transformers>=4.1.0
+scipy>=1.14.1
+openpyxl>=3.1.5
+torch>=2.6.0
+transformers>=4.46.0

taxo-1.0.0/src/taxo/__init__.py

@@ -0,0 +1,533 @@
+"""Python module for tagging documents with taxonomy terms
+Author: Andreas Hamm (andreas.hamm@dlr.de)
+Version: 1.0.0
+Copyright: DLR
+License: MIT
+"""
+
+
+from collections import defaultdict
+from typing import Any, Callable, Dict, Iterator, List, Optional, Tuple, Union
+import nltk
+from nltk.tokenize import sent_tokenize
+import pandas as pd
+import numpy as np
+import networkx as nx
+from sentence_transformers import SentenceTransformer, util
+from transformers import pipeline
+import ast
+
+# Maps two-letter ISO codes to the punkt_tab language directory names NLTK's
+# Punkt tokenizer loads. These must match the directory names on disk
+# (e.g. "german", "english"): punkt.py does a plain ``find("tokenizers/punkt_tab/{lang}/")``
+# with no case normalization, so capitalization must match the filesystem
+# (which is fine on Windows but would fail on case-sensitive Linux/macOS).
+# Only models actually installed as ``punkt_tab`` will work.
+_ISO_TO_NLTK_LANGUAGE = {
+    "en": "english",
+    "de": "german",
+}
+
+
+def _ensure_punkt(language: str = "english") -> bool:
+    """Ensures the punkt_tab tokenizer for ``language`` is available.
+
+    NLTK only downloads punkt_tab lazily and raises ``LookupError`` otherwise.
+    On first use it attempts a silent download of ``punkt_tab`` and retries.
+
+    Args:
+        language (str): The punkt_tab language directory name, e.g. ``"english"`` or ``"german"``.
+
+    Returns:
+        bool: ``True`` if the tokenizer is available afterwards, ``False`` otherwise.
+    """
+    try:
+        nltk.data.find(f"tokenizers/punkt_tab/{language}")
+        return True
+    except LookupError:
+        try:
+            return bool(nltk.download("punkt_tab", quiet=True))
+        except Exception:
+            return False
+
+class ConceptTaxonomy:
+    """Class for working with a concept hierarchy including concept embeddings
+
+    Attributes:
+        default_lang (str): Default language (two-letter ISO code of a supported language)
+        tg (networkx.DiGraph): Graph defining the concept hierarchy. The concepts are nodes with attributes
+                               - ``key``
+                               - ``label_{isolang}``
+                               - ``description_{isolang}``
+                               where ``isolang`` is the two-letter ISO code of each supported language (e.g., ``"en"``, ``"de"``).
+                               The edges point from concept nodes to their parents.
+        root (str): Key of the formal root node.
+        transformer (sentence_transformers.SentenceTransformer): SentenceTransformer model used for initial embeddings (only set if ``transformer`` was passed).
+        classifier (None or transformers.pipeline): Optionally a Huggingface zero-shot-classification pipeline.
+        desc_lang (str): Current language of concept descriptions (two-letter ISO code of a supported language).
+        desc_concepts (List[Tuple[str, str, str]]): List of ``(key, label, sentence)`` triples for all concepts, with labels and
+                                          descriptive sentences in the current language.
+        model (sentence_transformers.SentenceTransformer): SentenceTransformer model used for current embeddings (only set if ``transformer`` was passed).
+        emb_concepts (Optional[numpy.ndarray]): Row-normalized embedding vectors for all concepts in ``desc_concepts``.
+        token (None or str): Optionally Huggingface Access token.
+    """
+
+    def __init__(
+        self,
+        taxgraph: nx.DiGraph,
+        root: str,
+        long_descr: bool = True,
+        transformer: Optional[Union[str, SentenceTransformer]] = None,
+        default_lang: str = "en",
+        classifier: Optional[str] = None,
+        token: Optional[str] = None,
+    ):
+        """Constructs a ``ConceptTaxonomy``, optionally computing an initial embedding.
+
+        Args:
+            taxgraph (networkx.DiGraph): Graph defining the concept hierarchy. The concepts are nodes with attributes
+                                         - ``key``
+                                         - ``label_{isolang}``
+                                         - ``description_{isolang}``
+                                         where ``isolang`` is the two-letter ISO code of each supported language (e.g., ``"en"``, ``"de"``)
+            root (str): Key of the formal root node. Must be a node in ``taxgraph``.
+            long_descr (bool, optional): Whether to include the concept descriptions in the concept comparison sentences. Defaults to ``True``.
+            transformer (Union[str, SentenceTransformer], optional): SentenceTransformer model (or its name) used for initial embeddings. Defaults to ``None``.
+            default_lang (str, optional): Default language (two-letter ISO code of a supported language). Defaults to ``"en"``.
+            classifier (str, optional): Optionally a Huggingface zero-shot-classification model name. Defaults to ``None``.
+            token (str, optional): Huggingface access token. Defaults to ``None``.
+
+        Raises:
+            ValueError: If ``root`` is not a node in ``taxgraph``.
+        """
+        self.default_lang = default_lang
+        self.tg = taxgraph
+        self.root = root
+        self.token = token
+        # Ensure the punkt_tab sentence tokenizer for the default language is
+        # available, since score_concepts_sentencewise depends on it.
+        _ensure_punkt(_ISO_TO_NLTK_LANGUAGE.get(self.default_lang, "english"))
+        if long_descr:
+            self.describe_concepts()
+        else:
+            self.describe_concepts(long=False)
+        if transformer:
+            self.transformer = transformer
+            self.embed_concepts(transformer)
+        if classifier:
+            self.classifier = pipeline("zero-shot-classification", model=classifier)
+        else:
+            self.classifier = None
+
+    def from_df(
+        self,
+        df_concepts: pd.DataFrame,
+        rename_columns: Optional[Dict[str, str]] = None,
+    ) -> "ConceptTaxonomy":
+        """Reads in new concepts from a dataframe.
+
+        Args:
+            df_concepts (pandas.DataFrame): Dataframe. Each row contains a node to be added to the graph defining the concept hierarchy.
+                                            The dataframe has to have the columns
+                                            - ``key`` for the concept key
+                                            - ``parents``, a list with the parent keys
+                                            - ``label_xx`` with ``xx`` the default language for the concept label
+                                            - ``description_xx`` with ``xx`` the default language for the concept description (might be empty)
+                                            and optionally further ``label_yz``, ``description_yz`` for other supported languages, and also other attributes
+            rename_columns (dict, optional): Dictionary for optionally renaming column headings. Defaults to ``{}``.
+
+        Returns:
+            ConceptTaxonomy: This instance, to allow method chaining.
+        """
+        if rename_columns is None:
+            rename_columns = {}
+        if rename_columns:
+            df_concepts = df_concepts.rename(columns=rename_columns)
+        if len(set(df_concepts.keys()) & set(["key", f"label_{self.default_lang}", f"description_{self.default_lang}", "parents"])) < 4:
+            self.describe_concepts()
+            if self.transformer:
+                self.embed_concepts()
+            return self
+        for index, row in df_concepts.iterrows():
+            self.tg.add_node(row["key"], **{k: v for k, v in row.items() if k not in ["key", "parents"]})
+        for index, row in df_concepts.iterrows():
+            for p in ast.literal_eval(row["parents"]):
+                self.tg.add_edge(row["key"], p)
+        # Refresh the derived concept/embedding caches, since the nodes above
+        # changed but the caches from __init__ did not.
+        self.describe_concepts()
+        if self.transformer:
+            self.embed_concepts()
+        return self
+
+    def to_df(self) -> pd.DataFrame:
+        """Writes the concept hierarchy into a dataframe.
+
+        Returns:
+            pandas.DataFrame: Dataframe. Each row contains one concept. The columns are ``key``, all attributes of the concept nodes, and ``parents``.
+        """
+        records = []
+        for conc, attr in self.tg.nodes.items():
+            records.append({
+                "key": conc,
+                **attr,
+                "parents": [p for p in self.tg.predecessors(conc)]
+            })
+        return pd.DataFrame(records)
+
+
+    def describe_concepts(
+        self,
+        lang: Optional[str] = None,
+        intro: Optional[str] = None,
+        long: bool = True,
+    ) -> None:
+        """Fills the attributes :pyattr:`desc_concepts` (a list of ``(key, label, sentence)`` triples) and :pyattr:`desc_lang`.
+
+        Args:
+            lang (str, optional): Two-letter ISO code for language to use for concept names and descriptions. Defaults to ``None``, which means ``self.default_lang``.
+            intro (str, optional): Phrase for forming concept comparison sentences. Defaults to ``'This text is about '`` (which gets translated into ``'Dieser Text handelt von '`` in German).
+            long (bool, optional): Whether to include the concept descriptions in the concept comparison sentences. Defaults to ``True``.
+        """
+        if lang is None:
+            lang = self.default_lang
+        if intro is None:
+            if lang == "de":
+                intro = "Dieser Text handelt von "
+            elif lang == "en":
+                intro = "This text is about "
+            else:
+                intro = ""
+        self.desc_lang = lang
+        self.desc_concepts = []
+        for ckey, cattrib in self.tg.nodes.items():
+            label = cattrib.get(f"label_{lang}", cattrib.get("label_en"))
+            if label is None:
+                continue
+            concept_sent = intro + label
+            if long:
+                description = cattrib.get(f"description_{lang}", cattrib.get("description_en"))
+                if description:
+                    concept_sent += " (" + description + ")"
+            self.desc_concepts.append((ckey, label, concept_sent))
+
+    def embed_concepts(self, model: Optional[SentenceTransformer] = None) -> None:
+        """Fills the attribute :pyattr:`emb_concepts` (the embedding of all concept comparison sentences) and sets :pyattr:`model`.
+        On large taxonomies, this method will have a considerable execution time.
+
+        Args:
+            model (sentence_transformers.SentenceTransformer, optional): SentenceTransformer model to be used for embeddings. Defaults to ``None``, which reuses ``self.transformer``.
+        """
+        if model is None:
+            model = self.transformer
+        self.model = model
+        if len(self.desc_concepts) == 0:
+            self.emb_concepts = []
+            return
+        emb_sents = model.encode([x[2] for x in self.desc_concepts], convert_to_numpy=True)
+        self.emb_concepts = emb_sents / np.linalg.norm(emb_sents, axis=1)[:, None]
+
+    def get_concepts_in_text(
+        self,
+        text: str,
+        top_k: int = 10,
+        threshold: float = 0.3,
+        sim_func: Optional[Callable[[Any, Any], Any]] = None,
+    ) -> List[Tuple[str, str, float]]:
+        """Similarity search for taxonomy concepts in a given text.
+        Uses either a zero-shot classification transformers pipeline (given as optional class attribute ``classifier``) or, by default, cosine similarity in the SentenceTransformer model's embedding space.
+
+        Args:
+            text (str): Text.
+            top_k (int, optional): Maximum number of concepts returned. Defaults to ``10``.
+            threshold (float, optional): Minimum similarity threshold. Defaults to ``0.3``.
+            sim_func (Callable[[Tensor, Tensor], Tensor], optional): Similarity function passed to ``util.semantic_search``. Defaults to ``None``, which means cosine similarity.
+
+        Returns:
+            list: List of found concepts as ``(concept key, concept name, similarity score)`` triples, sorted by descending score.
+        """
+        if len(self.desc_concepts) == 0:
+            return []
+        if self.classifier:
+            premises = [cd[2] for cd in self.desc_concepts]
+            clion = self.classifier(text, premises, hypothesis_template="{}", multi_class=True)
+            scored = [(cd[0], cd[1], score)
+                      for cd, score in zip(self.desc_concepts, clion["scores"])
+                      if score > threshold]
+            return sorted(scored, key=lambda z: -z[2])[:top_k]
+        emb_sent = self.model.encode(text, convert_to_numpy=True)
+        if sim_func is not None:
+            return [(self.desc_concepts[x["corpus_id"]][0], self.desc_concepts[x["corpus_id"]][1], x["score"])
+                    for x in util.semantic_search(emb_sent, self.emb_concepts, top_k=top_k, score_function=sim_func)[0] if x["score"] > threshold]
+        return [(self.desc_concepts[x["corpus_id"]][0], self.desc_concepts[x["corpus_id"]][1], x["score"])
+                for x in util.semantic_search(emb_sent, self.emb_concepts, top_k=top_k)[0] if x["score"] > threshold]
+
+    def get_parent_concepts(
+        self,
+        child: str,
+        top_parent: Optional[str] = None,
+        lang: Optional[str] = None,
+    ) -> Iterator[List[Union[str, Tuple[str, str, str]]]]:
+        """Returns paths of predecessors of a given child concept, up back to a ``top_parent``.
+
+        Args:
+            child (str): Key of child concept.
+            top_parent (str, optional): Key of earliest ancestor. Defaults to ``None``, in which case it is the root node of the hierarchy.
+            lang (str, optional): Two-letter ISO code for language to use for concept names and descriptions. Defaults to ``None``, in which case just the concept keys are returned.
+
+        Returns:
+            list: An iterator of paths to ``top_parent``, each path being a list of concepts. If ``lang`` is ``None``, each path is a list of concept keys; otherwise each concept is a ``(key, name, description)`` triple.
+        """
+        if top_parent is None:
+            top_parent = self.root
+        paths = nx.all_simple_paths(self.tg, target=top_parent, source=child)
+        if lang is None:
+            return paths
+        return [[(k, self.tg.nodes[k][f"label_{lang}"], self.tg.nodes[k][f"description_{lang}"]) for k in p] for p in paths]
+
+    def show_concepts(
+        self,
+        concepts: List[List[str]],
+        lang: Optional[str] = None,
+        scores: Optional[List[float]] = None,
+    ) -> List[List[Union[Tuple[str, str], Tuple[str, str, float]]]]:
+        """Convenience function for producing output of concept results.
+
+        Args:
+            concepts (list): List of hierarchy paths leading to concepts, each path represented as a list of concept keys.
+            lang (str, optional): Two-letter ISO code for language to use for concept names and descriptions. Defaults to ``None``, which means ``self.default_lang``.
+            scores (list, optional): List of scores along a hierarchical concept path. Defaults to ``None``.
+
+        Returns:
+            list: List of lists of ``(concept name, concept key)`` pairs, or ``(concept name, concept key, aggregated score)`` triples when ``scores`` is given.
+        """
+        if lang is None:
+            lang = self.default_lang
+        if scores is None:
+            return [
+                [(self.tg.nodes[ck].get(f"label_{lang}", ck), ck) for ck in path]
+                for path in concepts
+            ]
+        return [
+            [(self.tg.nodes[ck].get(f"label_{lang}", ck), ck, sc) for ck in path]
+            for path, sc in zip(concepts, scores)
+        ]
+
+
+class TaxedDocument:
+    """Class for working with a document to detect related concepts of a taxonomy in it
+
+    Attributes:
+        taxonomy (ConceptTaxonomy): The taxonomy from which the concept tags to the document are taken
+        doc_text (str): Text of the document
+        overall_concepts (Dict[str, float]): Concept scores for whole text (Key: Concept key, Value: Score)
+        sentences_concepts (Dict[str, List[float]]): Sentencewise concept scores (Key: Concept key, Value: List of sentence scores)
+        aggregated_concepts (Dict[str, float]): Aggregated concept scores (Key: Concept key, Value: Score)
+        hierarchy_distribution (Dict[int, Dict[str, float]]): Aggregated concept scores summed up along hierarchical upward paths.
+                                                               Key: Hierarchy level, Value: Dict with Key: Concept key, Value: Summed-up score divided by path length
+        hierarchy_restriction (Dict[int, List[str]]): Consistency-restricted concepts on hierarchical levels.
+                                                       Key: Hierarchical level, Value: Restricted list of concepts that make up the required percentage of scores
+        top_k (int): Current number of concepts retained when checking similarity to text
+        threshold (float): Current minimum similarity for considering concepts
+        weight_dev (Dict[int, float]): Dictionary for setting sentence weights different from 1 (Key: sentence number, Value: Sentence weight)
+        kparm (float): Current saturation parameter for aggregating similarities
+    """
+
+    def __init__(self, taxonomy: ConceptTaxonomy, doc_text: str):
+        """Constructor for TaxedDocument.
+
+        Args:
+            taxonomy (ConceptTaxonomy): The taxonomy to use.
+            doc_text (str): Text of the document.
+        """
+        self.taxonomy = taxonomy
+        self.doc_text = doc_text
+        self.overall_concepts = None
+        self.sentences_concepts = None
+        self.aggregated_concepts = None
+        self.hierarchy_distribution = None
+        self.hierarchy_restriction = {}
+        self.set_parameters()
+
+    def set_parameters(
+        self,
+        top_k: int = 10,
+        threshold: float = 0.3,
+        weight_dev: Optional[Dict[int, float]] = None,
+        kparm: float = 1.0,
+    ) -> None:
+        """Set the parameters for taxonomy concept assignment to document text.
+
+        Args:
+            top_k (int, optional): Number of concepts retained when checking similarity to text. Defaults to 10.
+            threshold (float, optional): Minimum similarity for considering concepts. Defaults to 0.3.
+            weight_dev (Dict[int, float], optional): Dictionary for setting sentence weights different from 1. Defaults to ``{0: 1.5}``.
+            kparm (float, optional): Saturation parameter for aggregating similarities. Defaults to 1.0.
+        """
+        if weight_dev is None:
+            weight_dev = {0: 1.5}
+        self.top_k = top_k
+        self.threshold = threshold
+        self.weight_dev = weight_dev
+        self.kparm = kparm
+
+    def score_concepts_overall(self) -> None:
+        """Finds concepts by simple similarity search for the whole text and stores them in attribute :pyattr:`overall_concepts`
+        as a dict with concept key as key and similarity score as value.
+        """
+        self.overall_concepts = {
+            x[0]: x[2]
+            for x in self.taxonomy.get_concepts_in_text(self.doc_text, top_k=self.top_k, threshold=self.threshold)
+        }
+
+    def score_concepts_sentencewise(self) -> None:
+        """Finds concepts by sentencewise similarity search and stores them in attribute :pyattr:`sentences_concepts`
+        as a dict with concept key as key and a list of similarity scores for affected sentences, taking ``weight_dev`` into consideration.
+        """
+        language = _ISO_TO_NLTK_LANGUAGE.get(self.taxonomy.default_lang, "english")
+        sentences = sent_tokenize(self.doc_text, language=language)
+        weights = [1] * len(sentences)
+        for i, g in self.weight_dev.items():
+            if i < len(sentences):
+                weights[i] = g
+        concepts_scores = defaultdict(list)
+        for w, s in zip(weights, sentences):
+            for c in self.taxonomy.get_concepts_in_text(s, top_k=self.top_k, threshold=self.threshold):
+                concepts_scores[c[0]].append(w * c[2])
+        self.sentences_concepts = dict(concepts_scores)
+
+    def aggregate_sentence_scores(self, min_agg_score: float = 0) -> None:
+        """Identifies those of the concepts found by sentencewise similarity search which exceed a certain score after score aggregation
+        and stores them in attribute :pyattr:`aggregated_concepts` as a dict with concept key as key and aggregated score as value.
+
+        Args:
+            min_agg_score (float, optional): Minimum score after sentence aggregation. Defaults to 0.
+        """
+        self.score_concepts_sentencewise()
+        a_concepts = {}
+        for c, v in self.sentences_concepts.items():
+            a_score = 0
+            for i, score in enumerate(sorted(v, reverse=True)):
+                a_score += score * self.kparm / (i + self.kparm) / (i + 1 + self.kparm)
+            if a_score >= min_agg_score:
+                a_concepts[c] = a_score
+        self.aggregated_concepts = a_concepts
+
+    def find_best_paths(self, concept: str, n: int) -> List[List[str]]:
+        """Finds paths from a given concept to the root of the taxonomy hierarchy along which the mean of aggregated concept scores is greatest.
+
+        Args:
+            concept (str): Key of concept.
+            n (int): Number of paths to be returned.
+
+        Returns:
+            list: List of paths, each path being a list of concept keys.
+        """
+        scored_paths = [(p, sum([self.aggregated_concepts.get(ci, 0) for ci in p]) / len(p))
+            for p in self.taxonomy.get_parent_concepts(concept)
+        ]
+        sorted_paths = sorted(scored_paths, key=lambda x: x[1], reverse=True)
+        return [x[0] for x in sorted_paths[:n]]
+
+    def evaluate_hierarchy(self, max_paths: int = 2) -> None:
+        """Shares aggregated scores of concepts with their hierarchical predecessors and stores the results in attribute :pyattr:`hierarchy_distribution` as
+        a dict of dicts, where the outer keys are hierarchy levels and the inner keys are concept keys, followed by the summed-up score shares.
+
+        Args:
+            max_paths (int, optional): Number of paths to the root considered per concept. Defaults to 2.
+        """
+        levels = defaultdict(lambda: defaultdict(float))
+        for c, asc in self.aggregated_concepts.items():
+            cps = self.find_best_paths(c, max_paths)
+            for cp in cps:
+                for i, ci in enumerate(reversed(cp[:-1])):
+                    levels[i][ci] += asc / len(cps)
+        self.hierarchy_distribution = {i: {c: v for c, v in cs.items()} for i, cs in levels.items()}
+
+    def restrict_hierarchy(self, level: int, percentage: float) -> None:
+        """Forms truncated lists of highest-scored concepts of a given hierarchy level such that the summed-up hierarchical scores of the remaining
+        concepts exceed a given percentage of the total summed-up score, and stores the results in attribute :pyattr:`hierarchy_restriction` as a dict
+        with the level number as key and the truncated list of concept keys as value.
+
+        Args:
+            level (int): Hierarchy level.
+            percentage (float): Percentage of the total summed-up score that the remaining concepts must exceed.
+        """
+        if self.hierarchy_distribution.get(level):
+            sum_list = []
+            list_sum = 0
+            for elem in sorted(self.hierarchy_distribution[level].items(), key=lambda x: x[1], reverse=True):
+                sum_list.append((elem, list_sum))
+                list_sum += elem[1]
+            self.hierarchy_restriction[level] = [y[0][0] for y in sum_list if y[1] / list_sum * 100 < percentage]
+        else:
+            self.hierarchy_restriction[level] = []
+
+    def get_relevant_concepts(
+        self,
+        min_score: Optional[float] = None,
+        max_number: int = 100,
+    ) -> List[Tuple[List[str], float]]:
+        """Returns the hierarchical paths to all concepts exceeding a minimum aggregated score, as long as the steps of these paths are contained
+        in the truncated level lists of concepts in ``hierarchy_restriction``.
+
+        Args:
+            min_score (float, optional): Minimum aggregated score. Defaults to ``None``, in which case the class attribute ``threshold`` is used.
+            max_number (int, optional): Maximum number of concept paths returned. Defaults to 100.
+
+        Returns:
+            list: List of resulting concept paths, where each concept path is a pair of the list of concept keys constituting the path (root-first) and the aggregated score.
+        """
+        if min_score is None:
+            min_score = self.threshold
+        concept_paths = sorted(
+            [(self.taxonomy.get_parent_concepts(c), s) for c, s in self.aggregated_concepts.items() if s >= min_score],
+            key=lambda x: x[1], reverse=True
+        )[:max_number]
+        concept_paths = [(cp[-2::-1], cps[1]) for cps in concept_paths for cp in cps[0]]
+        for level, rest in self.hierarchy_restriction.items():
+            concept_paths = [cp for cp in concept_paths if set(cp[0][level:level + 1]).issubset(set(rest))]
+        return concept_paths
+
+    def identify_concepts(
+        self,
+        top_k: int,
+        min_similarity: float,
+        title_weight: float,
+        saturation: float,
+        min_agg: float,
+        top_percent: float,
+        second_percent: float,
+        max_baseconcepts: int,
+        max_paths: int,
+        with_scores: bool = False,
+    ) -> List[List[Union[Tuple[str, str], Tuple[str, str, float]]]]:
+        """Convenience function for identifying concepts, given the various parameters of the algorithm.
+
+        Args:
+            top_k (int): Number of concepts retained when checking similarity to text.
+            min_similarity (float): Minimum value for similarity on sentence level.
+            title_weight (float): Weight for first sentence (which is supposed to be the title of the document).
+            saturation (float): Saturation parameter for score aggregation.
+            min_agg (float): Minimum value for aggregated score.
+            top_percent (float): Percentage for hierarchy truncation on top level.
+            second_percent (float): Percentage for hierarchy truncation on second level.
+            max_baseconcepts (int): Maximum number of returned (leaf) concepts.
+            max_paths (int): Number of paths to the root considered per concept.
+            with_scores (bool, optional): Whether to include scores. Defaults to ``False``.
+
+        Returns:
+            list: List of lists of ``(concept name, concept key)`` pairs, or ``(concept name, concept key, aggregated score)`` triples when ``with_scores`` is ``True``.
+        """
+        self.set_parameters(top_k=top_k, threshold=min_similarity, weight_dev={0: title_weight}, kparm=saturation)
+        self.aggregate_sentence_scores(min_agg_score=min_agg)
+        self.evaluate_hierarchy(max_paths=max_paths)
+        self.restrict_hierarchy(0, top_percent)
+        self.restrict_hierarchy(1, second_percent)
+        relevant_concepts_and_scores = self.get_relevant_concepts(min_score=min_agg, max_number=max_baseconcepts)
+        relevant_concepts = [rc[0] for rc in relevant_concepts_and_scores]
+        relevant_scores = [rc[1] for rc in relevant_concepts_and_scores]
+        if not with_scores:
+            return self.taxonomy.show_concepts(relevant_concepts)
+        else:
+            return self.taxonomy.show_concepts(relevant_concepts, scores=relevant_scores)

taxo-1.0.0/usage.ipynb

@@ -0,0 +1,281 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f2ccd734",
+   "metadata": {},
+   "source": "## taxo\n\nThis notebook demonstrates the use of taxo."
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ae9b02d2",
+   "metadata": {},
+   "source": [
+    "First import the packages used."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "19b68fc3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import networkx as nx\n",
+    "from sentence_transformers import SentenceTransformer\n",
+    "\n",
+    "import taxo"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "186799df",
+   "metadata": {},
+   "source": [
+    "Then fill a pandas dataframe with a taxonomy, for instance by loading it from an excel sheet.\n",
+    "\n",
+    "Each row corresponds to a taxonomy concept. Each concept has to have a key, a label_en, a description_en (optionally also label and description in other languages), and a parents attribute, which is a string that interprets as the list of its parent concepts (note that not only trees but also more general DAGs are allowed as taxonomic hierarchies).\n",
+    "\n",
+    "There has to be on special formal concept which is the top parent of all other concepts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "4707d994",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_media_tax = pd.read_excel(\"media.xlsx\")\n",
+    "df_media_tax.fillna(\"\", inplace=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b5bf204a",
+   "metadata": {},
+   "source": [
+    "The example contained in the file `media.xlsx` is extracted from the IPTC media topics taxonomy, a taxonomy often used for subject classification of news articles.\n",
+    "\n",
+    "Note that the top parent node has the key \"M00000000\".\n",
+    "\n",
+    "In the following, we use the embedding model [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "49c013c5",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<taxo.ConceptTaxonomy at 0x21ce1ad3770>"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "media_tax = taxo.ConceptTaxonomy(nx.DiGraph(), \"M00000000\", transformer=SentenceTransformer(\"all-MiniLM-L6-v2\"))\n",
+    "media_tax.from_df(df_media_tax)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a4cc827d",
+   "metadata": {},
+   "source": [
+    "Now, given an example text ..."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "fbd75c2f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = \"\"\"\n",
+    "AI opens up new possibilities for air traffic control and the cockpit.\n",
+    "\n",
+    "How will artificial intelligence (AI) support air traffic controllers and pilots in the future? Over the past four years, \n",
+    "the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt; DLR) has been investigating this question as part \n",
+    "of the LOKI project (Kollaboration von Luftfahrt-OperateurInnen und KI-Systemen, or 'Collaboration between Aviation \n",
+    "Operators and AI Systems' in English). In collaboration with national and international companies and research \n",
+    "institutions from the air traffic control sector, as well as public authorities, the project also developed relevant \n",
+    "recommendations for introducing AI into aviation.\n",
+    "\n",
+    "The aviation industry is currently working intensively on introducing AI systems. Unlike conventional assistance systems, \n",
+    "these can process large volumes of data, be trained on the basis of this data and so adapt flexibly to new situations. \n",
+    "AI systems offer the potential to increase the efficiency of air traffic and relieve the workload on highly qualified \n",
+    "specialists, while at the same time ensuring safety, transparency and human control at all times.\n",
+    "\n",
+    "\"In the DLR LOKI project, we investigated how humans and AI systems can cooperate successfully in aviation. Our aim was to \n",
+    "make this collaboration trustworthy, transparent and user-centred,\" says the project’s scientific lead, Carmen Bruder from \n",
+    "the DLR Institute of Aerospace Medicine.\n",
+    "\n",
+    "The researchers developed a digital air traffic controller as an AI-supported team member for air traffic control and an \n",
+    "intelligent assistance system for pilots, and made recommendations for the safe introduction of trustworthy AI into aviation.\n",
+    "\n",
+    "One focus of the research was the development of the digital air traffic controller DIRC (Digital Interactive Reliable \n",
+    "Controller). Unlike traditional assistance systems, the DIRC not only supports human controllers with information or \n",
+    "recommendations but can also take on certain tasks independently. This allows for dynamic coordination with human air traffic \n",
+    "controllers regarding which tasks are to be undertaken.\n",
+    "\n",
+    "Simulations showed that human and digital air traffic controllers can divide their tasks efficiently. Together, they managed \n",
+    "up to 25 percent more than the current maximum traffic volume specified for an airspace sector. The research thus addresses \n",
+    "a key challenge facing European air traffic: rising traffic volumes coupled with the increasing challenge of maintaining \n",
+    "staffing levels in air traffic control.\n",
+    "\n",
+    "In addition to air traffic control, the researchers developed the Intelligent Pilot Assistance System (IPAS), a demonstrator \n",
+    "designed to support pilots in the cockpit.\n",
+    "\n",
+    "The system is designed to support crews particularly in time-critical situations, for example when selecting alternative \n",
+    "flight routes or suitable alternate airports. The AI analyses large amounts of relevant information and processes it to aid \n",
+    "decision-making. At the same time, the team investigated what information pilots need in order to understand and reliably \n",
+    "assess the recommendations of an AI system.\n",
+    "\n",
+    "In addition to technical developments, the project investigated the conditions necessary for trustworthy collaboration \n",
+    "between humans and AI systems. The researchers analysed the expectations and requirements of aviation personnel and assessed \n",
+    "how collaboration between humans and AI in aviation should be structured.\n",
+    "\n",
+    "A key finding is that trust is not built solely on a system’s performance. Crucial factors include transparent information \n",
+    "about the AI's capabilities and limitations, suitable training opportunities and the gradual introduction of new functions. \n",
+    "Based on the project findings, important recommendations were formulated; taking these into account will help ensure the \n",
+    "successful development and introduction of AI systems in aviation.\n",
+    "\n",
+    "The findings form a basis for the further development of AI systems in aviation and will feed into the European debate on the \n",
+    "safe introduction of such systems.\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4432982c",
+   "metadata": {},
+   "source": [
+    "... we can prepare it for tagging."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "10997545",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "taxed_document = taxo.TaxedDocument(media_tax, text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6024862e",
+   "metadata": {},
+   "source": [
+    "The following set of hyperparameters typically works well with this taxonomy. Other taxonomies might require tuning of these parameters."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "3dc3a401",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tagging_parameters = (\n",
+    "    10,  # Max number of concepts initially assigned to each sentence\n",
+    "    0.25,  # Minimum similarity value on sentence level\n",
+    "    2,  # Weight for title sentence\n",
+    "    1.5,  # Saturation parameter for score aggregation\n",
+    "    0.25,  # Minimum value for aggregated score \n",
+    "    80,  # Percentage for hierarchy truncation on top level\n",
+    "    80,  # Percentage for hierarchy truncation on second level\n",
+    "    7,  # Maximum number of returned leaf concepts \n",
+    "    2,  # Number of paths to the root considered per concept\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4c00e4f1",
+   "metadata": {},
+   "source": [
+    "Now we are ready to identify the taxonomy tags for the given text. The result shows the highest scored concepts including their hierarchical anchestry."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "24e9f432",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[[('economy, business and finance', 'M04000000', 0.572107398702342),\n",
+       "  ('economic sector', 'M20000209', 0.572107398702342),\n",
+       "  ('transport', 'M20000337', 0.572107398702342),\n",
+       "  ('air transport', 'M20000338', 0.572107398702342)],\n",
+       " [('science and technology', 'M13000000', 0.4709945980217274),\n",
+       "  ('technology and engineering', 'M20000756', 0.4709945980217274),\n",
+       "  ('information technology and computer science',\n",
+       "   'M20000763',\n",
+       "   0.4709945980217274),\n",
+       "  ('artificial intelligence', 'M20001298', 0.4709945980217274)],\n",
+       " [('disaster, accident and emergency incident',\n",
+       "   'M03000000',\n",
+       "   0.4322114299130465),\n",
+       "  ('accident and emergency incident', 'M20000139', 0.4322114299130465),\n",
+       "  ('transportation accident and incident', 'M20000143', 0.4322114299130465),\n",
+       "  ('air and space accident and incident', 'M20000144', 0.4322114299130465)],\n",
+       " [('science and technology', 'M13000000', 0.4314064954667891),\n",
+       "  ('technology and engineering', 'M20000756', 0.4314064954667891),\n",
+       "  ('aerospace engineering', 'M20000757', 0.4314064954667891)],\n",
+       " [('science and technology', 'M13000000', 0.42194015078626435),\n",
+       "  ('scientific research', 'M20000735', 0.42194015078626435),\n",
+       "  ('scientific innovation', 'M20000736', 0.42194015078626435)],\n",
+       " [('disaster, accident and emergency incident',\n",
+       "   'M03000000',\n",
+       "   0.3640833247315276),\n",
+       "  ('emergency incident', 'M20000160', 0.3640833247315276),\n",
+       "  ('transport incident', 'M20000162', 0.3640833247315276),\n",
+       "  ('air and space incident', 'M20000163', 0.3640833247315276)]]"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "taxed_document.identify_concepts(*tagging_parameters, with_scores=True)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}