hyperbase-parser-ab 0.1.0__tar.gz

hyperbase_parser_ab-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,29 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install build dependencies
+        run: pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

hyperbase_parser_ab-0.1.0/.gitignore

@@ -0,0 +1,69 @@
+# Compiled source #
+###################
+*.com
+*.class
+*.dll
+*.exe
+*.o
+*.so
+build/
+
+# Packages #
+############
+# it's better to unpack these files and commit the raw source
+# git has its own built in compression methods
+*.7z
+*.dmg
+*.gz
+*.iso
+*.jar
+*.rar
+*.tar
+*.zip
+
+# Logs and databases #
+######################
+*.log
+*.sql
+*.sqlite
+
+# OS generated files #
+######################
+.DS_Store*
+ehthumbs.db
+Icon?
+Thumbs.db
+
+# IDE stuff #
+#############
+.idea
+.vscode
+
+
+# Python version/venv #
+#######################
+.python-version
+
+# Python bytecode #
+###################
+__pycache__
+*.pyc
+*.pyo
+
+# Python package stuff #
+########################
+*.egg-info
+
+# Jupyter notebooks #
+#####################
+.ipynb_checkpoints
+
+# db files #
+#####################
+*.db
+
+/venv
+/dist
+/site
+
+/models

hyperbase_parser_ab-0.1.0/CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+## [0.1.0] - 02-04-2026 - extracted from graphbrain
+
+### Added
+
+- Atomizier, a multilingual classifier for atom types.
+- Can now parse all languages supported by spaCy.
+
+### Changed
+
+- Original alpha-beta parser from Graphbrain was extracted to create this plugin.

hyperbase_parser_ab-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (C) 2026 CNRS - Centre national de la recherche scientifique
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

hyperbase_parser_ab-0.1.0/PKG-INFO

@@ -0,0 +1,62 @@
+Metadata-Version: 2.4
+Name: hyperbase-parser-ab
+Version: 0.1.0
+Summary: Semantic Hypergraph AlphaBeta Parser
+Project-URL: Homepage, https://hyperquest.ai/hyperbase
+Author-email: "Telmo Menezes et al." <telmo@telmomenezes.net>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: AI,Knowledge Representation,NLP,Natural Language Understanding,Parsing,Semantic Hypergraphs
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.10
+Requires-Dist: hyperbase>=0.8.0
+Requires-Dist: pip
+Requires-Dist: scikit-learn>=1.3.0
+Requires-Dist: spacy>=3.8.0
+Requires-Dist: torch>=2.0.0
+Requires-Dist: transformers>=4.46.0
+Provides-Extra: dev
+Requires-Dist: coverage>=7.4.3; extra == 'dev'
+Requires-Dist: datasets>=4.0.0; extra == 'dev'
+Requires-Dist: mypy>=1.8.0; extra == 'dev'
+Requires-Dist: pre-commit>=3.6.2; extra == 'dev'
+Requires-Dist: pytest>=9.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.2.2; extra == 'dev'
+Requires-Dist: types-passlib>=1.7.7.20240106; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Hyperbase Alpha-Beta Parser
+
+## A semantic hypergraph parser for natural language
+
+The Alpha-Beta parser is a [Hyperbase](https://hyperquest.ai/hyperbase) plugin that converts natural language text into *Semantic Hypergraphs (SH)*. It works in two stages:
+
+- **Alpha stage**: A multilingual neural token classifier (based on DistilBERT) assigns one of 39 semantic atom types to each token in a sentence -- for example, concepts, predicates, modifiers, builders, triggers and conjunctions.
+- **Beta stage**: A rule-based engine combines classified atoms into ordered, recursive hyperedges using syntactic and semantic composition rules, producing structured representations that can be manipulated with Hyperbase.
+
+## Supported languages
+
+The parser supports any language with a [spaCy](https://spacy.io) model available, including English, French, German, Italian, Portuguese and Spanish.
+
+While the parser is theoretically language-agnostic and could in principle support languages such as Mandarin, which differ substantially in morphological and syntactic structure, the authors' linguistic competence is limited to Germanic and Romance languages. We welcome the help of native speakers or domain experts in validating/improving support for other language families.
+
+## Installation and manual
+
+Installation instructions, the manual and more information can be found here: <https://hyperquest.ai/hyperbase>
+
+## Contributing
+
+Pull requests are welcome. For major changes, please open an issue first
+to discuss what you would like to change.
+
+Please make sure to update tests as appropriate.
+
+## License
+
+[MIT](https://choosealicense.com/licenses/mit/)

hyperbase_parser_ab-0.1.0/README.md

@@ -0,0 +1,29 @@
+# Hyperbase Alpha-Beta Parser
+
+## A semantic hypergraph parser for natural language
+
+The Alpha-Beta parser is a [Hyperbase](https://hyperquest.ai/hyperbase) plugin that converts natural language text into *Semantic Hypergraphs (SH)*. It works in two stages:
+
+- **Alpha stage**: A multilingual neural token classifier (based on DistilBERT) assigns one of 39 semantic atom types to each token in a sentence -- for example, concepts, predicates, modifiers, builders, triggers and conjunctions.
+- **Beta stage**: A rule-based engine combines classified atoms into ordered, recursive hyperedges using syntactic and semantic composition rules, producing structured representations that can be manipulated with Hyperbase.
+
+## Supported languages
+
+The parser supports any language with a [spaCy](https://spacy.io) model available, including English, French, German, Italian, Portuguese and Spanish.
+
+While the parser is theoretically language-agnostic and could in principle support languages such as Mandarin, which differ substantially in morphological and syntactic structure, the authors' linguistic competence is limited to Germanic and Romance languages. We welcome the help of native speakers or domain experts in validating/improving support for other language families.
+
+## Installation and manual
+
+Installation instructions, the manual and more information can be found here: <https://hyperquest.ai/hyperbase>
+
+## Contributing
+
+Pull requests are welcome. For major changes, please open an issue first
+to discuss what you would like to change.
+
+Please make sure to update tests as appropriate.
+
+## License
+
+[MIT](https://choosealicense.com/licenses/mit/)

hyperbase_parser_ab-0.1.0/VERSION

@@ -0,0 +1 @@
+0.1.0

hyperbase_parser_ab-0.1.0/pyproject.toml

@@ -0,0 +1,72 @@
+[project]
+name = "hyperbase-parser-ab"
+dynamic = ["version"]
+description = "Semantic Hypergraph AlphaBeta Parser"
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+authors = [
+    { name = "Telmo Menezes et al.", email = "telmo@telmomenezes.net" },
+]
+keywords = [
+    "NLP",
+    "AI",
+    "Knowledge Representation",
+    "Natural Language Understanding",
+    "Parsing",
+    "Semantic Hypergraphs",
+]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+    "Environment :: Console",
+    "Intended Audience :: Science/Research",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+]
+dependencies = [
+    "hyperbase>=0.8.0",
+    "scikit-learn>=1.3.0",
+    "spacy>=3.8.0",
+    "torch>=2.0.0",
+    "transformers>=4.46.0",
+    "pip",  # so that spaCy models can be easily installed with uv
+]
+
+[tool.uv.sources]
+hyperbase = { workspace = true }
+
+[project.optional-dependencies]
+dev = [
+    "mypy>=1.8.0",
+    "ruff>=0.2.2",
+    "pre-commit>=3.6.2",
+    "types-passlib>=1.7.7.20240106",
+    "coverage>=7.4.3",
+    "datasets>=4.0.0",
+    "pytest>=9.0.0",
+]
+
+[project.urls]
+Homepage = "https://hyperquest.ai/hyperbase"
+
+[project.entry-points."hyperbase.parsers"]
+alphabeta = "hyperbase_parser_ab:AlphaBetaParser"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.version]
+path = "VERSION"
+pattern = "(?P<version>.+)"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/hyperbase_parser_ab"]
+
+[tool.mypy]
+strict = true
+
+[tool.ruff]
+target-version = "py310"

hyperbase_parser_ab-0.1.0/scripts/generate_alpha_training_data.py

@@ -0,0 +1,107 @@
+import argparse
+import json
+
+from hyperbase import hedge
+from hyperbase_parser_ab import AlphaBetaParser
+
+
+if __name__ == '__main__':
+    arg_parser = argparse.ArgumentParser(
+        description='Generate alpha training data.')
+    arg_parser.add_argument('infile', type=str, help='input jsonl file')
+    arg_parser.add_argument('outfile', type=str, help='output tsv file')
+    arg_parser.add_argument(
+        '--lang', type=str, default='en', help='language (default: en)')
+    args = arg_parser.parse_args()
+
+    total_sentences = 0
+    ignored_sentences = 0
+    failed_parses = 0
+    total_atoms = 0
+
+    parser = AlphaBetaParser(lang=args.lang)
+
+    with open(args.infile, 'r') as infile, open(args.outfile, 'w') as outfile:
+        for line in infile.readlines():
+            case = json.loads(line)
+            sentence = case['sentence']
+            atoms = case['atoms']
+            parses = parser.parse_sentence(sentence)
+            spacy_sentence = list(parser.doc.sents)[0] if parser.doc else None
+            if not spacy_sentence or not parses:
+                failed_parses += 1
+            elif case['ignore']:
+                ignored_sentences += 1
+            elif len(atoms) == len(spacy_sentence):
+                total_sentences += 1
+                total_atoms += len(atoms)
+
+                for i in range(len(atoms)):
+                    atom = atoms[i]
+                    token = spacy_sentence[i]
+                    atom_edge = hedge(atom)
+                    if atom_edge is None:
+                        continue
+
+                    word_before = ''
+                    word_after = ''
+                    pos_before = ''
+                    pos_after = ''
+                    tag_before = ''
+                    tag_after = ''
+                    dep_before = ''
+                    dep_after = ''
+                    punct_before = False
+                    punct_after = False
+                    if i > 0:
+                        word_before = str(spacy_sentence[i - 1])
+                        pos_before = spacy_sentence[i - 1].pos_
+                        tag_before = spacy_sentence[i - 1].tag_
+                        dep_before = spacy_sentence[i - 1].dep_
+                        if spacy_sentence[i - 1].pos_ == 'PUNCT':
+                            punct_before = True
+                    if i < len(atoms) - 1:
+                        word_after = str(spacy_sentence[i + 1])
+                        pos_after = spacy_sentence[i + 1].pos_
+                        tag_after = spacy_sentence[i + 1].tag_
+                        dep_after = spacy_sentence[i + 1].dep_
+                        if spacy_sentence[i + 1].pos_ == 'PUNCT':
+                            punct_after = True
+
+                    head = token.head
+                    is_root = head is None
+                    has_lefts = token.n_lefts > 0
+                    has_rights = token.n_rights > 0
+                    outfile.write(('{}' + '\t{}' * 25 + '\n').format(
+                        atom_edge.mtype(),
+                        str(token),
+                        token.pos_,
+                        token.tag_,
+                        token.dep_,
+                        str(head) if head else '',
+                        head.pos_ if head else '',
+                        head.tag_ if head else '',
+                        head.dep_ if head else '',
+                        is_root,
+                        has_lefts,
+                        has_rights,
+                        token.ent_type_,
+                        token.shape_[:2],
+                        word_before,
+                        word_after,
+                        punct_before,
+                        punct_after,
+                        pos_before,
+                        pos_after,
+                        tag_before,
+                        tag_after,
+                        dep_before,
+                        dep_after,
+                        case['correct'],
+                        case['source']))
+            else:
+                failed_parses += 1
+            print('sentences: {}; ignored: {}; failed: {}; atoms: {}'.format(
+                total_sentences, ignored_sentences, failed_parses,
+                total_atoms))
+        print('done.')

hyperbase_parser_ab-0.1.0/scripts/train_atomizer.py

@@ -0,0 +1,159 @@
+import json
+
+import numpy as np
+from numpy.typing import NDArray
+from datasets import Dataset
+from transformers import (
+    AutoTokenizer,
+    AutoModelForTokenClassification,
+    TrainingArguments,
+    Trainer
+)
+
+
+def tokenize_and_align_labels(examples: dict[str, list]) -> dict[str, list]:
+    """Tokenize each sample and align the original token labels
+       to the new subword (tokenized) structure."""
+
+    tokenized_outputs = tokenizer(
+        examples["tokens"],
+        truncation=True,
+        is_split_into_words=True,     # Important for token-based tasks
+        return_offsets_mapping=True,  # We'll use this if needed
+        padding="max_length",         # or "longest" / "do_not_pad"
+        max_length=200                # adjust as needed
+    )
+
+    labels_aligned: list[list[int]] = []
+    for i, labels in enumerate(examples["labels"]):
+        # The tokenizer may split single words into multiple subwords.
+        # We create a label list the same length as input_ids,
+        # repeating the label for all subwords of the original token.
+        word_ids: list[int | None] = tokenized_outputs.word_ids(batch_index=i)
+        label_ids: list[int] = []
+        previous_word_idx: int | None = None
+
+        for word_idx in word_ids:
+            if word_idx is None:
+                # This is a special token like [CLS], [SEP], or padding
+                label_ids.append(-100)
+            else:
+                label_ids.append(label_to_id[labels[word_idx]])
+            previous_word_idx = word_idx
+
+        labels_aligned.append(label_ids)
+
+    # We don't need offset_mapping during model training, so we remove it
+    tokenized_outputs["offset_mapping"] = [None for _ in examples["tokens"]]
+
+    tokenized_outputs["labels"] = labels_aligned
+    return tokenized_outputs
+
+
+def compute_metrics(eval_pred: tuple[NDArray, NDArray]) -> dict[str, float]:
+    """Compute accuracy at the token level (simple example).
+       You can also compute F1, precision, recall, etc. by ignoring
+       the -100 special tokens."""
+    logits: NDArray
+    labels: NDArray
+    logits, labels = eval_pred
+    predictions: NDArray = np.argmax(logits, axis=-1)
+
+    # Flatten ignoring -100
+    true_predictions: list[int] = []
+    true_labels: list[int] = []
+    for pred, lab in zip(predictions, labels):
+        for p, l in zip(pred, lab):
+            if l != -100:  # skip special tokens
+                true_predictions.append(p)
+                true_labels.append(l)
+
+    results: dict[str, float] = accuracy_metric.compute(
+        references=true_labels,
+        predictions=true_predictions
+    )
+    return {"accuracy": results["accuracy"]}
+
+
+if __name__ == '__main__':
+    with open("sentences.jsonl", "rt") as f:
+        sentences: list[dict] = [json.loads(line) for line in f]
+
+    dataset_dict: dict[str, list] = {
+        "tokens": [sentence["words"] for sentence in sentences],
+        "labels": [sentence["types"] for sentence in sentences]
+    }
+
+    full_dataset: Dataset = Dataset.from_dict(dataset_dict)
+
+    max_words: int = max([len(sentence["words"]) for sentence in sentences])
+
+
+    labels: set[str] = set()
+    for sentence in sentences:
+        labels |= set(sentence["types"])
+    print(labels)
+    label_to_id: dict[str, int] = {label: i for i, label in enumerate(labels)}
+    id_to_label: dict[int, str] = {i: label for label, i in label_to_id.items()}
+
+    dataset = full_dataset.train_test_split(test_size=0.25, seed=42)
+    train_dataset = dataset["train"]
+    test_dataset = dataset["test"]
+
+    print("Num train samples:", len(train_dataset))
+    print("Num test samples: ", len(test_dataset))
+
+
+    model_checkpoint: str = "distilbert-base-multilingual-cased"
+    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True, add_prefix_space=True)
+
+    # Apply to train/test datasets
+    train_dataset = train_dataset.map(tokenize_and_align_labels, batched=True)
+    test_dataset = test_dataset.map(tokenize_and_align_labels, batched=True)
+
+    # Remove columns we don't feed directly to the model
+    # train_dataset = train_dataset.remove_columns(["tokens", "labels"])
+    # test_dataset = test_dataset.remove_columns(["tokens", "labels"])
+
+    # Set format for PyTorch
+    train_dataset.set_format("torch")
+    test_dataset.set_format("torch")
+
+    model = AutoModelForTokenClassification.from_pretrained(
+        model_checkpoint,
+        num_labels=len(labels),
+        id2label=id_to_label,
+        label2id=label_to_id
+    )
+
+    accuracy_metric = evaluate.load("accuracy")  # type: ignore[attr-defined]
+
+    training_args: TrainingArguments = TrainingArguments(
+        output_dir="./test-roberta-token-classifier",
+        eval_strategy="epoch",
+        save_strategy="epoch",
+        learning_rate=5e-5,
+        per_device_train_batch_size=2,
+        per_device_eval_batch_size=2,
+        num_train_epochs=3,
+        weight_decay=0.01,
+        logging_dir="./logs",
+        logging_steps=10,
+        report_to="none"  # Set to "tensorboard" if you want logs
+    )
+
+    trainer: Trainer = Trainer(
+        model=model,
+        args=training_args,
+        train_dataset=train_dataset,
+        eval_dataset=test_dataset,
+        processing_class=tokenizer,
+        compute_metrics=compute_metrics
+    )
+
+    trainer.train()
+
+    results: dict[str, float] = trainer.evaluate(test_dataset)  # type: ignore[arg-type]
+    print("Test set results:", results)
+
+    trainer.save_model("./token-classifier")

hyperbase_parser_ab-0.1.0/src/hyperbase_parser_ab/__init__.py

@@ -0,0 +1,3 @@
+from hyperbase_parser_ab.parser import AlphaBetaParser
+
+__all__ = ["AlphaBetaParser"]

hyperbase_parser_ab-0.1.0/src/hyperbase_parser_ab/alpha.py

@@ -0,0 +1,69 @@
+import numpy as np
+from numpy.typing import NDArray
+from scipy.sparse import spmatrix
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.preprocessing import OneHotEncoder
+from spacy.tokens import Span
+
+from hyperbase_parser_ab.atomizer import Atomizer
+
+
+class Alpha(object):
+    def __init__(self, cases_str: str | None = None, use_atomizer: bool = False) -> None:
+        if use_atomizer:
+            self.atomizer: Atomizer | None = Atomizer()
+        elif cases_str:
+            self.atomizer = None
+
+            X: list[tuple[str, str, str, str, str]] = []
+            y: list[list[str]] = []
+
+            for line in cases_str.strip().split('\n'):
+                sline: str = line.strip()
+                if len(sline) > 0:
+                    row: list[str] = sline.strip().split('\t')
+                    true_value: str = row[0]
+                    tag: str = row[3]
+                    dep: str = row[4]
+                    hpos: str = row[6]
+                    hdep: str = row[8]
+                    pos_after: str = row[19]
+
+                    y.append([true_value])
+                    X.append((tag, dep, hpos, hdep, pos_after))
+
+            if len(y) > 0:
+                self.empty: bool = False
+
+                self.encX: OneHotEncoder = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
+                self.encX.fit(np.array(X))
+                self.ency: OneHotEncoder = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
+                self.ency.fit(np.array(y))
+
+                X_: NDArray | spmatrix = self.encX.transform(np.array(X))
+                y_: NDArray | spmatrix = self.ency.transform(np.array(y))
+
+                self.clf: RandomForestClassifier = RandomForestClassifier(random_state=777)
+                self.clf.fit(X_, y_)
+        else:
+            self.empty = True
+
+    def predict(self, sentence: Span, features: list[tuple[str, str, str, str, str]]) -> tuple[str, ...] | list[str]:
+        if self.atomizer:
+            preds: list[tuple[str, str]] = self.atomizer.atomize(
+                sentence=str(sentence),
+                tokens=[str(token) for token in sentence])
+            atom_types: list[str] = [pred[1] for pred in preds]
+
+            # force known cases
+            for i in range(len(atom_types)):
+                if sentence[i].pos_ == 'VERB':
+                    atom_types[i] = 'P'
+            return atom_types
+        else:
+            # an empty classifier always predicts 'C'
+            if self.empty:
+                return tuple('C' for _ in range(len(features)))
+            _features: NDArray | spmatrix = self.encX.transform(np.array(features))
+            preds_arr: NDArray | spmatrix = self.ency.inverse_transform(self.clf.predict(_features))
+            return tuple(pred[0] if pred else 'C' for pred in preds_arr)