corp-extractor 0.2.11__py3-none-any.whl → 0.4.0__py3-none-any.whl

{corp_extractor-0.2.11.dist-info → corp_extractor-0.4.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: corp-extractor
-Version: 0.2.11
+Version: 0.4.0
 Summary: Extract structured statements from text using T5-Gemma 2 and Diverse Beam Search
 Project-URL: Homepage, https://github.com/corp-o-rate/statement-extractor
 Project-URL: Documentation, https://github.com/corp-o-rate/statement-extractor#readme
@@ -24,19 +24,17 @@ Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Classifier: Topic :: Text Processing :: Linguistic
 Requires-Python: >=3.10
 Requires-Dist: click>=8.0.0
+Requires-Dist: gliner2
 Requires-Dist: numpy>=1.24.0
 Requires-Dist: pydantic>=2.0.0
+Requires-Dist: sentence-transformers>=2.2.0
 Requires-Dist: torch>=2.0.0
 Requires-Dist: transformers>=5.0.0rc3
-Provides-Extra: all
-Requires-Dist: sentence-transformers>=2.2.0; extra == 'all'
 Provides-Extra: dev
 Requires-Dist: mypy>=1.0.0; extra == 'dev'
 Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
 Requires-Dist: pytest>=7.0.0; extra == 'dev'
 Requires-Dist: ruff>=0.1.0; extra == 'dev'
-Provides-Extra: embeddings
-Requires-Dist: sentence-transformers>=2.2.0; extra == 'embeddings'
 Description-Content-Type: text/markdown
 
 # Corp Extractor
@@ -51,40 +49,40 @@ Extract structured subject-predicate-object statements from unstructured text us
 
 - **Structured Extraction**: Converts unstructured text into subject-predicate-object triples
 - **Entity Type Recognition**: Identifies 12 entity types (ORG, PERSON, GPE, LOC, PRODUCT, EVENT, etc.)
-- **Quality Scoring** *(v0.2.0)*: Each triple scored for groundedness (0-1) based on source text
-- **Beam Merging** *(v0.2.0)*: Combines top beams for better coverage instead of picking one
-- **Embedding-based Dedup** *(v0.2.0)*: Uses semantic similarity to detect near-duplicate predicates
-- **Predicate Taxonomies** *(v0.2.0)*: Normalize predicates to canonical forms via embeddings
-- **Contextualized Matching** *(v0.2.2)*: Compares full "Subject Predicate Object" against source text for better accuracy
-- **Entity Type Merging** *(v0.2.3)*: Automatically merges UNKNOWN entity types with specific types during deduplication
-- **Reversal Detection** *(v0.2.3)*: Detects and corrects subject-object reversals using embedding comparison
-- **Command Line Interface** *(v0.2.4)*: Full-featured CLI for terminal usage
+- **GLiNER2 Integration** *(v0.4.0)*: Uses GLiNER2 (205M params) for entity recognition and relation extraction
+- **Predefined Predicates** *(v0.4.0)*: Optional `--predicates` list for GLiNER2 relation extraction mode
+- **Entity-based Scoring** *(v0.4.0)*: Confidence combines semantic similarity (50%) + entity recognition scores (25% each)
+- **Multi-Candidate Extraction**: Generates 3 candidates per statement (hybrid, GLiNER2-only, predicate-split)
+- **Best Triple Selection**: Keeps only highest-scoring triple per source (use `--all-triples` to keep all)
+- **Extraction Method Tracking**: Each statement includes `extraction_method` field (hybrid, gliner, split, model)
+- **Beam Merging**: Combines top beams for better coverage instead of picking one
+- **Embedding-based Dedup**: Uses semantic similarity to detect near-duplicate predicates
+- **Predicate Taxonomies**: Normalize predicates to canonical forms via embeddings
+- **Contextualized Matching**: Compares full "Subject Predicate Object" against source text for better accuracy
+- **Entity Type Merging**: Automatically merges UNKNOWN entity types with specific types during deduplication
+- **Reversal Detection**: Detects and corrects subject-object reversals using embedding comparison
+- **Command Line Interface**: Full-featured CLI for terminal usage
 - **Multiple Output Formats**: Get results as Pydantic models, JSON, XML, or dictionaries
 
 ## Installation
 
 ```bash
-# Recommended: include embedding support for smart deduplication
-pip install "corp-extractor[embeddings]"
-
-# Minimal installation (no embedding features)
 pip install corp-extractor
 ```
 
-**Note**: This package requires `transformers>=5.0.0` (pre-release) for T5-Gemma2 model support. Install with `--pre` flag if needed:
-```bash
-pip install --pre "corp-extractor[embeddings]"
-```
+The GLiNER2 model (205M params) is downloaded automatically on first use.
+
+**Note**: This package requires `transformers>=5.0.0` for T5-Gemma2 model support.
 
 **For GPU support**, install PyTorch with CUDA first:
 ```bash
 pip install torch --index-url https://download.pytorch.org/whl/cu121
-pip install "corp-extractor[embeddings]"
+pip install corp-extractor
 ```
 
 **For Apple Silicon (M1/M2/M3)**, MPS acceleration is automatically detected:
 ```bash
-pip install "corp-extractor[embeddings]"  # MPS used automatically
+pip install corp-extractor  # MPS used automatically
 ```
 
 ## Quick Start
@@ -182,6 +180,9 @@ Options:
   --no-dedup                   Disable deduplication
   --no-embeddings              Disable embedding-based dedup (faster)
   --no-merge                   Disable beam merging
+  --no-gliner                  Disable GLiNER2 extraction (use raw model output)
+  --predicates TEXT            Comma-separated predicate types for GLiNER2 relation extraction
+  --all-triples                Keep all candidate triples (default: best per source)
   --dedup-threshold FLOAT      Deduplication threshold (default: 0.65)
   --min-confidence FLOAT       Min confidence filter (default: 0)
   --taxonomy PATH              Load predicate taxonomy from file
@@ -284,7 +285,111 @@ for stmt in fixed_statements:
 
 During deduplication, reversed duplicates (e.g., "A -> P -> B" and "B -> P -> A") are now detected and merged, with the correct orientation determined by source text similarity.
 
-## Disable Embeddings (Faster, No Extra Dependencies)
+## New in v0.4.0: GLiNER2 Integration
+
+v0.4.0 replaces spaCy with **GLiNER2** (205M params) for entity recognition and relation extraction. GLiNER2 is a unified model that handles NER, text classification, structured data extraction, and relation extraction with CPU-optimized inference.
+
+### Why GLiNER2?
+
+The T5-Gemma model excels at:
+- **Triple isolation** - identifying that a relationship exists
+- **Coreference resolution** - resolving pronouns to named entities
+
+GLiNER2 now handles:
+- **Entity recognition** - refining subject/object boundaries
+- **Relation extraction** - when predefined predicates are provided
+- **Entity scoring** - scoring how "entity-like" subjects/objects are
+
+### Two Extraction Modes
+
+**Mode 1: With Predicate List** (GLiNER2 relation extraction)
+```python
+from statement_extractor import extract_statements, ExtractionOptions
+
+options = ExtractionOptions(predicates=["works_for", "founded", "acquired", "headquartered_in"])
+result = extract_statements("John works for Apple Inc. in Cupertino.", options)
+```
+
+Or via CLI:
+```bash
+corp-extractor "John works for Apple Inc." --predicates "works_for,founded,acquired"
+```
+
+**Mode 2: Without Predicate List** (entity-refined extraction)
+```python
+result = extract_statements("Apple announced a new iPhone.")
+# Uses GLiNER2 for entity extraction to refine boundaries
+# Extracts predicate from source text using T5-Gemma's hint
+```
+
+### Three Candidate Extraction Methods
+
+For each statement, three candidates are generated and the best is selected:
+
+| Method | Description |
+|--------|-------------|
+| `hybrid` | Model subject/object + GLiNER2/extracted predicate |
+| `gliner` | All components refined by GLiNER2 |
+| `split` | Source text split around the predicate |
+
+```python
+for stmt in result:
+    print(f"{stmt.subject.text} --[{stmt.predicate}]--> {stmt.object.text}")
+    print(f"  Method: {stmt.extraction_method}")  # hybrid, gliner, split, or model
+    print(f"  Confidence: {stmt.confidence_score:.2f}")
+```
+
+### Combined Quality Scoring
+
+Confidence scores combine **semantic similarity** and **entity recognition**:
+
+| Component | Weight | Description |
+|-----------|--------|-------------|
+| Semantic similarity | 50% | Cosine similarity between source text and reassembled triple |
+| Subject entity score | 25% | How entity-like the subject is (via GLiNER2) |
+| Object entity score | 25% | How entity-like the object is (via GLiNER2) |
+
+**Entity scoring (via GLiNER2):**
+- Recognized entity with high confidence: 1.0
+- Recognized entity with moderate confidence: 0.8
+- Partially recognized: 0.6
+- Not recognized: 0.2
+
+### Extraction Method Tracking
+
+Each statement includes an `extraction_method` field:
+- `hybrid` - Model subject/object + GLiNER2 predicate
+- `gliner` - All components refined by GLiNER2
+- `split` - Subject/object from splitting source text around predicate
+- `model` - All components from T5-Gemma model (only when `--no-gliner`)
+
+### Best Triple Selection
+
+By default, only the **highest-scoring triple** is kept for each source sentence.
+
+To keep all candidate triples:
+```python
+options = ExtractionOptions(all_triples=True)
+result = extract_statements(text, options)
+```
+
+Or via CLI:
+```bash
+corp-extractor "Your text" --all-triples --verbose
+```
+
+**Disable GLiNER2 extraction** to use only model output:
+```python
+options = ExtractionOptions(use_gliner_extraction=False)
+result = extract_statements(text, options)
+```
+
+Or via CLI:
+```bash
+corp-extractor "Your text" --no-gliner
+```
+
+## Disable Embeddings
 
 ```python
 options = ExtractionOptions(
@@ -353,21 +458,23 @@ for text in texts:
 This library uses the T5-Gemma 2 statement extraction model with **Diverse Beam Search** ([Vijayakumar et al., 2016](https://arxiv.org/abs/1610.02424)):
 
 1. **Diverse Beam Search**: Generates 4+ candidate outputs using beam groups with diversity penalty
-2. **Quality Scoring** *(v0.2.0)*: Each triple scored for groundedness in source text
-3. **Beam Merging** *(v0.2.0)*: Top beams combined for better coverage
-4. **Embedding Dedup** *(v0.2.0)*: Semantic similarity removes near-duplicate predicates
-5. **Predicate Normalization** *(v0.2.0)*: Optional taxonomy matching via embeddings
-6. **Contextualized Matching** *(v0.2.2)*: Full statement context used for canonicalization and dedup
-7. **Entity Type Merging** *(v0.2.3)*: UNKNOWN types merged with specific types during dedup
-8. **Reversal Detection** *(v0.2.3)*: Subject-object reversals detected and corrected via embedding comparison
+2. **Quality Scoring**: Each triple scored for groundedness in source text
+3. **Beam Merging**: Top beams combined for better coverage
+4. **Embedding Dedup**: Semantic similarity removes near-duplicate predicates
+5. **Predicate Normalization**: Optional taxonomy matching via embeddings
+6. **Contextualized Matching**: Full statement context used for canonicalization and dedup
+7. **Entity Type Merging**: UNKNOWN types merged with specific types during dedup
+8. **Reversal Detection**: Subject-object reversals detected and corrected via embedding comparison
+9. **GLiNER2 Extraction** *(v0.4.0)*: Entity recognition and relation extraction for improved accuracy
 
 ## Requirements
 
 - Python 3.10+
 - PyTorch 2.0+
-- Transformers 4.35+
+- Transformers 5.0+
 - Pydantic 2.0+
-- sentence-transformers 2.2+ *(optional, for embedding features)*
+- sentence-transformers 2.2+
+- GLiNER2 (model downloaded automatically on first use)
 - ~2GB VRAM (GPU) or ~4GB RAM (CPU)
 
 ## Links

corp_extractor-0.4.0.dist-info/RECORD

@@ -0,0 +1,12 @@
+statement_extractor/__init__.py,sha256=KwZfWnTB9oevTLw0TrNlYFu67qIYO-34JqDtcpjOhZI,3013
+statement_extractor/canonicalization.py,sha256=ZMLs6RLWJa_rOJ8XZ7PoHFU13-zeJkOMDnvK-ZaFa5s,5991
+statement_extractor/cli.py,sha256=FOkkihVfoROc-Biu8ICCzlLJeDScYYNLLJHnv0GCGGM,9507
+statement_extractor/extractor.py,sha256=d0HnCeCPybw-4jDxH_ffZ4LY9Klvqnza_wa90Bd4Q18,40074
+statement_extractor/gliner_extraction.py,sha256=KNs3n5-fnoUwY1wvbPwZL8j-3YVstmioJlcjp2k1FmY,10491
+statement_extractor/models.py,sha256=cyCQc3vlYB3qlg6-uL5Vt4odIiulKtHzz1Cyrf0lEAU,12198
+statement_extractor/predicate_comparer.py,sha256=jcuaBi5BYqD3TKoyj3pR9dxtX5ihfDJvjdhEd2LHCwc,26184
+statement_extractor/scoring.py,sha256=s_8nhavBNzPPFmGf2FyBummH4tgP7YGpXoMhl2Jh3Xw,16650
+corp_extractor-0.4.0.dist-info/METADATA,sha256=8f2CDtZG757kaB6XMfbBVdNSRMyS5-4Lflc_LoZCC_8,17725
+corp_extractor-0.4.0.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
+corp_extractor-0.4.0.dist-info/entry_points.txt,sha256=i0iKFqPIusvb-QTQ1zNnFgAqatgVah-jIhahbs5TToQ,115
+corp_extractor-0.4.0.dist-info/RECORD,,

statement_extractor/__init__.py

@@ -29,12 +29,13 @@ Example:
     >>> data = extract_statements_as_dict("Some text...")
 """
 
-__version__ = "0.2.5"
+__version__ = "0.3.0"
 
 # Core models
 from .models import (
     Entity,
     EntityType,
+    ExtractionMethod,
     ExtractionOptions,
     ExtractionResult,
     Statement,
@@ -73,6 +74,7 @@ __all__ = [
     # Core models
     "Entity",
     "EntityType",
+    "ExtractionMethod",
     "ExtractionOptions",
     "ExtractionResult",
     "Statement",

statement_extractor/cli.py

@@ -34,6 +34,7 @@ def _configure_logging(verbose: bool) -> None:
         "statement_extractor.scoring",
         "statement_extractor.predicate_comparer",
         "statement_extractor.canonicalization",
+        "statement_extractor.gliner_extraction",
     ]:
         logging.getLogger(logger_name).setLevel(level)
 
@@ -65,6 +66,9 @@ from .models import (
 @click.option("--no-dedup", is_flag=True, help="Disable deduplication")
 @click.option("--no-embeddings", is_flag=True, help="Disable embedding-based deduplication (faster)")
 @click.option("--no-merge", is_flag=True, help="Disable beam merging (select single best beam)")
+@click.option("--no-gliner", is_flag=True, help="Disable GLiNER2 extraction (use raw model output)")
+@click.option("--predicates", type=str, help="Comma-separated list of predicate types for GLiNER2 relation extraction")
+@click.option("--all-triples", is_flag=True, help="Keep all candidate triples instead of selecting best per source")
 @click.option("--dedup-threshold", type=float, default=0.65, help="Similarity threshold for deduplication (default: 0.65)")
 # Quality options
 @click.option("--min-confidence", type=float, default=0.0, help="Minimum confidence threshold 0-1 (default: 0)")
@@ -89,6 +93,9 @@ def main(
     no_dedup: bool,
     no_embeddings: bool,
     no_merge: bool,
+    no_gliner: bool,
+    predicates: Optional[str],
+    all_triples: bool,
     dedup_threshold: float,
     min_confidence: float,
     taxonomy: Optional[str],
@@ -152,6 +159,13 @@ def main(
     # Configure scoring
     scoring_config = ScoringConfig(min_confidence=min_confidence)
 
+    # Parse predicates if provided
+    predicate_list = None
+    if predicates:
+        predicate_list = [p.strip() for p in predicates.split(",") if p.strip()]
+        if not quiet:
+            click.echo(f"Using predicate list: {predicate_list}", err=True)
+
     # Configure extraction options
     options = ExtractionOptions(
         num_beams=beams,
@@ -160,6 +174,9 @@ def main(
         deduplicate=not no_dedup,
         embedding_dedup=not no_embeddings,
         merge_beams=not no_merge,
+        use_gliner_extraction=not no_gliner,
+        predicates=predicate_list,
+        all_triples=all_triples,
         predicate_taxonomy=predicate_taxonomy,
         predicate_config=predicate_config,
         scoring_config=scoring_config,
@@ -225,6 +242,9 @@ def _print_table(result, verbose: bool):
         click.echo(f"   {stmt.object.text}{object_type}")
 
         if verbose:
+            # Always show extraction method
+            click.echo(f"   Method: {stmt.extraction_method.value}")
+
             if stmt.confidence_score is not None:
                 click.echo(f"   Confidence: {stmt.confidence_score:.2f}")