equity-aggregator 0.1.1__py3-none-any.whl → 0.1.5__py3-none-any.whl

equity_aggregator/README.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-The equity aggregator is a sophisticated financial data processing system that aggregates equity information from multiple authoritative sources (Euronext, LSE, SEC, XETRA) and enriches it with supplementary data from Yahoo Finance.
+The equity aggregator is a sophisticated financial data processing system that aggregates equity information from multiple discovery sources (LSEG, SEC, Stock Analysis, TradingView, XETRA and Intrinio) and enriches it with supplementary data from Yahoo Finance and the Global LEI Foundation.
 
 ## Architecture & Design
 
@@ -15,6 +15,7 @@ src/equity_aggregator/
 ├── cli/                    # Presentation Layer - User Interface
 ├── domain/                 # Business Logic Layer - Core Domain
 │   ├── pipeline/           # Aggregation pipeline orchestration
+│   ├── retrieval/          # Canonical equity download and retrieval
 │   └── _utils/             # Domain-specific utilities
 ├── adapters/               # Infrastructure Layer - External Integrations
 │   └── data_sources/       # Data source adapters
@@ -30,16 +31,16 @@ src/equity_aggregator/
 The system processes equity data through a six-stage async streaming pipeline:
 
 ```
-Raw Data Sources → Parse → Convert → Identify → Deduplicate → Enrich → Canonicalise → Storage
+Raw Data Sources → Parse → Convert → Identify → Group → Enrich → Canonicalise → Storage
 ```
 
 ### Pipeline Stages
 
 #### 1. **Resolve**
 
-Orchestrates parallel data fetching from authoritative feeds:
+Orchestrates parallel data fetching from discovery feeds:
 
-- Fetches data from Euronext, LSE, SEC, and XETRA concurrently
+- Fetches data from LSEG, SEC, Stock Analysis, TradingView, XETRA and Intrinio concurrently
 - Combines all feed data into a single stream for processing
 - Maintains feed source metadata for downstream processing
 
@@ -47,7 +48,7 @@ Orchestrates parallel data fetching from authoritative feeds:
 
 Validates and structures raw feed data:
 
-- Applies feed-specific schemas (`EuronextFeedData`, `LseFeedData`, etc.)
+- Applies feed-specific schemas (`LsegFeedData`, `SecFeedData`, etc.)
 - Filters out invalid records early in the pipeline
 - Normalises data formats across different sources
 
@@ -64,25 +65,25 @@ Standardises financial data to USD reference currency:
 Enriches records with global identification metadata:
 
 - Queries OpenFIGI API for FIGI identifiers
-- Adds CUSIP, ISIN, and other standard identifiers
 - Creates globally unique equity identities
 
-#### 5. **Deduplicate**
+#### 5. **Group**
 
-Merges duplicate equity records by FIGI identifier:
+Groups equities by Share Class FIGI:
 
-- Groups records with identical share_class_figi values
-- Uses fuzzy matching to merge similar company names within groups
-- Preserves the most complete data when merging
+- Groups records with identical Share Class FIGI values
+- Preserves all discovery feed source data for later merging
+- Each group represents the same equity from multiple discovery sources
+- Yields groups as `list[RawEquity]` for enrichment processing
 
 #### 6. **Enrich**
 
-Adds supplementary data from enrichment feeds:
+Fetches enrichment data and performs comprehensive single merge:
 
-- Fetches additional data from Yahoo Finance
-- Only fills missing fields from authoritative sources
-- Preserves data integrity and source hierarchy
-- Applies controlled concurrency to respect API limits
+- Fetches representative identifiers from discovery data sources
+- Queries enrichment feeds (Yahoo Finance, Global LEI Foundation) using these identifiers
+- Performs single merge of all sources (discovery + enrichment) for optimal data quality
+- Applies controlled concurrency to enrichment feeds to respect API limits
 
 #### 7. **Canonicalise**
 
@@ -100,7 +101,7 @@ The pipeline uses asynchronous operations to process thousands of equity records
 
 **Parallel Data Fetching**
 
-- All authoritative feeds (Euronext, LSE, SEC, XETRA) are fetched simultaneously
+- All discovery feeds (LSEG, SEC, Stock Analysis, TradingView, XETRA, Intrinio) are fetched simultaneously
 
 **Streaming Pipeline**
 
@@ -108,7 +109,9 @@ The pipeline uses asynchronous operations to process thousands of equity records
 
 **Controlled Concurrency**
 
-- External API calls (OpenFIGI, Yahoo Finance) use semaphores to limit concurrent requests and respect rate limits
+- External API calls (OpenFIGI, Yahoo Finance, GLEIF) use semaphores to limit concurrent requests and respect rate limits
+- Each enrichment feed has a configurable concurrency limit
+- Fetch operations include timeout protection to prevent indefinite blocking
 
 **Non-blocking Operations**
 
@@ -119,12 +122,12 @@ Illustration of Pipeline Flow:
 
 ```python
 async def aggregate_canonical_equities() -> list[CanonicalEquity]:
-    
+
     # Resolve creates an async stream from multiple sources
     stream = resolve()
 
     # Each transform receives and returns an async iterator
-    transforms = (parse, convert, identify, deduplicate, enrich, canonicalise)
+    transforms = (parse, convert, identify, group, enrich, canonicalise)
 
     # Pipe stream through each transform sequentially
     for stage in transforms:
@@ -143,17 +146,23 @@ schemas/
 ├── raw.py                    # RawEquity - intermediate pipeline format
 ├── canonical.py              # CanonicalEquity - final standardised format
 ├── types.py                  # Type definitions and validators
+├── validators.py             # Reusable validators for identifiers and financial data
 └── feeds/                    # Feed-specific data models
-    ├── euronext_feed_data.py
-    ├── lse_feed_data.py
+    ├── lseg_feed_data.py
     ├── sec_feed_data.py
+    ├── stock_analysis_feed_data.py
+    ├── tradingview_feed_data.py
+    ├── gleif_feed_data.py
+    ├── feed_validators.py
     ├── xetra_feed_data.py
-    └── yfinance_feed_data.py
+    ├── yfinance_feed_data.py
+    └── intrinio_feed_data.py
 ```
 
 ### Critical Role of Schemas
 
 #### 1. **Data Validation at Boundaries**
+
 Each feed has a dedicated Pydantic schema that:
 - Validates incoming data structure and types
 - Normalises field names and formats
@@ -171,11 +180,11 @@ def parse(stream: AsyncIterable[FeedRecord]) -> AsyncIterator[RawEquity]:
 
 #### 3. **Field Mapping & Normalisation**
 ```python
-class EuronextFeedData(BaseModel):
-    name: str = Field(..., description="Company name")
-    symbol: str = Field(..., description="Trading symbol")
+class LsegFeedData(BaseModel):
+    issuername: str = Field(..., description="Company name")
+    tidm: str = Field(..., description="Trading symbol")
     isin: str = Field(..., description="ISIN identifier")
-    mics: list[str] = Field(..., description="Market identifiers")
+    mics: list[str] | None = Field(..., description="Market identifiers")
 
     # Automatic field mapping from raw feed data
     @field_validator('symbol')
@@ -191,31 +200,32 @@ class EuronextFeedData(BaseModel):
 4. **Enriched RawEquity** → Final canonicalisation
 5. **CanonicalEquity** → Database persistence
 
-## Authoritative vs Enrichment Feeds
+## Discovery vs Enrichment Feeds
 
-### Authoritative Feeds (Primary Sources)
+### Discovery Feeds (Primary Sources)
 
-- **Euronext**: Pan-European Börse Stock Exchange
-- **LSE**: London Stock Exchange
-- **XETRA**: Deutsche Börse Stock Exchange
+- **LSEG**: London Stock Exchange Group trading platform
 - **SEC**: US Securities and Exchange Commission
+- **Stock Analysis**: US equities with comprehensive financial metrics
+- **TradingView**: US equities with comprehensive financial metrics
+- **XETRA**: Deutsche Börse Stock Exchange
+- **Intrinio**: US financial data API providing company, securities, and real-time quote data
 
 **Characteristics**:
 
-- Provide core equity data (names, symbols, identifiers)
-- Considered source of truth for fundamental information
-- Data from these feeds is **never** overwritten by enrichment
+- Provide core equity identifier data (names, symbols, codes)
+- Multiple discovery sources for the same equity are merged with enrichment data
 
 ### Enrichment Feeds (Supplementary Sources)
 
 - **Yahoo Finance**: Market data and financial metrics
+- **Global LEI Foundation**: ISIN->LEI mapping for Legal Entity Identifier enrichment
 
 **Characteristics**
 
-- Only supplements missing data; never overwrites authoritative values
 - Provides additional financial metrics (market cap, analyst ratings, etc.)
-- Respects data hierarchy and source priority
-- Applied after authoritative data processing is complete
+- Uses representative identifiers from discovery sources for look-up
+- Applied after grouping but before final merge
 
 ## Equity Aggregator Components
 
@@ -235,7 +245,7 @@ class EuronextFeedData(BaseModel):
 
 ### Adapters Layer
 
-- **data_sources/authoritative_feeds/**: Primary data source integrations
+- **data_sources/discovery_feeds/**: Primary data source integrations
 - **data_sources/enrichment_feeds/**: Supplementary data integrations
 - **data_sources/reference_lookup/**: External API services (OpenFIGI, exchange rates)
 

equity_aggregator/adapters/__init__.py

@@ -1,12 +1,15 @@
 # adapters/__init__.py
 
-from .data_sources.authoritative_feeds import (
-    fetch_equity_records_euronext,
-    fetch_equity_records_lse,
+from .data_sources.discovery_feeds import (
+    fetch_equity_records_intrinio,
+    fetch_equity_records_lseg,
     fetch_equity_records_sec,
+    fetch_equity_records_stock_analysis,
+    fetch_equity_records_tradingview,
     fetch_equity_records_xetra,
 )
 from .data_sources.enrichment_feeds import (
+    open_gleif_feed,
     open_yfinance_feed,
 )
 from .data_sources.reference_lookup import (
@@ -15,12 +18,15 @@ from .data_sources.reference_lookup import (
 )
 
 __all__ = [
-    # authoritative feeds
-    "fetch_equity_records_euronext",
-    "fetch_equity_records_lse",
-    "fetch_equity_records_xetra",
+    # discovery feeds
+    "fetch_equity_records_intrinio",
+    "fetch_equity_records_lseg",
     "fetch_equity_records_sec",
+    "fetch_equity_records_stock_analysis",
+    "fetch_equity_records_tradingview",
+    "fetch_equity_records_xetra",
     # enrichment feeds
+    "open_gleif_feed",
     "open_yfinance_feed",
     # reference lookup
     "fetch_equity_identification",

equity_aggregator/adapters/data_sources/__init__.py

@@ -1,8 +1,7 @@
 # data_sources/__init__.py
 
-from .authoritative_feeds import (
-    fetch_equity_records_euronext,
-    fetch_equity_records_lse,
+from .discovery_feeds import (
+    fetch_equity_records_lseg,
     fetch_equity_records_sec,
     fetch_equity_records_xetra,
 )
@@ -15,9 +14,8 @@ from .reference_lookup import (
 )
 
 __all__ = [
-    # authoritative feeds
-    "fetch_equity_records_euronext",
-    "fetch_equity_records_lse",
+    # discovery feeds
+    "fetch_equity_records_lseg",
     "fetch_equity_records_sec",
     "fetch_equity_records_xetra",
     # enrichment feeds

equity_aggregator/adapters/data_sources/_utils/_client.py

@@ -33,7 +33,7 @@ def make_client(**overrides: object) -> AsyncClient:
     # Set default timeouts for connections, reading, and writing
     timeout = Timeout(
         connect=3.0,  # 3s to establish TLS
-        read=None,  # no read timeout
+        read=300.0,  # up to 5 minutes to read a response
         write=5.0,  # up to 5s to send a body
         pool=None,  # no pool timeout
     )

equity_aggregator/adapters/data_sources/{authoritative_feeds → _utils}/_record_types.py

@@ -1,4 +1,4 @@
-# authoritative_feeds/_record_types.py
+# _utils/_record_types.py
 
 from collections.abc import AsyncIterator, Callable
 

equity_aggregator/adapters/data_sources/discovery_feeds/__init__.py

@@ -0,0 +1,17 @@
+# discovery_feeds/__init__.py
+
+from .intrinio import fetch_equity_records as fetch_equity_records_intrinio
+from .lseg import fetch_equity_records as fetch_equity_records_lseg
+from .sec import fetch_equity_records as fetch_equity_records_sec
+from .stock_analysis import fetch_equity_records as fetch_equity_records_stock_analysis
+from .tradingview import fetch_equity_records as fetch_equity_records_tradingview
+from .xetra import fetch_equity_records as fetch_equity_records_xetra
+
+__all__ = [
+    "fetch_equity_records_intrinio",
+    "fetch_equity_records_lseg",
+    "fetch_equity_records_sec",
+    "fetch_equity_records_stock_analysis",
+    "fetch_equity_records_tradingview",
+    "fetch_equity_records_xetra",
+]

equity_aggregator/adapters/data_sources/discovery_feeds/intrinio/__init__.py

@@ -0,0 +1,7 @@
+# intrinio/__init__.py
+
+from .intrinio import fetch_equity_records
+
+__all__ = [
+    "fetch_equity_records",
+]

equity_aggregator/adapters/data_sources/discovery_feeds/intrinio/_utils/__init__.py

@@ -0,0 +1,10 @@
+# _utils/__init__.py
+
+from .backoff import backoff_delays
+from .parser import parse_companies_response, parse_securities_response
+
+__all__ = [
+    "backoff_delays",
+    "parse_companies_response",
+    "parse_securities_response",
+]

equity_aggregator/adapters/data_sources/discovery_feeds/intrinio/_utils/backoff.py

@@ -0,0 +1,33 @@
+# intrinio/_utils/backoff.py
+
+
+import random
+from collections.abc import Iterator
+
+
+def backoff_delays(
+    *,
+    base: float = 5.0,
+    cap: float = 128.0,
+    jitter: float = 0.10,
+    attempts: int = 5,
+) -> Iterator[float]:
+    """
+    Yield an exponential backoff sequence with bounded jitter for retry delays.
+
+    Each delay is calculated as: delay * (1 ± jitter), doubling each time up to cap.
+
+    Args:
+        base (float): Initial delay in seconds.
+        cap (float): Maximum delay in seconds.
+        jitter (float): Fractional jitter (+/-) applied to each delay.
+        attempts (int): Number of delay values to yield.
+
+    Returns:
+        Iterator[float]: Sequence of delay values in seconds.
+    """
+    delay: float = base
+    for _ in range(attempts):
+        delta: float = delay * jitter * (2 * random.random() - 1)
+        yield max(0.0, min(delay + delta, cap))
+        delay = min(delay * 2, cap)

equity_aggregator/adapters/data_sources/discovery_feeds/intrinio/_utils/parser.py

@@ -0,0 +1,107 @@
+# _utils/parser.py
+
+from equity_aggregator.adapters.data_sources._utils._record_types import (
+    EquityRecord,
+)
+
+
+def parse_companies_response(payload: dict) -> tuple[list[EquityRecord], str | None]:
+    """
+    Parse Intrinio companies API response.
+
+    Args:
+        payload (dict): The JSON response from Intrinio API.
+
+    Returns:
+        tuple[list[EquityRecord], str | None]: Tuple of (parsed records,
+            next_page token).
+    """
+    companies = payload.get("companies", [])
+    next_page = payload.get("next_page")
+    valid_companies = [c for c in companies if _is_valid_company(c)]
+    records = [_extract_company_record(company) for company in valid_companies]
+    return records, next_page
+
+
+def parse_securities_response(payload: dict) -> list[EquityRecord]:
+    """
+    Parse Intrinio securities API response.
+
+    Uses the company data embedded in the response (which is authoritative
+    for these securities) rather than external company data, to avoid
+    ticker reassignment issues where stale company records have incorrect
+    identifiers.
+
+    Args:
+        payload (dict): The JSON response from Intrinio API.
+
+    Returns:
+        list[EquityRecord]: List of security records with company data merged.
+    """
+    securities = payload.get("securities", [])
+    company_data = payload.get("company", {})
+    company = _extract_company_record(company_data) if company_data else {}
+
+    return [
+        _extract_security_record(security, company)
+        for security in securities
+        if security.get("share_class_figi")
+    ]
+
+
+def _is_valid_company(company: dict | None) -> bool:
+    """
+    Check if a company record has required fields.
+
+    Args:
+        company (dict | None): Raw company data from API.
+
+    Returns:
+        bool: True if company has ticker and name, False otherwise.
+    """
+    return bool(company and company.get("ticker") and company.get("name"))
+
+
+def _extract_company_record(company: dict) -> EquityRecord:
+    """
+    Extract a company record from raw API data.
+
+    Args:
+        company (dict): Raw company data from Intrinio API.
+
+    Returns:
+        EquityRecord: Normalised company record.
+    """
+    return {
+        "company_id": company.get("id"),
+        "company_ticker": company.get("ticker"),
+        "name": company.get("name"),
+        "lei": company.get("lei"),
+        "cik": company.get("cik"),
+    }
+
+
+def _extract_security_record(security: dict, company: EquityRecord) -> EquityRecord:
+    """
+    Extract a security record merged with company data.
+
+    Args:
+        security (dict): Raw security data from Intrinio API.
+        company (EquityRecord): Company record to merge with security data.
+
+    Returns:
+        EquityRecord: Security record with company data merged.
+    """
+    return {
+        # Company data
+        "name": company.get("name"),
+        "lei": company.get("lei"),
+        "cik": company.get("cik"),
+        # Security data
+        "ticker": security.get("ticker"),
+        "share_class_figi": security.get("share_class_figi"),
+        "figi": security.get("figi"),
+        "composite_figi": security.get("composite_figi"),
+        "currency": security.get("currency"),
+        "exchange_mic": security.get("exchange_mic"),
+    }