equity-aggregator 0.1.1__py3-none-any.whl → 0.1.4__py3-none-any.whl

equity_aggregator/README.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-The equity aggregator is a sophisticated financial data processing system that aggregates equity information from multiple authoritative sources (Euronext, LSE, SEC, XETRA) and enriches it with supplementary data from Yahoo Finance.
+The equity aggregator is a sophisticated financial data processing system that aggregates equity information from multiple discovery sources (LSEG, SEC, Stock Analysis, TradingView, XETRA) and enriches it with supplementary data from Yahoo Finance and Intrinio.
 
 ## Architecture & Design
 
@@ -30,16 +30,16 @@ src/equity_aggregator/
 The system processes equity data through a six-stage async streaming pipeline:
 
 ```
-Raw Data Sources → Parse → Convert → Identify → Deduplicate → Enrich → Canonicalise → Storage
+Raw Data Sources → Parse → Convert → Identify → Group → Enrich → Canonicalise → Storage
 ```
 
 ### Pipeline Stages
 
 #### 1. **Resolve**
 
-Orchestrates parallel data fetching from authoritative feeds:
+Orchestrates parallel data fetching from discovery feeds:
 
-- Fetches data from Euronext, LSE, SEC, and XETRA concurrently
+- Fetches data from LSEG, SEC, Stock Analysis, TradingView, and XETRA concurrently
 - Combines all feed data into a single stream for processing
 - Maintains feed source metadata for downstream processing
 
@@ -47,7 +47,7 @@ Orchestrates parallel data fetching from authoritative feeds:
 
 Validates and structures raw feed data:
 
-- Applies feed-specific schemas (`EuronextFeedData`, `LseFeedData`, etc.)
+- Applies feed-specific schemas (`LsegFeedData`, `SecFeedData`, etc.)
 - Filters out invalid records early in the pipeline
 - Normalises data formats across different sources
 
@@ -67,21 +67,23 @@ Enriches records with global identification metadata:
 - Adds CUSIP, ISIN, and other standard identifiers
 - Creates globally unique equity identities
 
-#### 5. **Deduplicate**
+#### 5. **Group**
 
-Merges duplicate equity records by FIGI identifier:
+Groups equities by Share Class FIGI:
 
 - Groups records with identical share_class_figi values
-- Uses fuzzy matching to merge similar company names within groups
-- Preserves the most complete data when merging
+- Preserves all discovery feed source data for later merging
+- Each group represents the same equity from multiple discovery sources
+- Yields groups as `list[RawEquity]` for enrichment processing
 
 #### 6. **Enrich**
 
-Adds supplementary data from enrichment feeds:
+Fetches enrichment data and performs comprehensive single merge:
 
-- Fetches additional data from Yahoo Finance
-- Only fills missing fields from authoritative sources
-- Preserves data integrity and source hierarchy
+- Extracts representative identifiers from discovery sources using merge algorithms
+- Queries enrichment feeds (Yahoo Finance, Intrinio) with these identifiers
+- Performs single merge of all sources (discovery + enrichment) for optimal data quality
+- Uses same merge logic for identifiers and final merge (mode for IDs, fuzzy clustering for names, frequency for symbols)
 - Applies controlled concurrency to respect API limits
 
 #### 7. **Canonicalise**
@@ -100,7 +102,7 @@ The pipeline uses asynchronous operations to process thousands of equity records
 
 **Parallel Data Fetching**
 
-- All authoritative feeds (Euronext, LSE, SEC, XETRA) are fetched simultaneously
+- All discovery feeds (LSEG, SEC, Stock Analysis, TradingView, XETRA) are fetched simultaneously
 
 **Streaming Pipeline**
 
@@ -108,7 +110,7 @@ The pipeline uses asynchronous operations to process thousands of equity records
 
 **Controlled Concurrency**
 
-- External API calls (OpenFIGI, Yahoo Finance) use semaphores to limit concurrent requests and respect rate limits
+- External API calls (OpenFIGI, Yahoo Finance, Intrinio) use semaphores to limit concurrent requests and respect rate limits
 
 **Non-blocking Operations**
 
@@ -119,12 +121,12 @@ Illustration of Pipeline Flow:
 
 ```python
 async def aggregate_canonical_equities() -> list[CanonicalEquity]:
-    
+
     # Resolve creates an async stream from multiple sources
     stream = resolve()
 
     # Each transform receives and returns an async iterator
-    transforms = (parse, convert, identify, deduplicate, enrich, canonicalise)
+    transforms = (parse, convert, identify, group, enrich, canonicalise)
 
     # Pipe stream through each transform sequentially
     for stage in transforms:
@@ -144,11 +146,13 @@ schemas/
 ├── canonical.py              # CanonicalEquity - final standardised format
 ├── types.py                  # Type definitions and validators
 └── feeds/                    # Feed-specific data models
-    ├── euronext_feed_data.py
-    ├── lse_feed_data.py
+    ├── lseg_feed_data.py
     ├── sec_feed_data.py
+    ├── stock_analysis_feed_data.py
+    ├── tradingview_feed_data.py
     ├── xetra_feed_data.py
-    └── yfinance_feed_data.py
+    ├── yfinance_feed_data.py
+    └── intrinio_feed_data.py
 ```
 
 ### Critical Role of Schemas
@@ -171,11 +175,11 @@ def parse(stream: AsyncIterable[FeedRecord]) -> AsyncIterator[RawEquity]:
 
 #### 3. **Field Mapping & Normalisation**
 ```python
-class EuronextFeedData(BaseModel):
-    name: str = Field(..., description="Company name")
-    symbol: str = Field(..., description="Trading symbol")
+class LsegFeedData(BaseModel):
+    issuername: str = Field(..., description="Company name")
+    tidm: str = Field(..., description="Trading symbol")
     isin: str = Field(..., description="ISIN identifier")
-    mics: list[str] = Field(..., description="Market identifiers")
+    mics: list[str] | None = Field(..., description="Market identifiers")
 
     # Automatic field mapping from raw feed data
     @field_validator('symbol')
@@ -191,31 +195,31 @@ class EuronextFeedData(BaseModel):
 4. **Enriched RawEquity** → Final canonicalisation
 5. **CanonicalEquity** → Database persistence
 
-## Authoritative vs Enrichment Feeds
+## Discovery vs Enrichment Feeds
 
-### Authoritative Feeds (Primary Sources)
+### Discovery Feeds (Primary Sources)
 
-- **Euronext**: Pan-European Börse Stock Exchange
-- **LSE**: London Stock Exchange
-- **XETRA**: Deutsche Börse Stock Exchange
+- **LSEG**: London Stock Exchange Group trading platform
 - **SEC**: US Securities and Exchange Commission
+- **Stock Analysis**: US equities with comprehensive financial metrics
+- **TradingView**: US equities with comprehensive financial metrics
+- **XETRA**: Deutsche Börse Stock Exchange
 
 **Characteristics**:
 
-- Provide core equity data (names, symbols, identifiers)
-- Considered source of truth for fundamental information
-- Data from these feeds is **never** overwritten by enrichment
+- Provide core equity identifier data (names, symbols, codes)
+- Multiple discovery sources for the same equity are merged with enrichment data
 
 ### Enrichment Feeds (Supplementary Sources)
 
 - **Yahoo Finance**: Market data and financial metrics
+- **Intrinio**: Market data and financial metrics
 
 **Characteristics**
 
-- Only supplements missing data; never overwrites authoritative values
 - Provides additional financial metrics (market cap, analyst ratings, etc.)
-- Respects data hierarchy and source priority
-- Applied after authoritative data processing is complete
+- Uses representative identifiers from discovery sources for lookup
+- Applied after grouping but before final merge
 
 ## Equity Aggregator Components
 
@@ -235,7 +239,7 @@ class EuronextFeedData(BaseModel):
 
 ### Adapters Layer
 
-- **data_sources/authoritative_feeds/**: Primary data source integrations
+- **data_sources/discovery_feeds/**: Primary data source integrations
 - **data_sources/enrichment_feeds/**: Supplementary data integrations
 - **data_sources/reference_lookup/**: External API services (OpenFIGI, exchange rates)
 

equity_aggregator/adapters/__init__.py

@@ -1,12 +1,15 @@
 # adapters/__init__.py
 
-from .data_sources.authoritative_feeds import (
-    fetch_equity_records_euronext,
-    fetch_equity_records_lse,
+from .data_sources.discovery_feeds import (
+    fetch_equity_records_intrinio,
+    fetch_equity_records_lseg,
     fetch_equity_records_sec,
+    fetch_equity_records_stock_analysis,
+    fetch_equity_records_tradingview,
     fetch_equity_records_xetra,
 )
 from .data_sources.enrichment_feeds import (
+    open_gleif_feed,
     open_yfinance_feed,
 )
 from .data_sources.reference_lookup import (
@@ -15,12 +18,15 @@ from .data_sources.reference_lookup import (
 )
 
 __all__ = [
-    # authoritative feeds
-    "fetch_equity_records_euronext",
-    "fetch_equity_records_lse",
-    "fetch_equity_records_xetra",
+    # discovery feeds
+    "fetch_equity_records_intrinio",
+    "fetch_equity_records_lseg",
     "fetch_equity_records_sec",
+    "fetch_equity_records_stock_analysis",
+    "fetch_equity_records_tradingview",
+    "fetch_equity_records_xetra",
     # enrichment feeds
+    "open_gleif_feed",
     "open_yfinance_feed",
     # reference lookup
     "fetch_equity_identification",

equity_aggregator/adapters/data_sources/__init__.py

@@ -1,8 +1,7 @@
 # data_sources/__init__.py
 
-from .authoritative_feeds import (
-    fetch_equity_records_euronext,
-    fetch_equity_records_lse,
+from .discovery_feeds import (
+    fetch_equity_records_lseg,
     fetch_equity_records_sec,
     fetch_equity_records_xetra,
 )
@@ -15,9 +14,8 @@ from .reference_lookup import (
 )
 
 __all__ = [
-    # authoritative feeds
-    "fetch_equity_records_euronext",
-    "fetch_equity_records_lse",
+    # discovery feeds
+    "fetch_equity_records_lseg",
     "fetch_equity_records_sec",
     "fetch_equity_records_xetra",
     # enrichment feeds

equity_aggregator/adapters/data_sources/_utils/_client.py

@@ -33,7 +33,7 @@ def make_client(**overrides: object) -> AsyncClient:
     # Set default timeouts for connections, reading, and writing
     timeout = Timeout(
         connect=3.0,  # 3s to establish TLS
-        read=None,  # no read timeout
+        read=300.0,  # up to 5 minutes to read a response
         write=5.0,  # up to 5s to send a body
         pool=None,  # no pool timeout
     )

equity_aggregator/adapters/data_sources/{authoritative_feeds → _utils}/_record_types.py

@@ -1,4 +1,4 @@
-# authoritative_feeds/_record_types.py
+# _utils/_record_types.py
 
 from collections.abc import AsyncIterator, Callable
 

equity_aggregator/adapters/data_sources/discovery_feeds/__init__.py

@@ -0,0 +1,17 @@
+# discovery_feeds/__init__.py
+
+from .intrinio import fetch_equity_records as fetch_equity_records_intrinio
+from .lseg import fetch_equity_records as fetch_equity_records_lseg
+from .sec import fetch_equity_records as fetch_equity_records_sec
+from .stock_analysis import fetch_equity_records as fetch_equity_records_stock_analysis
+from .tradingview import fetch_equity_records as fetch_equity_records_tradingview
+from .xetra import fetch_equity_records as fetch_equity_records_xetra
+
+__all__ = [
+    "fetch_equity_records_intrinio",
+    "fetch_equity_records_lseg",
+    "fetch_equity_records_sec",
+    "fetch_equity_records_stock_analysis",
+    "fetch_equity_records_tradingview",
+    "fetch_equity_records_xetra",
+]

equity_aggregator/adapters/data_sources/discovery_feeds/intrinio/__init__.py

@@ -0,0 +1,7 @@
+# intrinio/__init__.py
+
+from .intrinio import fetch_equity_records
+
+__all__ = [
+    "fetch_equity_records",
+]

equity_aggregator/adapters/data_sources/discovery_feeds/intrinio/_utils/__init__.py

@@ -0,0 +1,10 @@
+# _utils/__init__.py
+
+from .backoff import backoff_delays
+from .parser import parse_companies_response, parse_securities_response
+
+__all__ = [
+    "backoff_delays",
+    "parse_companies_response",
+    "parse_securities_response",
+]

equity_aggregator/adapters/data_sources/discovery_feeds/intrinio/_utils/backoff.py

@@ -0,0 +1,33 @@
+# intrinio/_utils/backoff.py
+
+
+import random
+from collections.abc import Iterator
+
+
+def backoff_delays(
+    *,
+    base: float = 5.0,
+    cap: float = 128.0,
+    jitter: float = 0.10,
+    attempts: int = 5,
+) -> Iterator[float]:
+    """
+    Yield an exponential backoff sequence with bounded jitter for retry delays.
+
+    Each delay is calculated as: delay * (1 ± jitter), doubling each time up to cap.
+
+    Args:
+        base (float): Initial delay in seconds.
+        cap (float): Maximum delay in seconds.
+        jitter (float): Fractional jitter (+/-) applied to each delay.
+        attempts (int): Number of delay values to yield.
+
+    Returns:
+        Iterator[float]: Sequence of delay values in seconds.
+    """
+    delay: float = base
+    for _ in range(attempts):
+        delta: float = delay * jitter * (2 * random.random() - 1)
+        yield max(0.0, min(delay + delta, cap))
+        delay = min(delay * 2, cap)

equity_aggregator/adapters/data_sources/discovery_feeds/intrinio/_utils/parser.py

@@ -0,0 +1,107 @@
+# _utils/parser.py
+
+from equity_aggregator.adapters.data_sources._utils._record_types import (
+    EquityRecord,
+)
+
+
+def parse_companies_response(payload: dict) -> tuple[list[EquityRecord], str | None]:
+    """
+    Parse Intrinio companies API response.
+
+    Args:
+        payload (dict): The JSON response from Intrinio API.
+
+    Returns:
+        tuple[list[EquityRecord], str | None]: Tuple of (parsed records,
+            next_page token).
+    """
+    companies = payload.get("companies", [])
+    next_page = payload.get("next_page")
+    valid_companies = [c for c in companies if _is_valid_company(c)]
+    records = [_extract_company_record(company) for company in valid_companies]
+    return records, next_page
+
+
+def parse_securities_response(payload: dict) -> list[EquityRecord]:
+    """
+    Parse Intrinio securities API response.
+
+    Uses the company data embedded in the response (which is authoritative
+    for these securities) rather than external company data, to avoid
+    ticker reassignment issues where stale company records have incorrect
+    identifiers.
+
+    Args:
+        payload (dict): The JSON response from Intrinio API.
+
+    Returns:
+        list[EquityRecord]: List of security records with company data merged.
+    """
+    securities = payload.get("securities", [])
+    company_data = payload.get("company", {})
+    company = _extract_company_record(company_data) if company_data else {}
+
+    return [
+        _extract_security_record(security, company)
+        for security in securities
+        if security.get("share_class_figi")
+    ]
+
+
+def _is_valid_company(company: dict | None) -> bool:
+    """
+    Check if a company record has required fields.
+
+    Args:
+        company (dict | None): Raw company data from API.
+
+    Returns:
+        bool: True if company has ticker and name, False otherwise.
+    """
+    return bool(company and company.get("ticker") and company.get("name"))
+
+
+def _extract_company_record(company: dict) -> EquityRecord:
+    """
+    Extract a company record from raw API data.
+
+    Args:
+        company (dict): Raw company data from Intrinio API.
+
+    Returns:
+        EquityRecord: Normalised company record.
+    """
+    return {
+        "company_id": company.get("id"),
+        "company_ticker": company.get("ticker"),
+        "name": company.get("name"),
+        "lei": company.get("lei"),
+        "cik": company.get("cik"),
+    }
+
+
+def _extract_security_record(security: dict, company: EquityRecord) -> EquityRecord:
+    """
+    Extract a security record merged with company data.
+
+    Args:
+        security (dict): Raw security data from Intrinio API.
+        company (EquityRecord): Company record to merge with security data.
+
+    Returns:
+        EquityRecord: Security record with company data merged.
+    """
+    return {
+        # Company data
+        "name": company.get("name"),
+        "lei": company.get("lei"),
+        "cik": company.get("cik"),
+        # Security data
+        "ticker": security.get("ticker"),
+        "share_class_figi": security.get("share_class_figi"),
+        "figi": security.get("figi"),
+        "composite_figi": security.get("composite_figi"),
+        "currency": security.get("currency"),
+        "exchange_mic": security.get("exchange_mic"),
+    }