equity-aggregator 0.1.4__tar.gz → 0.1.5__tar.gz

{equity_aggregator-0.1.4 → equity_aggregator-0.1.5}/.github/workflows/publish-pypi.yml

@@ -77,6 +77,9 @@ jobs:
           sed -i "3s/.*/version = \"$NEW_VERSION\"/" pyproject.toml
           echo "new_version=$NEW_VERSION" >> $GITHUB_OUTPUT
 
+      - name: Update lockfile
+        run: uv lock
+
       - name: Configure git
         run: |
           git config user.name "github-actions[bot]"
@@ -84,7 +87,7 @@ jobs:
 
       - name: Commit and tag version bump
         run: |
-          git add pyproject.toml
+          git add pyproject.toml uv.lock
           git commit -m "chore: bump version to ${{ steps.bump_version.outputs.new_version }} [skip ci]"
           git tag "v${{ steps.bump_version.outputs.new_version }}"
 

{equity_aggregator-0.1.4 → equity_aggregator-0.1.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: equity-aggregator
-Version: 0.1.4
+Version: 0.1.5
 Summary: Financial equity data aggregation toolkit
 Project-URL: Homepage, https://github.com/gregorykelleher/equity-aggregator
 Project-URL: Repository, https://github.com/gregorykelleher/equity-aggregator
@@ -45,7 +45,7 @@ Discovery feeds provide raw equity data from primary market sources:
 | 🇺🇸 SEC | United States | Securities and Exchange Commission - US-listed equities |
 | 🇺🇸 Stock Analysis | United States | Stock Analysis - Global equities |
 | 🇺🇸 TradingView | United States | TradingView - US-listed equities |
-| 🇩🇪 XETRA | Germany | Deutsche Börse electronic trading platform - Global listed equities |
+| 🇩🇪 XETRA | International | Deutsche Börse electronic trading platform - Global listed equities |
 
 ### Enrichment Feeds
 

{equity_aggregator-0.1.4 → equity_aggregator-0.1.5}/README.md

@@ -23,7 +23,7 @@ Discovery feeds provide raw equity data from primary market sources:
 | 🇺🇸 SEC | United States | Securities and Exchange Commission - US-listed equities |
 | 🇺🇸 Stock Analysis | United States | Stock Analysis - Global equities |
 | 🇺🇸 TradingView | United States | TradingView - US-listed equities |
-| 🇩🇪 XETRA | Germany | Deutsche Börse electronic trading platform - Global listed equities |
+| 🇩🇪 XETRA | International | Deutsche Börse electronic trading platform - Global listed equities |
 
 ### Enrichment Feeds
 

{equity_aggregator-0.1.4 → equity_aggregator-0.1.5}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "equity-aggregator"
-version = "0.1.4"
+version = "0.1.5"
 description = "Financial equity data aggregation toolkit"
 readme = "README.md"
 license = "MIT"

{equity_aggregator-0.1.4 → equity_aggregator-0.1.5}/src/equity_aggregator/README.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-The equity aggregator is a sophisticated financial data processing system that aggregates equity information from multiple discovery sources (LSEG, SEC, Stock Analysis, TradingView, XETRA) and enriches it with supplementary data from Yahoo Finance and Intrinio.
+The equity aggregator is a sophisticated financial data processing system that aggregates equity information from multiple discovery sources (LSEG, SEC, Stock Analysis, TradingView, XETRA and Intrinio) and enriches it with supplementary data from Yahoo Finance and the Global LEI Foundation.
 
 ## Architecture & Design
 
@@ -15,6 +15,7 @@ src/equity_aggregator/
 ├── cli/                    # Presentation Layer - User Interface
 ├── domain/                 # Business Logic Layer - Core Domain
 │   ├── pipeline/           # Aggregation pipeline orchestration
+│   ├── retrieval/          # Canonical equity download and retrieval
 │   └── _utils/             # Domain-specific utilities
 ├── adapters/               # Infrastructure Layer - External Integrations
 │   └── data_sources/       # Data source adapters
@@ -39,7 +40,7 @@ Raw Data Sources → Parse → Convert → Identify → Group → Enrich → Can
 
 Orchestrates parallel data fetching from discovery feeds:
 
-- Fetches data from LSEG, SEC, Stock Analysis, TradingView, and XETRA concurrently
+- Fetches data from LSEG, SEC, Stock Analysis, TradingView, XETRA and Intrinio concurrently
 - Combines all feed data into a single stream for processing
 - Maintains feed source metadata for downstream processing
 
@@ -64,14 +65,13 @@ Standardises financial data to USD reference currency:
 Enriches records with global identification metadata:
 
 - Queries OpenFIGI API for FIGI identifiers
-- Adds CUSIP, ISIN, and other standard identifiers
 - Creates globally unique equity identities
 
 #### 5. **Group**
 
 Groups equities by Share Class FIGI:
 
-- Groups records with identical share_class_figi values
+- Groups records with identical Share Class FIGI values
 - Preserves all discovery feed source data for later merging
 - Each group represents the same equity from multiple discovery sources
 - Yields groups as `list[RawEquity]` for enrichment processing
@@ -80,11 +80,10 @@ Groups equities by Share Class FIGI:
 
 Fetches enrichment data and performs comprehensive single merge:
 
-- Extracts representative identifiers from discovery sources using merge algorithms
-- Queries enrichment feeds (Yahoo Finance, Intrinio) with these identifiers
+- Fetches representative identifiers from discovery data sources
+- Queries enrichment feeds (Yahoo Finance, Global LEI Foundation) using these identifiers
 - Performs single merge of all sources (discovery + enrichment) for optimal data quality
-- Uses same merge logic for identifiers and final merge (mode for IDs, fuzzy clustering for names, frequency for symbols)
-- Applies controlled concurrency to respect API limits
+- Applies controlled concurrency to enrichment feeds to respect API limits
 
 #### 7. **Canonicalise**
 
@@ -102,7 +101,7 @@ The pipeline uses asynchronous operations to process thousands of equity records
 
 **Parallel Data Fetching**
 
-- All discovery feeds (LSEG, SEC, Stock Analysis, TradingView, XETRA) are fetched simultaneously
+- All discovery feeds (LSEG, SEC, Stock Analysis, TradingView, XETRA, Intrinio) are fetched simultaneously
 
 **Streaming Pipeline**
 
@@ -110,7 +109,9 @@ The pipeline uses asynchronous operations to process thousands of equity records
 
 **Controlled Concurrency**
 
-- External API calls (OpenFIGI, Yahoo Finance, Intrinio) use semaphores to limit concurrent requests and respect rate limits
+- External API calls (OpenFIGI, Yahoo Finance, GLEIF) use semaphores to limit concurrent requests and respect rate limits
+- Each enrichment feed has a configurable concurrency limit
+- Fetch operations include timeout protection to prevent indefinite blocking
 
 **Non-blocking Operations**
 
@@ -145,11 +146,14 @@ schemas/
 ├── raw.py                    # RawEquity - intermediate pipeline format
 ├── canonical.py              # CanonicalEquity - final standardised format
 ├── types.py                  # Type definitions and validators
+├── validators.py             # Reusable validators for identifiers and financial data
 └── feeds/                    # Feed-specific data models
     ├── lseg_feed_data.py
     ├── sec_feed_data.py
     ├── stock_analysis_feed_data.py
     ├── tradingview_feed_data.py
+    ├── gleif_feed_data.py
+    ├── feed_validators.py
     ├── xetra_feed_data.py
     ├── yfinance_feed_data.py
     └── intrinio_feed_data.py
@@ -158,6 +162,7 @@ schemas/
 ### Critical Role of Schemas
 
 #### 1. **Data Validation at Boundaries**
+
 Each feed has a dedicated Pydantic schema that:
 - Validates incoming data structure and types
 - Normalises field names and formats
@@ -204,6 +209,7 @@ class LsegFeedData(BaseModel):
 - **Stock Analysis**: US equities with comprehensive financial metrics
 - **TradingView**: US equities with comprehensive financial metrics
 - **XETRA**: Deutsche Börse Stock Exchange
+- **Intrinio**: US financial data API providing company, securities, and real-time quote data
 
 **Characteristics**:
 
@@ -213,12 +219,12 @@ class LsegFeedData(BaseModel):
 ### Enrichment Feeds (Supplementary Sources)
 
 - **Yahoo Finance**: Market data and financial metrics
-- **Intrinio**: Market data and financial metrics
+- **Global LEI Foundation**: ISIN->LEI mapping for Legal Entity Identifier enrichment
 
 **Characteristics**
 
 - Provides additional financial metrics (market cap, analyst ratings, etc.)
-- Uses representative identifiers from discovery sources for lookup
+- Uses representative identifiers from discovery sources for look-up
 - Applied after grouping but before final merge
 
 ## Equity Aggregator Components

{equity_aggregator-0.1.4 → equity_aggregator-0.1.5}/tests/unit/domain/pipeline/transforms/test_enrich.py

@@ -1,8 +1,8 @@
 # transforms/test_enrich.py
 
 import asyncio
-import os
-from collections.abc import AsyncIterable
+from collections.abc import AsyncIterable, AsyncIterator
+from contextlib import AsyncExitStack, asynccontextmanager
 from decimal import Decimal
 
 import pytest
@@ -10,8 +10,13 @@ import pytest
 from equity_aggregator.domain._utils import EquityIdentifiers, extract_identifiers
 from equity_aggregator.domain.pipeline.transforms.enrich import (
     EnrichmentFeed,
+    FeedSpec,
     _enrich_equity_group,
     _enrich_from_feed,
+    _init_feed,
+    _open_feeds,
+    _process_stream,
+    _rate_limited,
     _safe_fetch,
     _to_usd,
     _validate,
@@ -49,6 +54,30 @@ class ErrorFeedData:
         raise _ValidationError("validation failed")
 
 
+class StubFeed:
+    """
+    Test feed that returns a fixed response.
+    """
+
+    def __init__(self, response: dict[str, object]) -> None:
+        self.response = response
+
+    async def fetch_equity(self, **kwargs: object) -> dict[str, object]:
+        return self.response
+
+
+def stub_feed_factory(response: dict[str, object]) -> object:
+    """
+    Create an async context manager factory for StubFeed.
+    """
+
+    @asynccontextmanager
+    async def factory() -> AsyncIterator[StubFeed]:
+        yield StubFeed(response)
+
+    return factory
+
+
 def test_extract_identifiers_single_source() -> None:
     """
     ARRANGE: single RawEquity with identifiers
@@ -301,21 +330,30 @@ def test_safe_fetch_success_returns_dict() -> None:
     assert actual == {"foo": "bar"}
 
 
-def test_enrich_empty_stream_yields_nothing() -> None:
+def test_process_stream_empty_stream_yields_nothing() -> None:
     """
-    ARRANGE: async stream that never yields
-    ACT:     run enrich()
+    ARRANGE: async stream that never yields and test feed
+    ACT:     run _process_stream
     ASSERT:  yields empty list
     """
 
-    os.environ.setdefault("INTRINIO_API_KEY", "DUMMY_KEY")
+    async def good_fetcher(
+        symbol: str,
+        name: str,
+        isin: str | None,
+        cusip: str | None,
+        **kwargs: object,
+    ) -> dict[str, object]:
+        return {}
+
+    mock_feed = EnrichmentFeed(fetch=good_fetcher, model=GoodFeedData)
 
     async def empty_src() -> AsyncIterable[list[RawEquity]]:
         if False:
             yield
 
     async def runner() -> list[RawEquity]:
-        return [equity async for equity in enrich(empty_src())]
+        return [equity async for equity in _process_stream(empty_src(), (mock_feed,))]
 
     actual = asyncio.run(runner())
     assert actual == []
@@ -728,15 +766,12 @@ def test_to_usd_converts_currency() -> None:
     assert actual.currency == "USD"
 
 
-@pytest.mark.skip(reason="TODO - revisit this unit test")
-def test_enrich_passes_through_groups() -> None:
+def test_process_stream_passes_through_groups() -> None:
     """
-    ARRANGE: async stream of equity groups
-    ACT:     run enrich() over that stream
-    ASSERT:  yields merged equities
+    ARRANGE: async stream of equity groups and test feeds
+    ACT:     run _process_stream over that stream
+    ASSERT:  yields merged equities for each group
     """
-    os.environ.setdefault("INTRINIO_API_KEY", "DUMMY_KEY")
-
     first_equity = RawEquity(
         name="ONE",
         symbol="ONE",
@@ -757,12 +792,30 @@ def test_enrich_passes_through_groups() -> None:
         market_cap=Decimal("200"),
     )
 
+    async def good_fetcher(
+        symbol: str,
+        name: str,
+        isin: str | None,
+        cusip: str | None,
+        **kwargs: object,
+    ) -> dict[str, object]:
+        return {
+            "name": name,
+            "symbol": symbol,
+            "mics": ["XLON"],
+            "currency": "USD",
+            "last_price": Decimal("10"),
+            "market_cap": Decimal("1000"),
+        }
+
+    mock_feed = EnrichmentFeed(fetch=good_fetcher, model=GoodFeedData)
+
     async def source() -> AsyncIterable[list[RawEquity]]:
         yield [first_equity]
         yield [second_equity]
 
     async def runner() -> list[RawEquity]:
-        return [equity async for equity in enrich(source())]
+        return [equity async for equity in _process_stream(source(), (mock_feed,))]
 
     actual = asyncio.run(runner())
 
@@ -840,3 +893,146 @@ def test_enrich_from_feed_returns_validated_when_currency_is_none() -> None:
     actual = asyncio.run(_enrich_from_feed(identifiers, mock_feed))
 
     assert actual.lei == "G1MSYD2DQZX30V5DMX12"
+
+
+def test_rate_limited_passes_through_result() -> None:
+    """
+    ARRANGE: async function wrapped with _rate_limited
+    ACT:     call the wrapped function
+    ASSERT:  returns the original function's result
+    """
+
+    async def simple_fetcher(**kwargs: object) -> dict[str, object]:
+        return {"result": "success"}
+
+    semaphore = asyncio.Semaphore(10)
+    wrapped = _rate_limited(simple_fetcher, semaphore)
+
+    actual = asyncio.run(wrapped(symbol="TEST"))
+
+    assert actual == {"result": "success"}
+
+
+def test_rate_limited_serialises_calls_with_semaphore_of_one() -> None:
+    """
+    ARRANGE: semaphore with limit of 1, two concurrent calls
+    ACT:     start both calls simultaneously
+    ASSERT:  first call completes before second call starts
+    """
+    call_order: list[str] = []
+
+    async def tracking_fetcher(order: str, **kwargs: object) -> dict[str, object]:
+        call_order.append(f"{order}_start")
+        await asyncio.sleep(0.01)
+        call_order.append(f"{order}_end")
+        return {}
+
+    semaphore = asyncio.Semaphore(1)
+    wrapped = _rate_limited(tracking_fetcher, semaphore)
+
+    async def runner() -> None:
+        await asyncio.gather(
+            wrapped(order="first"),
+            wrapped(order="second"),
+        )
+
+    asyncio.run(runner())
+
+    assert call_order.index("first_end") < call_order.index("second_start")
+
+
+def test_rate_limited_raises_timeout_error() -> None:
+    """
+    ARRANGE: slow function with short timeout
+    ACT:     call the wrapped function
+    ASSERT:  raises TimeoutError
+    """
+
+    async def slow_fetcher(**kwargs: object) -> dict[str, object]:
+        await asyncio.sleep(0.1)
+        return {}
+
+    semaphore = asyncio.Semaphore(10)
+    wrapped = _rate_limited(slow_fetcher, semaphore, timeout=0.01)
+
+    with pytest.raises(TimeoutError):
+        asyncio.run(wrapped())
+
+
+def test_open_feeds_yields_single_feed_for_single_spec() -> None:
+    """
+    ARRANGE: single FeedSpec with stub factory
+    ACT:     open feeds using _open_feeds
+    ASSERT:  yields tuple containing one feed
+    """
+    specs = (FeedSpec(stub_feed_factory({}), GoodFeedData, 5),)
+
+    async def runner() -> int:
+        async with _open_feeds(specs) as feeds:
+            return len(feeds)
+
+    actual = asyncio.run(runner())
+
+    assert actual == 1
+
+
+def test_open_feeds_fetch_returns_stub_response() -> None:
+    """
+    ARRANGE: FeedSpec with stub factory returning specific data
+    ACT:     open feeds and call fetch
+    ASSERT:  returns the stub response
+    """
+    specs = (FeedSpec(stub_feed_factory({"key": "value"}), GoodFeedData, 5),)
+
+    async def runner() -> dict[str, object]:
+        async with _open_feeds(specs) as feeds:
+            return await feeds[0].fetch(symbol="X", name="X")
+
+    actual = asyncio.run(runner())
+
+    assert actual == {"key": "value"}
+
+
+def test_init_feed_returns_functional_enrichment_feed() -> None:
+    """
+    ARRANGE: FeedSpec with stub factory
+    ACT:     initialise feed using _init_feed
+    ASSERT:  fetch returns the stub response
+    """
+    spec = FeedSpec(stub_feed_factory({"result": "ok"}), GoodFeedData, 5)
+
+    async def runner() -> dict[str, object]:
+        async with AsyncExitStack() as stack:
+            feed = await _init_feed(spec, stack)
+            return await feed.fetch(symbol="X", name="X")
+
+    actual = asyncio.run(runner())
+
+    assert actual == {"result": "ok"}
+
+
+def test_enrich_yields_merged_equity() -> None:
+    """
+    ARRANGE: async stream with one equity group
+    ACT:     run enrich() over that stream
+    ASSERT:  yields the merged equity (enrichment failures are handled gracefully)
+    """
+    equity = RawEquity(
+        name="TEST",
+        symbol="TEST",
+        share_class_figi="BBG000TEST01",
+        mics=["XNAS"],
+        currency="USD",
+        last_price=Decimal("100"),
+        market_cap=Decimal("1000000"),
+    )
+
+    async def source() -> AsyncIterable[list[RawEquity]]:
+        yield [equity]
+
+    async def runner() -> list[RawEquity]:
+        return [e async for e in enrich(source())]
+
+    actual = asyncio.run(runner())
+
+    assert actual[0].symbol == "TEST"

{equity_aggregator-0.1.4 → equity_aggregator-0.1.5}/uv.lock

@@ -160,7 +160,7 @@ wheels = [
 
 [[package]]
 name = "equity-aggregator"
-version = "0.1.3"
+version = "0.1.5"
 source = { editable = "." }
 dependencies = [
     { name = "httpx", extra = ["http2"] },