bigdata-smart-batching 1.2.1__tar.gz → 1.3.1__tar.gz

{bigdata_smart_batching-1.2.1 → bigdata_smart_batching-1.3.1}/CHANGELOG.md

@@ -5,6 +5,19 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+
+## [1.3.1] - 2026-05-18
+
+### Added
+
+- Document-level **source** filters: `INCLUDE` / `EXCLUDE` with a non-empty list of source ID strings, validated with `validate_source_filter`, modeled as `SourceFilter` / `SourceFilterInput` with `SourceMode` (type alias of `CategoryMode`). Serialized to `query.filters.source` on co-mention, volume, and search payloads. `plan_search(..., source=...)` and `source_filter` plumbed through `SmartBatchingPlanner` co-mention and volume flows. Payload builders `build_comention_payload`, `build_volume_payload`, and `build_search_query` accept optional `source_filter`. Types and validator re-exported from the package root.
+
+## [1.3.0] - 2026-05-12
+
+### Added
+
+- Document-level **source** filters: `INCLUDE` / `EXCLUDE` with a non-empty list of source ID strings, validated with `validate_source_filter`, modeled as `SourceFilter` / `SourceFilterInput` with `SourceMode` (type alias of `CategoryMode`). Serialized to `query.filters.source` on co-mention, volume, and search payloads. `plan_search(..., source=...)` and `source_filter` plumbed through `SmartBatchingPlanner` co-mention and volume flows. Payload builders `build_comention_payload`, `build_volume_payload`, and `build_search_query` accept optional `source_filter`. Types and validator re-exported from the package root.
+
 ## [1.2.1] - 2026-05-12
 
 ### Fixed

{bigdata_smart_batching-1.2.1 → bigdata_smart_batching-1.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bigdata-smart-batching
-Version: 1.2.1
+Version: 1.3.1
 Summary: High-performance semantic search with intelligent company grouping and parallel execution
 Project-URL: homepage, https://bigdata.com/api
 Author-email: "Bigdata.com" <support@ravenpack.com>
@@ -29,6 +29,7 @@ This module provides a two-step pipeline for efficient semantic search:
 - **Parallel execution** -- rate-limited concurrent requests with semaphore-controlled concurrency.
 - **Proportional sampling** -- retrieve a percentage of results while preserving the distribution across baskets.
 - **Category filtering** -- restrict searches to specific document categories (news, transcripts, filings, ...).
+- **Source filtering** -- include or exclude documents by source identifiers (`INCLUDE` / `EXCLUDE` plus a list of source IDs).
 - **Sentiment filtering** -- restrict searches to one or more inclusive document-level sentiment ranges.
 - **Throughput controls** -- shared sliding-window rate limiter and concurrency semaphore across all planning HTTP work, with shared exponential backoff on `429` / `403` / timeout retries.
 - **Deterministic plans** -- every request stays within service limits; plans can be saved and replayed.
@@ -164,6 +165,43 @@ Valid category values are exposed via `VALID_CATEGORY_VALUES` and include:
 `news_public`, `podcasts`, `research`, `research_academic_journals`,
 `research_investment_research`, `transcripts`.
 
+### Source Filtering
+
+Restrict searches by **source** (document source IDs) using the `source` argument.
+It accepts either a `SourceFilter` dataclass or a plain dict with `mode`
+(`INCLUDE` or `EXCLUDE`) and `values` (a non-empty list of source ID strings).
+The filter is serialized to `query.filters.source` on co-mention, volume, and
+search payloads during planning.
+
+```python
+from bigdata_smart_batching import SourceFilter, SourceMode, plan_search
+
+plan = plan_search(
+    universe="id_name_mapping_us_top_3000.csv",
+    start_date="2023-01-01",
+    end_date="2023-12-31",
+    text="merger acquisition",
+    source=SourceFilter(
+        mode=SourceMode.INCLUDE,
+        values=("source-id-1", "source-id-2"),
+    ),
+)
+
+# Equivalent dict form:
+plan = plan_search(
+    universe="id_name_mapping_us_top_3000.csv",
+    start_date="2023-01-01",
+    end_date="2023-12-31",
+    text="merger acquisition",
+    source={"mode": "EXCLUDE", "values": ["source-id-to-drop"]},
+)
+```
+
+`SourceMode` is the same `INCLUDE` / `EXCLUDE` enum as `CategoryMode`. String
+values are trimmed; empty strings are rejected. Use
+`validate_source_filter(source)` to validate and normalize input into the API
+dict shape.
+
 ### Sentiment Filtering
 
 Restrict searches to documents whose sentiment falls inside one or more
@@ -263,6 +301,7 @@ Step 2: EXECUTION
 | `min_period_days` | `int` | `30` | Minimum days per sub-period |
 | `min_entities_per_basket` | `int` | `1` | Lower bound on entities per basket |
 | `category` | `CategoryFilter \| dict \| None` | `None` | Optional category filter (`mode` + `values`) |
+| `source` | `SourceFilter \| dict \| None` | `None` | Optional source filter by source IDs (`mode` + `values`) |
 | `sentiment` | `SentimentRange \| dict \| list \| None` | `None` | Optional sentiment filter: one inclusive `{min, max}` range or a list of disjoint ranges |
 | `content_diversification` | `bool` | `True` | Enable/disable content diversification in ranking |
 | `requests_per_minute` | `int` | `100` | Global rate limit shared across all planning HTTP work (co-mention + volume) |
@@ -289,14 +328,18 @@ Step 2: EXECUTION
 - `load_universe_from_csv(csv_path)` -- load entity IDs from a CSV only.
 - `save_plan(plan, path)` / `load_plan(path)` -- persist plans as JSON.
 - `validate_category_filter(category)` -- validate and normalize a category filter to the API dict shape.
+- `validate_source_filter(source)` -- validate and normalize a source filter to the API dict shape (`filters.source`).
 - `validate_sentiment_range(sentiment)` -- validate and normalize sentiment range(s) into the list-of-`{min, max}` dicts emitted as `filters.sentiment.ranges`.
-- `build_comention_payload(...)`, `build_volume_payload(...)`, `build_search_query(...)` -- low-level payload builders used by the planner; useful if you need to construct custom requests.
+- `build_comention_payload(...)`, `build_volume_payload(...)`, `build_search_query(...)` -- low-level payload builders used by the planner; each accepts optional `category_filter`, `source_filter`, and `sentiment_filter` alongside entities and dates. Useful if you need to construct custom requests.
 
 ### Public Types
 
 - `CategoryFilter` -- frozen dataclass with `mode: CategoryMode` and `values: tuple[str, ...]`.
 - `CategoryMode` -- `StrEnum` with `INCLUDE` and `EXCLUDE` values.
 - `CategoryFilterInput` -- type alias: `CategoryFilter | dict[str, str | list[str]]`.
+- `SourceFilter` -- frozen dataclass with `mode: SourceMode` and `values: tuple[str, ...]` (source IDs).
+- `SourceMode` -- type alias of `CategoryMode` (`INCLUDE`, `EXCLUDE`).
+- `SourceFilterInput` -- type alias: `SourceFilter | dict[str, str | list[str]]`.
 - `SentimentRange` -- frozen dataclass with inclusive `min: float` and `max: float`.
 - `SentimentRangeInput` -- type alias for one `SentimentRange` / `{min, max}` dict, or a non-empty list of either.
 - `UniverseInput` -- type alias: `str | list[str]`.

{bigdata_smart_batching-1.2.1 → bigdata_smart_batching-1.3.1}/README.md

@@ -13,6 +13,7 @@ This module provides a two-step pipeline for efficient semantic search:
 - **Parallel execution** -- rate-limited concurrent requests with semaphore-controlled concurrency.
 - **Proportional sampling** -- retrieve a percentage of results while preserving the distribution across baskets.
 - **Category filtering** -- restrict searches to specific document categories (news, transcripts, filings, ...).
+- **Source filtering** -- include or exclude documents by source identifiers (`INCLUDE` / `EXCLUDE` plus a list of source IDs).
 - **Sentiment filtering** -- restrict searches to one or more inclusive document-level sentiment ranges.
 - **Throughput controls** -- shared sliding-window rate limiter and concurrency semaphore across all planning HTTP work, with shared exponential backoff on `429` / `403` / timeout retries.
 - **Deterministic plans** -- every request stays within service limits; plans can be saved and replayed.
@@ -148,6 +149,43 @@ Valid category values are exposed via `VALID_CATEGORY_VALUES` and include:
 `news_public`, `podcasts`, `research`, `research_academic_journals`,
 `research_investment_research`, `transcripts`.
 
+### Source Filtering
+
+Restrict searches by **source** (document source IDs) using the `source` argument.
+It accepts either a `SourceFilter` dataclass or a plain dict with `mode`
+(`INCLUDE` or `EXCLUDE`) and `values` (a non-empty list of source ID strings).
+The filter is serialized to `query.filters.source` on co-mention, volume, and
+search payloads during planning.
+
+```python
+from bigdata_smart_batching import SourceFilter, SourceMode, plan_search
+
+plan = plan_search(
+    universe="id_name_mapping_us_top_3000.csv",
+    start_date="2023-01-01",
+    end_date="2023-12-31",
+    text="merger acquisition",
+    source=SourceFilter(
+        mode=SourceMode.INCLUDE,
+        values=("source-id-1", "source-id-2"),
+    ),
+)
+
+# Equivalent dict form:
+plan = plan_search(
+    universe="id_name_mapping_us_top_3000.csv",
+    start_date="2023-01-01",
+    end_date="2023-12-31",
+    text="merger acquisition",
+    source={"mode": "EXCLUDE", "values": ["source-id-to-drop"]},
+)
+```
+
+`SourceMode` is the same `INCLUDE` / `EXCLUDE` enum as `CategoryMode`. String
+values are trimmed; empty strings are rejected. Use
+`validate_source_filter(source)` to validate and normalize input into the API
+dict shape.
+
 ### Sentiment Filtering
 
 Restrict searches to documents whose sentiment falls inside one or more
@@ -247,6 +285,7 @@ Step 2: EXECUTION
 | `min_period_days` | `int` | `30` | Minimum days per sub-period |
 | `min_entities_per_basket` | `int` | `1` | Lower bound on entities per basket |
 | `category` | `CategoryFilter \| dict \| None` | `None` | Optional category filter (`mode` + `values`) |
+| `source` | `SourceFilter \| dict \| None` | `None` | Optional source filter by source IDs (`mode` + `values`) |
 | `sentiment` | `SentimentRange \| dict \| list \| None` | `None` | Optional sentiment filter: one inclusive `{min, max}` range or a list of disjoint ranges |
 | `content_diversification` | `bool` | `True` | Enable/disable content diversification in ranking |
 | `requests_per_minute` | `int` | `100` | Global rate limit shared across all planning HTTP work (co-mention + volume) |
@@ -273,14 +312,18 @@ Step 2: EXECUTION
 - `load_universe_from_csv(csv_path)` -- load entity IDs from a CSV only.
 - `save_plan(plan, path)` / `load_plan(path)` -- persist plans as JSON.
 - `validate_category_filter(category)` -- validate and normalize a category filter to the API dict shape.
+- `validate_source_filter(source)` -- validate and normalize a source filter to the API dict shape (`filters.source`).
 - `validate_sentiment_range(sentiment)` -- validate and normalize sentiment range(s) into the list-of-`{min, max}` dicts emitted as `filters.sentiment.ranges`.
-- `build_comention_payload(...)`, `build_volume_payload(...)`, `build_search_query(...)` -- low-level payload builders used by the planner; useful if you need to construct custom requests.
+- `build_comention_payload(...)`, `build_volume_payload(...)`, `build_search_query(...)` -- low-level payload builders used by the planner; each accepts optional `category_filter`, `source_filter`, and `sentiment_filter` alongside entities and dates. Useful if you need to construct custom requests.
 
 ### Public Types
 
 - `CategoryFilter` -- frozen dataclass with `mode: CategoryMode` and `values: tuple[str, ...]`.
 - `CategoryMode` -- `StrEnum` with `INCLUDE` and `EXCLUDE` values.
 - `CategoryFilterInput` -- type alias: `CategoryFilter | dict[str, str | list[str]]`.
+- `SourceFilter` -- frozen dataclass with `mode: SourceMode` and `values: tuple[str, ...]` (source IDs).
+- `SourceMode` -- type alias of `CategoryMode` (`INCLUDE`, `EXCLUDE`).
+- `SourceFilterInput` -- type alias: `SourceFilter | dict[str, str | list[str]]`.
 - `SentimentRange` -- frozen dataclass with inclusive `min: float` and `max: float`.
 - `SentimentRangeInput` -- type alias for one `SentimentRange` / `{min, max}` dict, or a non-empty list of either.
 - `UniverseInput` -- type alias: `str | list[str]`.

{bigdata_smart_batching-1.2.1 → bigdata_smart_batching-1.3.1}/examples/example.py

@@ -67,6 +67,7 @@ def main() -> None:
                 # Optional: filter by category to restrict document sources.
                 # Accepts a CategoryFilter instance or a plain dict:
                 category={"mode": "INCLUDE", "values": ["news_premium"]},
+                source={"mode": "INCLUDE", "values": ["5A5702"]},
                 content_diversification=False,
                 sentiment=[{"min": -1, "max": -0.5},{"min": 0.5, "max": 1}],
             )
@@ -80,12 +81,12 @@ def main() -> None:
         print(f"Plan execution time: {plan_elapsed_s:.3f}s")
 
         # Step 2: Execute search with 10% of total chunks (preserves distribution)
-        if False:
+        if True:
             search_t0 = time.perf_counter()
             results_raw = execute_search(
                 search_plan=plan,
-                chunk_percentage=1,
-                requests_per_minute=300,
+                chunk_percentage=0.1,
+                requests_per_minute=450,
             )
             search_elapsed_s = time.perf_counter() - search_t0
             print(f"Search execution time: {search_elapsed_s:.3f}s")