market-data-normalizer 1.3.1__tar.gz

market_data_normalizer-1.3.1/.github/workflows/ci.yml

@@ -0,0 +1,27 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          pip install pytest
+      - name: Run tests
+        run: pytest -q

market_data_normalizer-1.3.1/.github/workflows/publish.yml

@@ -0,0 +1,38 @@
+name: Publish to PyPI
+
+# Publishes a release to PyPI when a version tag is pushed.
+# Authentication uses PyPI Trusted Publishing (OIDC) — no long-lived
+# API token is stored anywhere.
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install pytest
+      - run: pytest -q
+
+  publish:
+    needs: test
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write        # required for Trusted Publishing
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Build distributions
+        run: |
+          python -m pip install --upgrade build
+          python -m build
+      - name: Publish
+        uses: pypa/gh-action-pypi-publish@release/v1

market_data_normalizer-1.3.1/.gitignore

@@ -0,0 +1,28 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+env/
+
+# Tooling
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# OS / editor
+.DS_Store
+.idea/
+.vscode/
+
+# Never commit secrets
+*.env
+.env
+*.token
+secrets*

market_data_normalizer-1.3.1/CHANGELOG.md

@@ -0,0 +1,140 @@
+# Changelog
+
+All notable changes to this project are documented here.
+The format is based on [Keep a Changelog](https://keepachangelog.com/).
+
+## [1.3.1] - 2026-08-10
+
+### Changed
+- Packaging: the project is now published on PyPI as
+  `market-data-normalizer` (`pip install market-data-normalizer`; the
+  import name stays `mdnorm`). Added Python version classifiers, project
+  URLs for the changelog and issue tracker, and an install section in the
+  README. Releases are published from CI via PyPI Trusted Publishing, so
+  no long-lived API token exists. No library code changed in this release.
+
+## [1.3.0] - 2026-08-08
+
+### Added
+- Trading sessions and calendar filtering (`mdnorm.sessions`): a `Session`
+  describes a recurring local-time window — `in_session`, `filter_session`,
+  `session_date` and `group_by_session_date` decide what belongs to it.
+  Handles intraday windows, overnight sessions that cross midnight, and
+  daylight-saving transitions via `zoneinfo`; ready-made `US_EQUITY_RTH`
+  and `US_FUTURES_OVERNIGHT` are included.
+- Matching `Pipeline` step (`.session(...)`) and CLI flags `--session
+  HH:MM-HH:MM` and `--tz ZONE`, applied before aggregation.
+
+### Fixed
+- `canonical_symbol` mangled single-listed instruments: tickers without a
+  quote leg were split by a blind 3-character rule, turning `AAPL` into
+  `A-APL`. Equities, ETFs and indices now keep their ticker (`AAPL`,
+  `SPY`, `BRK.B`), while traded pairs are unchanged.
+
+## [1.2.0] - 2026-08-07
+
+### Added
+- Event-driven bars, the standard alternatives to time bars: ``count_bars``
+  (one bar per N trades), ``volume_bars`` (close at a cumulative base-unit
+  threshold) and ``dollar_bars`` (close at a traded-notional threshold).
+  For these bars ``start_ns`` is the first trade's timestamp and
+  ``interval_ns`` the realized span; the trailing partial bar is included.
+- Matching ``Pipeline`` steps (``.count_bars()``, ``.volume_bars()``,
+  ``.dollar_bars()``) and CLI flags (``--every-trades``, ``--every-volume``,
+  ``--every-notional``) as alternatives to ``--interval``.
+
+## [1.1.0] - 2026-08-05
+
+### Added
+- Transparent gzip support across all file I/O: any ``.gz`` path
+  (``.csv.gz``, ``.jsonl.gz``, ``.ndjson.gz``) is compressed/decompressed
+  automatically in ``read_csv_trades`` / ``write_records_csv`` /
+  ``write_jsonl`` / ``read_jsonl_events`` and the CLI. Standard library only.
+- Streaming readers for large files: ``iter_csv_trades`` and
+  ``iter_jsonl_events`` yield normalized events one at a time instead of
+  loading the whole file into memory.
+
+## [1.0.0] - 2026-08-04
+
+### Added
+- `Pipeline` — declarative, reusable processing chains: compose `dedupe`,
+  `clean`, `time_bars`, `resample`, `fill_gaps` (plus custom steps via
+  `apply`) and run the same pipeline across venues and files. Quality
+  reports from `clean` are exposed on `pipeline.last_issues`.
+- NDJSON / JSON Lines I/O: `write_jsonl` (events and bars, one compact JSON
+  object per line) and `read_jsonl_events` / `event_from_dict` for lossless
+  round-trips. Standard library only.
+- Command-line interface: `mdnorm bars`, `mdnorm quality` and
+  `mdnorm convert` (CSV <-> NDJSON), with human-friendly intervals
+  (`30s`, `1m`, `4h`, `1d`). Installed as the `mdnorm` console script;
+  also runnable as `python -m mdnorm`.
+
+### Changed
+- Project status raised to stable (`Development Status :: 5`); the public
+  API of `0.x` is carried over unchanged.
+
+## [0.9.0] - 2026-08-03
+
+### Added
+- File-level CSV I/O: `read_csv_trades(path, ...)` reads a CSV of trades into
+  normalized events, and `write_records_csv(items, path)` writes events/bars
+  to a CSV (union of fields). Standard library only.
+
+## [0.8.0] - 2026-08-02
+
+### Added
+- Stream consolidation: `merge_streams(*streams)` merges multiple venue feeds
+  into one timestamp-ordered timeline (stable), and `dedupe(events)` drops exact
+  duplicate events from reconnects/replays, preserving first-seen order.
+
+## [0.7.0] - 2026-08-01
+
+### Added
+- Serialization: `event_to_dict`, `bar_to_dict` and `to_records` flatten
+  events and bars into plain, JSON-serialisable dicts (Decimals as strings by
+  default, `as_float=True` for numeric output) — ready for pandas / CSV / JSON.
+
+## [0.6.0] - 2026-07-31
+
+### Added
+- `fill_gaps(bars)` — return a gapless bar series, inserting flat zero-volume
+  bars (OHLC = previous close) for any missing interval. Pairs with
+  `time_bars` and `resample_bars` for a continuous grid.
+
+## [0.5.0] - 2026-07-30
+
+### Added
+- `resample_bars(bars, interval_ns)` — downsample OHLCV bars to a coarser
+  interval (e.g. 1-minute to 5-minute) with correct OHLC aggregation and
+  volume-weighted VWAP.
+
+## [0.4.0] - 2026-07-30
+
+### Added
+- Data-quality module: `find_issues` and `clean` detect and drop bad ticks
+  (price outliers), gaps, out-of-order records and non-positive price/size,
+  returning a structured `QualityIssue` report.
+
+## [0.3.0] - 2026-07-28
+
+### Added
+- OHLCV time-bar aggregation: `time_bars(events, interval_ns)` and the `Bar`
+  type (open/high/low/close/volume/trades/vwap). Handles out-of-order input
+  and ignores non-trade events.
+
+## [0.2.0] - 2026-07-27
+
+### Added
+- Quote (bid/ask) normalization: `from_ws_quote` (exchange book-ticker
+  messages) and `from_csv_quote` (CSV bid/ask rows).
+- `MarketEvent.mid_price` and `MarketEvent.spread` convenience properties.
+- Dedicated test suite for quote events (`tests/test_quotes.py`).
+
+## [0.1.0] - 2026-07-27
+
+### Added
+- Initial release.
+- Unified `MarketEvent` schema (Decimal prices, ns UTC timestamps,
+  canonical `BASE-QUOTE` symbols).
+- Trade normalizers for CSV, exchange WebSocket JSON, and FIX.
+- Cross-venue equivalence tests.

market_data_normalizer-1.3.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 HarvestGroup360 (AMII LTD)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

market_data_normalizer-1.3.1/PKG-INFO

@@ -0,0 +1,330 @@
+Metadata-Version: 2.4
+Name: market-data-normalizer
+Version: 1.3.1
+Summary: Normalize heterogeneous market-data feeds (CSV, WebSocket JSON, FIX) into one exchange-agnostic schema.
+Project-URL: Homepage, https://harvestgroup360.com
+Project-URL: Repository, https://github.com/Harvestgroup360/market-data-normalizer
+Project-URL: Changelog, https://github.com/Harvestgroup360/market-data-normalizer/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/Harvestgroup360/market-data-normalizer/issues
+Author-email: HarvestGroup360 <github@harvestgroup360.com>
+License: MIT License
+        
+        Copyright (c) 2026 HarvestGroup360 (AMII LTD)
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: fix-protocol,market-data,normalization,quantitative-finance,tick-data,trading
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Financial and Insurance Industry
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Office/Business :: Financial :: Investment
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# market-data-normalizer (`mdnorm`)
+
+[![CI](https://github.com/Harvestgroup360/market-data-normalizer/actions/workflows/ci.yml/badge.svg)](https://github.com/Harvestgroup360/market-data-normalizer/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml)
+[![PyPI](https://img.shields.io/pypi/v/market-data-normalizer.svg)](https://pypi.org/project/market-data-normalizer/)
+
+Normalize heterogeneous market-data feeds — CSV tick dumps, exchange
+WebSocket JSON, and FIX — into a single, exchange-agnostic event schema, so
+downstream research and execution code never has to care where a tick came
+from.
+
+Zero runtime dependencies. Pure Python (3.10+). `Decimal` prices, integer
+nanosecond timestamps.
+
+## Why
+
+Every venue spells the same thing differently: `BTCUSDT` vs `XBT/USD`,
+millisecond epochs vs FIX `UTCTimestamp`, `is_buyer_maker` booleans vs side
+codes. Research notebooks and backtesters end up littered with per-venue
+parsing branches. `mdnorm` pushes that mess to the edge and hands the rest of
+your stack one clean type.
+
+## Install
+
+```console
+pip install market-data-normalizer
+```
+
+The distribution is named `market-data-normalizer`; the import name is
+`mdnorm`:
+
+```python
+import mdnorm
+```
+
+Pure Python, no runtime dependencies, Python 3.10+.
+
+## Quick start
+
+```python
+from mdnorm import from_csv_row, from_ws_json, from_fix
+
+# CSV row (ISO-8601 timestamp)
+from_csv_row(
+    {"symbol": "btc/usd", "ts": "2026-01-02T00:00:00Z",
+     "price": "42000.5", "size": "0.25", "side": "buy"},
+    venue="coinbase",
+)
+
+# Exchange WebSocket trade message
+from_ws_json({"s": "BTCUSDT", "p": "42000.5", "q": "0.25",
+              "T": 1767312000000, "m": False}, venue="binance")
+
+# FIX execution report (SOH-delimited in the wild; "|" here for readability)
+from_fix("55=BTC/USD|31=42000.5|32=0.25|54=1|60=20260102-00:00:00",
+         venue="lmax", sep="|")
+```
+
+All three calls above produce the **same** `MarketEvent`.
+
+### Quotes (bid/ask)
+
+```python
+from mdnorm import from_ws_quote
+
+q = from_ws_quote(
+    {"s": "BTCUSDT", "b": "41999.5", "B": "1.2",
+     "a": "42000.5", "A": "0.8", "T": 1767312000000},
+    venue="binance",
+)
+q.mid_price   # Decimal("42000.0")
+q.spread      # Decimal("1.0")
+```
+
+`from_csv_quote` does the same for CSV rows with bid/ask columns.
+
+### OHLCV bars
+
+```python
+from mdnorm import time_bars
+
+bars = time_bars(events, interval_ns=60_000_000_000)  # 1-minute bars
+bars[0].open, bars[0].high, bars[0].low, bars[0].close, bars[0].volume, bars[0].vwap
+```
+
+`time_bars` reduces a stream of trade events into fixed-interval OHLCV `Bar`s
+(with VWAP and trade count), sorting out-of-order input and skipping quotes.
+
+`resample_bars(bars, interval_ns)` downsamples bars to a coarser interval
+(e.g. 1-minute → 5-minute) with correct OHLC aggregation and volume-weighted
+VWAP.
+
+`fill_gaps(bars)` returns a gapless series, inserting flat zero-volume bars
+(OHLC = previous close) for any interval with no trades — a continuous grid for
+backtests and feature pipelines.
+
+### Event-driven bars
+
+Time bars are not the only clock. Sample by activity instead:
+
+```python
+from decimal import Decimal
+from mdnorm import count_bars, volume_bars, dollar_bars
+
+count_bars(events, every=500)                       # tick bars
+volume_bars(events, min_volume=Decimal("100"))      # volume bars
+dollar_bars(events, min_notional=Decimal("1e6"))    # dollar bars
+```
+
+### Trading sessions
+
+Filter a feed down to the hours that matter, with daylight saving handled
+for you:
+
+```python
+from mdnorm import US_EQUITY_RTH, filter_session, group_by_session_date
+
+rth = filter_session(events, US_EQUITY_RTH)        # 09:30-16:00 New York
+by_day = group_by_session_date(events, US_EQUITY_RTH)
+```
+
+Overnight windows (a session that opens at 18:00 and closes at 17:00 the
+next day) are supported, and `session_date` keeps a whole night in one
+bucket. From the command line:
+
+```console
+$ mdnorm bars trades.csv --interval 5m --session 09:30-16:00 --tz America/New_York -o rth.csv
+```
+
+### Data quality
+
+```python
+from mdnorm.quality import find_issues, clean
+
+find_issues(events)          # list of QualityIssue (outlier / gap / out_of_order / non_positive)
+cleaned, issues = clean(events)  # drop bad ticks & invalid rows, keep a report
+```
+
+`clean` removes price outliers and non-positive price/size records and returns
+the surviving events plus everything it flagged.
+
+### Serialization
+
+```python
+from mdnorm import to_records
+
+to_records(events)                 # list of flat dicts (Decimals as strings)
+to_records(bars, as_float=True)    # numeric output for DataFrames
+```
+
+`to_records` (and `event_to_dict` / `bar_to_dict`) flatten events and bars into
+plain, JSON-serialisable dicts — drop straight into `pandas.DataFrame`, a
+`csv.DictWriter`, or `json.dumps`.
+
+### Consolidating streams
+
+```python
+from mdnorm import merge_streams, dedupe
+
+timeline = dedupe(merge_streams(binance_events, coinbase_events))
+```
+
+`merge_streams` interleaves multiple venue feeds into one timestamp-ordered
+timeline; `dedupe` drops exact duplicate events left behind by reconnects and
+replays.
+
+### CSV files
+
+```python
+from mdnorm import read_csv_trades, write_records_csv
+
+events = read_csv_trades("trades.csv", venue="coinbase")   # file -> events
+write_records_csv(bars, "bars.csv", as_float=True)          # events/bars -> file
+```
+
+`read_csv_trades` parses a whole CSV of trades into normalized events;
+`write_records_csv` writes events or bars back out. Standard library only.
+
+### NDJSON / JSON Lines
+
+```python
+from mdnorm import write_jsonl, read_jsonl_events
+
+write_jsonl(events, "events.jsonl")          # one JSON object per line
+events2 = read_jsonl_events("events.jsonl")  # lossless round-trip
+
+# large files: stream lazily, .gz handled transparently
+for e in iter_jsonl_events("dump.jsonl.gz"):
+    ...
+```
+
+### Pipelines
+
+Declare a processing chain once, reuse it everywhere:
+
+```python
+from decimal import Decimal
+from mdnorm import Pipeline
+
+pipe = (
+    Pipeline()
+    .dedupe()
+    .clean(max_return=Decimal("0.1"))
+    .time_bars(60_000_000_000)   # 1-minute bars
+    .fill_gaps()
+)
+bars = pipe.run(events)
+print(pipe.last_issues)          # quality report from clean()
+```
+
+### Command line
+
+The common conversions ship as a zero-dependency CLI:
+
+```console
+$ mdnorm bars trades.csv --venue binance --interval 1m -o bars.csv
+$ mdnorm quality trades.csv --max-gap 5m
+$ mdnorm convert trades.csv -o trades.jsonl
+```
+
+Also available as `python -m mdnorm`.
+
+## The unified schema
+
+```python
+@dataclass(frozen=True, slots=True)
+class MarketEvent:
+    symbol: str          # canonical "BASE-QUOTE", e.g. "BTC-USD"
+    venue: str           # source venue
+    event_type: EventType  # TRADE | QUOTE
+    ts_ns: int           # nanoseconds since Unix epoch (UTC)
+    price: Decimal | None
+    size:  Decimal | None
+    side:  Side | None     # BUY | SELL
+    # ... plus bid/ask fields for quotes
+```
+
+## Design notes
+
+- **Money is `Decimal`.** Prices and sizes never touch binary floats, so
+  `42000.10` stays `42000.10`.
+- **Time is integer nanoseconds, UTC.** One comparable integer regardless of
+  whether the source gave seconds, milliseconds, or a FIX timestamp string.
+- **Symbols are canonicalized** to `BASE-QUOTE`, with venue aliases resolved
+  (`XBT` → `BTC`) and quote currencies detected longest-match-first so
+  `USDT` wins over `USD`.
+- **Normalizers are pure functions** — one raw record in, one `MarketEvent`
+  out — which keeps them trivial to unit-test and compose into any streaming
+  or batch pipeline.
+
+## Architecture
+
+```
+raw feed ──► normalizer ─────────────► MarketEvent ──► your pipeline
+ (CSV /      (from_csv_row /            (unified,       (research,
+  WS JSON /   from_ws_json /             immutable)      backtest,
+  FIX)        from_fix)                                  execution)
+                    │
+                    ├── symbols.canonical_symbol()   BTCUSDT → BTC-USDT
+                    └── timeutil.*_to_ns()           any time → ns UTC
+```
+
+## Tests
+
+```bash
+pip install pytest
+pytest -q
+```
+
+The suite includes a cross-venue equivalence test proving CSV, WebSocket and
+FIX representations of one trade collapse to an identical event.
+
+## License
+
+MIT © HarvestGroup360 (AMII LTD). See [LICENSE](LICENSE).
+
+---
+
+Maintained by [HarvestGroup360](https://harvestgroup360.com) as part of our
+open quantitative-infrastructure tooling.