pfeed 0.0.1.dev12__tar.gz → 0.0.1.dev13__tar.gz

pfeed-0.0.1.dev13/PKG-INFO

@@ -0,0 +1,267 @@
+Metadata-Version: 2.1
+Name: pfeed
+Version: 0.0.1.dev13
+Summary: Data pipeline for algo-trading, getting and storing both real-time and historical data made easy.
+Home-page: https://pfund.ai
+License: Apache-2.0
+Keywords: trading,algo-trading,data pipeline,ETL,data lake,data warehouse,data integration,historical data,live data,data streaming
+Author: Stephen Yau
+Author-email: softwareentrepreneer+pfeed@gmail.com
+Requires-Python: >=3.10,<3.13
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Provides-Extra: all
+Provides-Extra: boost
+Provides-Extra: data
+Provides-Extra: df
+Requires-Dist: beautifulsoup4 (>=4.12.3,<5.0.0)
+Requires-Dist: connectorx (>=0.3.3,<0.4.0) ; extra == "boost" or extra == "all"
+Requires-Dist: fastparquet (>=2024.5.0,<2025.0.0)
+Requires-Dist: minio (>=7.2.8,<8.0.0) ; extra == "data" or extra == "all"
+Requires-Dist: pandas (>=2.2.2,<3.0.0) ; extra == "df" or extra == "all"
+Requires-Dist: pfund (>=0.0.1.dev13,<0.0.2)
+Requires-Dist: polars (>=1.5.0,<2.0.0) ; extra == "df" or extra == "all"
+Requires-Dist: psutil (>=6.0.0,<7.0.0) ; extra == "data" or extra == "all"
+Requires-Dist: pyarrow (>=15.0.0,<16.0.0) ; extra == "boost" or extra == "all"
+Requires-Dist: ray (>=2.34.0,<3.0.0) ; extra == "boost" or extra == "all"
+Requires-Dist: s3fs (>=2024.6.1,<2025.0.0) ; extra == "data" or extra == "all"
+Requires-Dist: yfinance (>=0.2.43,<0.3.0)
+Project-URL: Documentation, https://pfeed-docs.pfund.ai
+Project-URL: Repository, https://github.com/PFund-Software-Ltd/pfeed
+Description-Content-Type: text/markdown
+
+# PFeed: Data Pipeline for Algo-Trading, Getting and Storing Real-Time and Historical Data Made Easy.
+
+![GitHub stars](https://img.shields.io/github/stars/PFund-Software-Ltd/pfeed?style=social)
+![PyPI downloads](https://img.shields.io/pypi/dm/pfeed?label=downloads)
+[![PyPI](https://img.shields.io/pypi/v/pfeed.svg)](https://pypi.org/project/pfeed)
+![PyPI - Support Python Versions](https://img.shields.io/pypi/pyversions/pfeed)
+[![Jupyter Book Badge](https://raw.githubusercontent.com/PFund-Software-Ltd/pfeed/main/docs/images/jupyterbook.svg)](https://jupyterbook.org)
+[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
+
+[MinIO]: https://min.io/
+[PFund]: https://github.com/PFund-Software-Ltd/pfund
+[Ray]: https://github.com/ray-project/ray
+[Polars]: https://github.com/pola-rs/polars
+[Prefect]: https://www.prefect.io
+[Timescaledb]: https://www.timescale.com/
+[Dask]: https://www.dask.org/
+[Spark]: https://spark.apache.org/docs/latest/api/python/index.html
+[DuckDB]: https://github.com/duckdb/duckdb
+[Daft]: https://github.com/Eventual-Inc/Daft
+[PyTrade.org]: https://pytrade.org
+[Databento]: https://databento.com/
+[Polygon]: https://polygon.io/
+[Bybit]: https://bybit.com/
+[FirstRate Data]: https://firstratedata.com
+
+## Problem
+Starting algo-trading requires reliable, clean data. However, the time-consuming and mundane tasks of data cleaning and storage often discourage traders from embarking on their algo-trading journey.
+
+## Solution
+By leveraging modern data engineering tools, `pfeed` handles the tedious data work and **outputs backtesting-ready data**, accelerating traders to get to the strategy development phase.
+
+---
+PFeed (/piː fiːd/) is a data pipeline for algorithmic trading, serving as a bridge between raw data sources and traders by automating the process of data collection, cleaning, transformation, and storage, loading clean data into a **local data lake for quantitative analysis**.
+
+## Core Features
+- [x] Unified approach for interacting with various data sources and obtaining historical and live data
+- [x] ETL data pipline for transforming raw data to clean data and storing it in [MinIO] (optional)
+- [x] Fast data downloading, utilizing [Ray] for parallelization
+- [x] Supports multiple data tools (e.g. Pandas, [Polars], [Dask], [Spark], [DuckDB], [Daft])
+- [ ] Integrates with [Prefect] to control data flows
+- [ ] Listens to PFund's trade engine and adds trade history to a local database [Timescaledb] (optional)
+
+> It is designed to be used alongside [PFund] — A Complete Algo-Trading Framework for Machine Learning, TradFi, CeFi and DeFi ready.
+
+---
+
+<details>
+<summary>Table of Contents</summary>
+
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+    - [Main Usage: Data Feed](#main-usage-data-feed)
+    - [Download Historical Data on Command Line](#download-historical-data-on-command-line)
+    - [Download Historical Data in Python](#download-historical-data-in-python)
+    - [List Current Config](#list-current-config)
+    - [Run PFeed's docker-compose.yml](#run-pfeeds-docker-composeyml)
+- [Supported Data Sources](#supported-data-sources)
+- [Supported Data Tools](#supported-data-tools)
+- [Related Projects](#related-projects)
+- [Disclaimer](#disclaimer)
+
+</details>
+
+
+## Installation
+### Using [Poetry](https://python-poetry.org) (Recommended)
+```bash
+# [RECOMMENDED]: Download data (e.g. Bybit and Yahoo Finance) + Data tools (e.g. pandas, polars) + Data storage (e.g. MinIO) + Boosted performance (e.g. Ray)
+poetry add "pfeed[all]"
+
+# [Download data + Data tools + Data storage]
+poetry add "pfeed[df,data]"
+
+# [Download data + Data tools]
+poetry add "pfeed[df]"
+
+# [Download data only]:
+poetry add pfeed
+
+# update to the latest version:
+poetry update pfeed
+```
+
+### Using Pip
+```bash
+# same as above, you can choose to install "pfeed[all]", "pfeed[df,data]", "pfeed[df]" or "pfeed"
+pip install "pfeed[all]"
+
+# install the latest version:
+pip install -U pfeed
+```
+
+### Checking your installation
+```bash
+$ pfeed --version
+```
+
+## Quick Start
+### 1. Get Historical Data in Dataframe (No storage)
+Get [Bybit]'s data in dataframe, e.g. 1-minute data (data is downloaded on the fly if not stored locally)
+
+```python
+import pfeed as pe
+
+feed = pe.BybitFeed(data_tool='polars')
+
+df = feed.get_historical_data(
+    'BTC_USDT_PERP',
+    resolution='1minute',  # 'raw' or '1tick'/'1t' or '2second'/'2s' etc.
+    start_date='2024-03-01',
+    end_date='2024-03-01',
+)
+```
+
+Printing the first few rows of `df`:
+|    | ts                  | product       | resolution   |    open |    high |     low |   close |   volume |
+|---:|:--------------------|:--------------|:-------------|--------:|--------:|--------:|--------:|---------:|
+|  0 | 2024-03-01 00:00:00 | BTC_USDT_PERP | 1m           | 61184.1 | 61244.5 | 61175.8 | 61244.5 |  159.142 |
+|  1 | 2024-03-01 00:01:00 | BTC_USDT_PERP | 1m           | 61245.3 | 61276.5 | 61200.7 | 61232.2 |  227.242 |
+|  2 | 2024-03-01 00:02:00 | BTC_USDT_PERP | 1m           | 61232.2 | 61249   | 61180   | 61184.2 |   91.446 |
+
+> By using pfeed, you are just a few lines of code away from a standardized dataframe, how convenient!
+
+
+
+### 2. Download Historical Data on the Command Line Interface (CLI)
+```bash
+# download data, default data type (dtype) is 'raw' data
+pfeed download -d BYBIT -p BTC_USDT_PERP --start-date 2024-03-01 --end-date 2024-03-08
+
+# download multiple products BTC_USDT_PERP and ETH_USDT_PERP and minute data
+pfeed download -d BYBIT -p BTC_USDT_PERP -p ETH_USDT_PERP --dtypes minute
+
+# download all perpetuals data from bybit
+pfeed download -d BYBIT --ptypes PERP
+
+# download all the data from bybit (CAUTION: your local machine probably won't have enough space for this!)
+pfeed download -d BYBIT
+
+# store data into MinIO (need to start MinIO by running `pfeed docker-compose up -d` first)
+pfeed download -d BYBIT -p BTC_USDT_PERP --use-minio
+
+# enable debug mode and turn off using Ray
+pfeed download -d BYBIT -p BTC_USDT_PERP --debug --no-ray
+```
+
+### 3. Download Historical Data in Python
+```python
+import pfeed as pe
+
+# compared to the CLI approach, this approach is more convenient for downloading multiple products
+pe.download(
+    data_source='bybit',
+    pdts=[
+        'BTC_USDT_PERP',
+        'ETH_USDT_PERP',
+        'BCH_USDT_PERP',
+    ],
+    dtypes=['raw'],  # data types, e.g. 'raw', 'tick', 'second', 'minute' etc.
+    start_date='2024-03-01',
+    end_date='2024-03-08',
+    use_minio=False,
+)
+```
+
+### List Current Config
+```bash
+# list the current config:
+pfeed config --list
+
+# change the data storage location to your local project's 'data' folder:
+pfeed config --data-path ./data
+
+# for more commands:
+pfeed --help
+```
+
+### Run PFeed's docker-compose.yml
+```bash
+# same as 'docker-compose', only difference is it has pointed to pfeed's docker-compose.yml file
+pfeed docker-compose [COMMAND]
+
+# e.g. start services
+pfeed docker-compose up -d
+
+# e.g. stop services
+pfeed docker-compose down
+```
+
+
+## Supported Data Sources
+| Data Source               | Get Historical Data | Download Historical Data | Get Live/Paper Data | Stream Live/Paper Data |
+| ------------------------- | ------------------- | ------------------------ | ------------------- | ---------------------- |
+| Yahoo Finance             | 🟢                  | ⚪                        | ⚪                  | ⚪                     |
+| Bybit                     | 🟢                  | 🟢                        | 🟡                  | 🔴                     |
+| *Interactive Brokers (IB) | 🔴                  | ⚪                        | 🔴                  | 🔴                     |
+| *[FirstRate Data]         | 🔴                  | 🔴                        | ⚪                  | ⚪                     |
+| [Databento]               | 🔴                  | 🔴                        | 🔴                  | 🔴                     |
+| [Polygon]                 | 🔴                  | 🔴                        | 🔴                  | 🔴                     |
+| Binance                   | 🔴                  | 🔴                        | 🔴                  | 🔴                     |
+| OKX                       | 🔴                  | 🔴                        | 🔴                  | 🔴                     |
+
+🟢 = finished \
+🟡 = in progress \
+🔴 = todo \
+⚪ = not applicable \
+\* = paid data
+
+
+## Supported Data Tools
+| Data Tools               | Supported |
+| ------------------------ | --------- |
+| Pandas                   | 🟢        |
+| [Polars]                 | 🟢        |
+| [Dask]                   | 🔴        |
+| [Spark]                  | 🔴        |
+| [DuckDB]                 | 🔴        |
+| [Daft]                   | 🔴        |
+
+
+## Related Projects
+- [PFund] — A Complete Algo-Trading Framework for Machine Learning, TradFi, CeFi and DeFi ready. Supports Vectorized and Event-Driven Backtesting, Paper and Live Trading
+- [PyTrade.org] - A curated list of Python libraries and resources for algorithmic trading.
+
+
+## Disclaimer
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+This framework is intended for educational and research purposes only. It should not be used for real trading without understanding the risks involved. Trading in financial markets involves significant risk, and there is always the potential for loss. Your trading results may vary. No representation is being made that any account will or is likely to achieve profits or losses similar to those discussed on this platform.
+
+The developers of this framework are not responsible for any financial losses incurred from using this software. This includes but not limited to losses resulting from inaccuracies in any financial data output by PFeed. Users should conduct their due diligence, verify the accuracy of any data produced by PFeed, and consult with a professional financial advisor before engaging in real trading activities.
+

pfeed-0.0.1.dev13/README.md

@@ -0,0 +1,231 @@
+# PFeed: Data Pipeline for Algo-Trading, Getting and Storing Real-Time and Historical Data Made Easy.
+
+![GitHub stars](https://img.shields.io/github/stars/PFund-Software-Ltd/pfeed?style=social)
+![PyPI downloads](https://img.shields.io/pypi/dm/pfeed?label=downloads)
+[![PyPI](https://img.shields.io/pypi/v/pfeed.svg)](https://pypi.org/project/pfeed)
+![PyPI - Support Python Versions](https://img.shields.io/pypi/pyversions/pfeed)
+[![Jupyter Book Badge](https://raw.githubusercontent.com/PFund-Software-Ltd/pfeed/main/docs/images/jupyterbook.svg)](https://jupyterbook.org)
+[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
+
+[MinIO]: https://min.io/
+[PFund]: https://github.com/PFund-Software-Ltd/pfund
+[Ray]: https://github.com/ray-project/ray
+[Polars]: https://github.com/pola-rs/polars
+[Prefect]: https://www.prefect.io
+[Timescaledb]: https://www.timescale.com/
+[Dask]: https://www.dask.org/
+[Spark]: https://spark.apache.org/docs/latest/api/python/index.html
+[DuckDB]: https://github.com/duckdb/duckdb
+[Daft]: https://github.com/Eventual-Inc/Daft
+[PyTrade.org]: https://pytrade.org
+[Databento]: https://databento.com/
+[Polygon]: https://polygon.io/
+[Bybit]: https://bybit.com/
+[FirstRate Data]: https://firstratedata.com
+
+## Problem
+Starting algo-trading requires reliable, clean data. However, the time-consuming and mundane tasks of data cleaning and storage often discourage traders from embarking on their algo-trading journey.
+
+## Solution
+By leveraging modern data engineering tools, `pfeed` handles the tedious data work and **outputs backtesting-ready data**, accelerating traders to get to the strategy development phase.
+
+---
+PFeed (/piː fiːd/) is a data pipeline for algorithmic trading, serving as a bridge between raw data sources and traders by automating the process of data collection, cleaning, transformation, and storage, loading clean data into a **local data lake for quantitative analysis**.
+
+## Core Features
+- [x] Unified approach for interacting with various data sources and obtaining historical and live data
+- [x] ETL data pipline for transforming raw data to clean data and storing it in [MinIO] (optional)
+- [x] Fast data downloading, utilizing [Ray] for parallelization
+- [x] Supports multiple data tools (e.g. Pandas, [Polars], [Dask], [Spark], [DuckDB], [Daft])
+- [ ] Integrates with [Prefect] to control data flows
+- [ ] Listens to PFund's trade engine and adds trade history to a local database [Timescaledb] (optional)
+
+> It is designed to be used alongside [PFund] — A Complete Algo-Trading Framework for Machine Learning, TradFi, CeFi and DeFi ready.
+
+---
+
+<details>
+<summary>Table of Contents</summary>
+
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+    - [Main Usage: Data Feed](#main-usage-data-feed)
+    - [Download Historical Data on Command Line](#download-historical-data-on-command-line)
+    - [Download Historical Data in Python](#download-historical-data-in-python)
+    - [List Current Config](#list-current-config)
+    - [Run PFeed's docker-compose.yml](#run-pfeeds-docker-composeyml)
+- [Supported Data Sources](#supported-data-sources)
+- [Supported Data Tools](#supported-data-tools)
+- [Related Projects](#related-projects)
+- [Disclaimer](#disclaimer)
+
+</details>
+
+
+## Installation
+### Using [Poetry](https://python-poetry.org) (Recommended)
+```bash
+# [RECOMMENDED]: Download data (e.g. Bybit and Yahoo Finance) + Data tools (e.g. pandas, polars) + Data storage (e.g. MinIO) + Boosted performance (e.g. Ray)
+poetry add "pfeed[all]"
+
+# [Download data + Data tools + Data storage]
+poetry add "pfeed[df,data]"
+
+# [Download data + Data tools]
+poetry add "pfeed[df]"
+
+# [Download data only]:
+poetry add pfeed
+
+# update to the latest version:
+poetry update pfeed
+```
+
+### Using Pip
+```bash
+# same as above, you can choose to install "pfeed[all]", "pfeed[df,data]", "pfeed[df]" or "pfeed"
+pip install "pfeed[all]"
+
+# install the latest version:
+pip install -U pfeed
+```
+
+### Checking your installation
+```bash
+$ pfeed --version
+```
+
+## Quick Start
+### 1. Get Historical Data in Dataframe (No storage)
+Get [Bybit]'s data in dataframe, e.g. 1-minute data (data is downloaded on the fly if not stored locally)
+
+```python
+import pfeed as pe
+
+feed = pe.BybitFeed(data_tool='polars')
+
+df = feed.get_historical_data(
+    'BTC_USDT_PERP',
+    resolution='1minute',  # 'raw' or '1tick'/'1t' or '2second'/'2s' etc.
+    start_date='2024-03-01',
+    end_date='2024-03-01',
+)
+```
+
+Printing the first few rows of `df`:
+|    | ts                  | product       | resolution   |    open |    high |     low |   close |   volume |
+|---:|:--------------------|:--------------|:-------------|--------:|--------:|--------:|--------:|---------:|
+|  0 | 2024-03-01 00:00:00 | BTC_USDT_PERP | 1m           | 61184.1 | 61244.5 | 61175.8 | 61244.5 |  159.142 |
+|  1 | 2024-03-01 00:01:00 | BTC_USDT_PERP | 1m           | 61245.3 | 61276.5 | 61200.7 | 61232.2 |  227.242 |
+|  2 | 2024-03-01 00:02:00 | BTC_USDT_PERP | 1m           | 61232.2 | 61249   | 61180   | 61184.2 |   91.446 |
+
+> By using pfeed, you are just a few lines of code away from a standardized dataframe, how convenient!
+
+
+
+### 2. Download Historical Data on the Command Line Interface (CLI)
+```bash
+# download data, default data type (dtype) is 'raw' data
+pfeed download -d BYBIT -p BTC_USDT_PERP --start-date 2024-03-01 --end-date 2024-03-08
+
+# download multiple products BTC_USDT_PERP and ETH_USDT_PERP and minute data
+pfeed download -d BYBIT -p BTC_USDT_PERP -p ETH_USDT_PERP --dtypes minute
+
+# download all perpetuals data from bybit
+pfeed download -d BYBIT --ptypes PERP
+
+# download all the data from bybit (CAUTION: your local machine probably won't have enough space for this!)
+pfeed download -d BYBIT
+
+# store data into MinIO (need to start MinIO by running `pfeed docker-compose up -d` first)
+pfeed download -d BYBIT -p BTC_USDT_PERP --use-minio
+
+# enable debug mode and turn off using Ray
+pfeed download -d BYBIT -p BTC_USDT_PERP --debug --no-ray
+```
+
+### 3. Download Historical Data in Python
+```python
+import pfeed as pe
+
+# compared to the CLI approach, this approach is more convenient for downloading multiple products
+pe.download(
+    data_source='bybit',
+    pdts=[
+        'BTC_USDT_PERP',
+        'ETH_USDT_PERP',
+        'BCH_USDT_PERP',
+    ],
+    dtypes=['raw'],  # data types, e.g. 'raw', 'tick', 'second', 'minute' etc.
+    start_date='2024-03-01',
+    end_date='2024-03-08',
+    use_minio=False,
+)
+```
+
+### List Current Config
+```bash
+# list the current config:
+pfeed config --list
+
+# change the data storage location to your local project's 'data' folder:
+pfeed config --data-path ./data
+
+# for more commands:
+pfeed --help
+```
+
+### Run PFeed's docker-compose.yml
+```bash
+# same as 'docker-compose', only difference is it has pointed to pfeed's docker-compose.yml file
+pfeed docker-compose [COMMAND]
+
+# e.g. start services
+pfeed docker-compose up -d
+
+# e.g. stop services
+pfeed docker-compose down
+```
+
+
+## Supported Data Sources
+| Data Source               | Get Historical Data | Download Historical Data | Get Live/Paper Data | Stream Live/Paper Data |
+| ------------------------- | ------------------- | ------------------------ | ------------------- | ---------------------- |
+| Yahoo Finance             | 🟢                  | ⚪                        | ⚪                  | ⚪                     |
+| Bybit                     | 🟢                  | 🟢                        | 🟡                  | 🔴                     |
+| *Interactive Brokers (IB) | 🔴                  | ⚪                        | 🔴                  | 🔴                     |
+| *[FirstRate Data]         | 🔴                  | 🔴                        | ⚪                  | ⚪                     |
+| [Databento]               | 🔴                  | 🔴                        | 🔴                  | 🔴                     |
+| [Polygon]                 | 🔴                  | 🔴                        | 🔴                  | 🔴                     |
+| Binance                   | 🔴                  | 🔴                        | 🔴                  | 🔴                     |
+| OKX                       | 🔴                  | 🔴                        | 🔴                  | 🔴                     |
+
+🟢 = finished \
+🟡 = in progress \
+🔴 = todo \
+⚪ = not applicable \
+\* = paid data
+
+
+## Supported Data Tools
+| Data Tools               | Supported |
+| ------------------------ | --------- |
+| Pandas                   | 🟢        |
+| [Polars]                 | 🟢        |
+| [Dask]                   | 🔴        |
+| [Spark]                  | 🔴        |
+| [DuckDB]                 | 🔴        |
+| [Daft]                   | 🔴        |
+
+
+## Related Projects
+- [PFund] — A Complete Algo-Trading Framework for Machine Learning, TradFi, CeFi and DeFi ready. Supports Vectorized and Event-Driven Backtesting, Paper and Live Trading
+- [PyTrade.org] - A curated list of Python libraries and resources for algorithmic trading.
+
+
+## Disclaimer
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+This framework is intended for educational and research purposes only. It should not be used for real trading without understanding the risks involved. Trading in financial markets involves significant risk, and there is always the potential for loss. Your trading results may vary. No representation is being made that any account will or is likely to achieve profits or losses similar to those discussed on this platform.
+
+The developers of this framework are not responsible for any financial losses incurred from using this software. This includes but not limited to losses resulting from inaccuracies in any financial data output by PFeed. Users should conduct their due diligence, verify the accuracy of any data produced by PFeed, and consult with a professional financial advisor before engaging in real trading activities.

pfeed-0.0.1.dev13/pfeed/__init__.py

@@ -0,0 +1,64 @@
+from __future__ import annotations
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from pfeed.types.common_literals import tSUPPORTED_DOWNLOAD_DATA_SOURCES, tSUPPORTED_DATA_TYPES
+
+import importlib
+from importlib.metadata import version
+
+from pfeed import etl
+from pfeed.config_handler import configure, get_config
+from pfeed.const.common import ALIASES
+from pfeed.sources import bybit
+from pfeed.feeds import BybitFeed, YahooFinanceFeed
+
+
+def download_historical_data(
+    data_source: tSUPPORTED_DOWNLOAD_DATA_SOURCES,
+    pdts: str | list[str] | None = None,
+    dtypes: tSUPPORTED_DATA_TYPES | list[tSUPPORTED_DATA_TYPES] | None = None,
+    ptypes: str | list[str] | None = None,
+    start_date: str | None = None,
+    end_date: str | None = None,
+    num_cpus: int = 8,
+    use_ray: bool = True,
+    use_minio: bool = False,
+):
+    data_source = importlib.import_module(f"pfeed.sources.{data_source.lower()}")
+    return data_source.download_historical_data(
+        pdts=pdts,
+        dtypes=dtypes,
+        ptypes=ptypes,
+        start_date=start_date,
+        end_date=end_date,
+        num_cpus=num_cpus,
+        use_ray=use_ray,
+        use_minio=use_minio,
+    )
+
+
+# TODO
+def stream_realtime_data(data_source: tSUPPORTED_DOWNLOAD_DATA_SOURCES):
+    data_source = importlib.import_module(f"pfeed.sources.{data_source.lower()}")
+    return data_source.stream_realtime_data()
+    
+
+
+download = download_historical_data
+stream = stream_realtime_data
+
+
+__version__ = version("pfeed")
+__all__ = (
+    "__version__",
+    "configure",
+    "get_config",
+    "ALIASES",
+    "etl",
+    "bybit",
+    "binance",
+    "YahooFinanceFeed",
+    "BybitFeed",
+    "BinanceFeed",
+)

{pfeed-0.0.1.dev12 → pfeed-0.0.1.dev13}/pfeed/cli/commands/config.py

@@ -10,7 +10,7 @@ from pfeed.config_handler import ConfigHandler
 
 
 def save_config(config: ConfigHandler, config_file_path: str | Path):
-    if type(config_file_path) is str:
+    if isinstance(config_file_path, str):
         config_file_path = Path(config_file_path)
     config_file_path.parent.mkdir(parents=True, exist_ok=True)
     with open(config_file_path, 'w') as f:
@@ -31,6 +31,8 @@ def remove_config(config_file_path: str | Path):
 @click.option('--logging-config', type=dict, help='Set the logging config')
 @click.option('--use-fork-process', type=bool, help='If True, multiprocessing.set_start_method("fork")')
 @click.option('--use-custom-excepthook', type=bool, help='If True, log uncaught exceptions to file')
+@click.option('--env-file', 'env_file_path', type=click.Path(resolve_path=True, exists=True), help='Path to the .env file')
+@click.option('--debug', is_flag=True, help='if enabled, debug mode will be enabled where logs at DEBUG level will be printed')
 @click.option('--list', '-l', is_flag=True, is_eager=True, help='List all available options')
 @click.option('--reset', is_flag=True, is_eager=True, help='Reset the configuration to defaults') 
 def config(ctx, **kwargs):

{pfeed-0.0.1.dev12 → pfeed-0.0.1.dev13}/pfeed/cli/commands/docker_compose.py

@@ -3,7 +3,6 @@ from pathlib import Path
 import importlib.resources
 import subprocess
 
-from dotenv import find_dotenv, load_dotenv
 import click
 
 from pfeed.const.paths import PROJ_NAME
@@ -18,14 +17,8 @@ from pfeed.const.paths import PROJ_NAME
 @click.option('--docker-file', 'docker_file_path', type=click.Path(exists=True), help='Path to the docker-compose.yml file')
 def docker_compose(ctx, env_file_path, docker_file_path):
     """Forwards commands to docker-compose with the package's docker-compose.yml file if not specified."""
-    if not env_file_path:
-        if env_file_path := find_dotenv(usecwd=True, raise_error_if_not_found=False):
-            click.echo(f'.env file path is not specified, using env file in "{env_file_path}"')
-        else:
-            click.echo('.env file is not found')
-    load_dotenv(env_file_path, override=True)
-    
     config = ctx.obj['config']
+    config.load_env_file(env_file_path)
     os.environ['PFEED_DATA_PATH'] = config.data_path
     
     if not docker_file_path:

{pfeed-0.0.1.dev12 → pfeed-0.0.1.dev13}/pfeed/cli/commands/download.py

@@ -1,9 +1,9 @@
 import importlib
 
 import click
-from dotenv import find_dotenv, load_dotenv
 
-from pfeed.const.commons import ALIASES, SUPPORTED_DOWNLOAD_DATA_SOURCES, SUPPORTED_DATA_TYPES
+import pfeed as pe
+from pfeed.const.common import ALIASES, SUPPORTED_DOWNLOAD_DATA_SOURCES, SUPPORTED_DATA_TYPES
 
 
 # add aliases to supported download data sources
@@ -16,38 +16,28 @@ SUPPORTED_DATA_TYPES_IMPLICIT_RAW_ALLOWED = SUPPORTED_DATA_TYPES + ['raw']
 
 @click.command()
 @click.pass_context
-@click.option('--env-file', 'env_file_path', type=click.Path(exists=True), help='Path to the .env file')
 @click.option('--data-source', '-d', required=True, type=click.Choice(SUPPORTED_DOWNLOAD_DATA_SOURCES_ALIASES_INCLUDED, case_sensitive=False), help='Data source')
-@click.option('--dtype', '--dt', 'dtypes', multiple=True, default=['raw'], type=click.Choice(SUPPORTED_DATA_TYPES_IMPLICIT_RAW_ALLOWED, case_sensitive=False), help=f'{SUPPORTED_DATA_TYPES=}. How to pass in multiple values: --dt raw --dt tick')
-@click.option('--pdt', '-p', 'pdts', multiple=True, default=[], help='List of trading products')
-@click.option('--ptype', '--pt', 'ptypes', multiple=True, default=[], help='List of product types, e.g. PERP = get all perpetuals')
+@click.option('--dtypes', '--dt', 'dtypes', multiple=True, default=['raw'], type=click.Choice(SUPPORTED_DATA_TYPES_IMPLICIT_RAW_ALLOWED, case_sensitive=False), help=f'{SUPPORTED_DATA_TYPES=}. How to pass in multiple values: --dt raw --dt tick')
+@click.option('--pdts', '-p', 'pdts', multiple=True, default=[], help='List of trading products')
+@click.option('--ptypes', '--pt', 'ptypes', multiple=True, default=[], help='List of product types, e.g. PERP = get all perpetuals')
 @click.option('--start-date', '-s', type=click.DateTime(formats=["%Y-%m-%d"]), help='Start date in YYYY-MM-DD format')
 @click.option('--end-date', '-e', type=click.DateTime(formats=["%Y-%m-%d"]), help='End date in YYYY-MM-DD format')
-@click.option('--batch-size', default=8, type=int, help='batch size for Ray tasks')  # REVIEW
+@click.option('--num-cpus', '-n', default=8, type=int, help="number of logical CPUs used for Ray's tasks")
+@click.option('--use-minio', '-m', is_flag=True, help='if enabled, data will be loaded into Minio')
 @click.option('--no-ray', is_flag=True, help='if enabled, Ray will not be used')
-@click.option('--use-minio', is_flag=True, help='if enabled, data will be loaded into Minio')
+@click.option('--env-file', 'env_file_path', type=click.Path(exists=True), help='Path to the .env file')
 @click.option('--debug', is_flag=True, help='if enabled, debug mode will be enabled where logs at DEBUG level will be printed')
-def download(ctx, env_file_path, data_source, pdts, dtypes, ptypes, start_date, end_date, batch_size, no_ray, use_minio, debug):
-    if not env_file_path:
-        if env_file_path := find_dotenv(usecwd=True, raise_error_if_not_found=False):
-            click.echo(f'.env file path is not specified, using env file in "{env_file_path}"')
-        else:
-            click.echo('.env file is not found')
-    load_dotenv(env_file_path, override=True)
-    
-    if data_source in ALIASES:
-        data_source = ALIASES[data_source]
-        
+def download(data_source, dtypes, pdts, ptypes, start_date, end_date, num_cpus, no_ray, use_minio, env_file_path, debug):
+    pe.configure(env_file_path=env_file_path, debug=debug)
+    data_source = ALIASES.get(data_source, data_source)
     pipeline = importlib.import_module(f'pfeed.sources.{data_source.lower()}.download')
     pipeline.download_historical_data(
         pdts=pdts,
-        dtypes=list(dtypes),
-        ptypes=list(ptypes),
+        dtypes=dtypes,
+        ptypes=ptypes,
         start_date=start_date.date().strftime('%Y-%m-%d') if start_date else start_date,
         end_date=end_date.date().strftime('%Y-%m-%d') if end_date else end_date,
-        batch_size=batch_size,
+        num_cpus=num_cpus,
         use_ray=not no_ray,
         use_minio=use_minio,
-        debug=debug,
-        config=ctx.obj['config'],
     )