featureSQL 0.1.0__tar.gz

featuresql-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE

featuresql-0.1.0/LICENSE.qlib

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE

featuresql-0.1.0/PKG-INFO

@@ -0,0 +1,279 @@
+Metadata-Version: 2.4
+Name: featureSQL
+Version: 0.1.0
+Summary: Add your description here
+Author: 
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: LICENSE.qlib
+Requires-Dist: build>=1.4.0
+Requires-Dist: duckdb>=1.4.4
+Requires-Dist: fire>=0.7.1
+Requires-Dist: google-auth>=2.48.0
+Requires-Dist: google-cloud-storage>=3.9.0
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: numpy>=2.4.2
+Requires-Dist: pytest>=9.0.2
+Requires-Dist: yahooquery>=2.4.1
+Dynamic: license-file
+Dynamic: requires-python
+
+This repository is now packaged as the **featureSQL** Python module.  You can
+install it in one of two ways:
+
+1. **From PyPI:**
+
+   ```bash
+   pip install featureSQL
+   ```
+
+   This makes the `featureSQL` console script available on your PATH and lets
+   you `import featureSQL` from any Python program.
+
+2. **From source (development mode):**
+
+   ```bash
+   cd /path/to/featureSQL
+   pip install -e .
+   ```
+
+   This installs the package in editable mode, so local edits are reflected
+   immediately without reinstalling.  Useful when working on the project.
+
+After installation you can still use the original CLI helpers directly by
+importing:
+
+```python
+from featureSQL.cli import Run
+```
+
+and invoking `Run().download(...)` or `featureSQL` at the shell.  The remainder
+of this document is a how‑to reference that you can include in your own
+projects; it covers the three common workflows you asked about.
+
+---
+
+## 1. Download a list of symbols (OCHLVF) to CSV
+
+```bash
+# 1. prepare a text file listing tickers, one per line
+cat > .testsymbols.txt <<'EOF'
+AAPL
+AMZN
+GOOG
+TSLA
+EOF
+
+# 2. run the collector, writing CSVs into a directory (they’ll land in
+#    the `feature-csv` subfolder of whatever you pass to --data_path)
+uv run -m featureSQL.cli download \
+    --region US \
+    --start 2026-01-01 \
+    --end   2026-02-28 \
+    --symbols_file ./.testsymbols.txt \
+    --data_path ./source \
+    --store_type fs   # Output to local fs (default). Change to 'gcs' and set data_path to bucket name for GCS.
+```
+
+> **GCS setup:** when using `--store_type gcs` you must configure two
+> environment variables before running any commands or tests:
+>
+> ```bash
+> export GCS_SC_JSON='{"type":"service_account", …}'    # credentials
+> export GCS_BUCKET_NAME='your-bucket-name'               # target bucket
+> ```
+>
+> The unit tests will skip network‑dependent scenarios if `GCS_BUCKET_NAME`
+> is unset, and they mock the client in other cases.  However, any manual
+> invocation against a real bucket requires both variables to be set.
+
+*Files created*: `/path/to/target/csv/dir/feature-csv/AAPL.csv`,
+`feature-csv/AMZN.csv`, … (i.e. `feature-csv` under the directory you
+passed with `--data_path`).  Each CSV contains the usual Open/Close/High/Low/
+Volume/AdjClose data plus `symbol`/`date` columns.
+
+---
+
+## 2. Convert a directory of CSVs into a binary dataset
+
+This is the “dump all” step: it reads **all** CSV files in `data_path` and builds the calendar, instruments list and feature bins in the standard binary layout.
+
+```bash
+uv run -m featureSQL.dump_bin dump_all \
+    --data_path ./source/feature-csv \
+    --dump_dir   ./source/ \
+    --exclude_fields symbol,date \
+    --store_type fs         # don’t try to treat metadata as floats.
+# (./source/ is /path/to/output/dir/)
+```
+
+After the command finishes you’ll have a structure like:
+
+```
+/path/to/output/dir/
+  calendars/day.txt          # trading dates
+  instruments/all.txt        # ticker start/end dates
+  features/<ticker>/<field>.day.bin …
+```
+
+This directory can be consumed by any tool that understands the same layout – it’s a full **initialised** dataset.
+
+> 💡 run `dump_all` only once per collection.  To add new data later, use `dump_update` (see next section).
+
+---
+
+## 3. Download symbols and produce the binary dataset in one go
+
+The built‑in CLI helper (exposed via the `featureSQL` script or as
+`python -m featureSQL.cli`) can drive both phases and even run ad‑hoc
+queries against a binary dataset:
+
+```bash
+uv run -m featureSQL.cli download \
+    --region US \
+    --start 2026-01-01 \
+    --end   2026-02-28 \
+    --symbols AMZN,GOOG,TSLA \
+    --data_path source  \
+    --out_format bin \
+    --store_type fs
+```
+
+This will
+
+1. fetch OCHLVF data from Yahoo and save raw CSVs under
+   `--{data_path}/feature-csv` (or `./source` if you didn’t pass
+   `--data_path`),
+2. run the binary dumper. On the **first** invocation the helper uses
+   `DumpDataAll` to initialise the dataset; subsequent runs use
+   `DumpDataUpdate` to append new days.  The target directory is the same as
+   `--data_path` and you can still provide `exclude_fields="symbol,date"`.
+
+The first run must be preceded by an empty or non‑existent output directory
+(and/or you can manually run `dump_all` as shown above); thereafter the
+same command will **append** new days to the existing bins.
+
+## 4. Query the binary dataset using SQL
+
+Once you have an initialised dataset you can issue SQL against it.  The
+``query`` subcommand lazily loads only the symbols mentioned in the query
+and keeps a small LRU cache in memory.
+
+```bash
+uv run -m featureSQL.cli query \
+    --data_path source \
+    --max_symbols 100 \
+    --max_memory 2000000000 \
+    --store_type fs \
+    "select date, open, close, high, low, volume, adjclose from AAPL where volume > 1000000"
+```
+
+Joins work transparently as long as both tables have been dumped:
+
+```bash
+uv run -m featureSQL.cli query --data_path source --store_type fs \
+    "select a.open, n.close from AAPL a join NVDA n on a.date = n.date"
+```
+
+(The cache flags are optional; omit them to use unlimited resources.)
+---
+
+### Notes & tips
+
+- **Symbols list:** `--symbols` accepts a comma string, Python list/tuple, or
+  `--symbols_file` path.  If you provide a file that exists but contains no
+  tickers the downloader will now do nothing (previous behaviour fell through
+  to downloading the entire US universe).
+- **Reloading tickers:** use `--reload_symbols` to refresh the cached symbol list.
+- **Alternate flows:** to skip the binning step, omit `--out_format` or set it
+  to `yahoo` (the default).
+
+#### Maintaining binary dataset integrity
+
+Because the dumper appends new days based on the existing calendar, you
+must take care when fetching overlapping or out‑of‑order date ranges:
+
+1. **Always use a single start date that is at or before any previously
+   downloaded data.**  For a continuous collection you can simply run:  
+   ```bash
+   uv run -m featureSQL.cli download \
+       --region US \
+       --start 2025-01-01 \
+       --end   $(date +%Y-%m-%d) \
+       --symbols TSLA \
+       --data_path source --out_format bin
+   ```
+   The collector will skip existing CSVs and the dumper will append only the
+   new days, keeping the existing bins and calendar consistent.
+
+2. **If you need to back‑fill a gap or change the start date earlier:**
+   delete the old bin files (or the entire `source/features` tree) and run
+   with `--out_format bin` again (or manually use
+   `uv run -m featureSQL.dump_bin dump_all`) so that `DumpDataAll` recomputes the
+   calendar from scratch.  The update mode never rewrites the date index,
+   so appending older data without rebuilding will cause the printed dates to
+   be incorrect.
+
+3. **Automate detection if desired.**
+   You can extend the downloader to inspect the last date in existing CSVs
+   and request only missing days, and/or modify the dumper to warn when the
+   incoming data’s maximum date does not exceed the current calendar end.
+
+Following these practices prevents “wrong” date offsets from appearing when
+you use `uv run -m featureSQL.cli view …` (or simply `featureSQL view …`), and ensures the binary dataset remains a
+faithful time series.
+
+Feel free to copy‑paste these examples into your own docs or scripts!
+
+---
+
+## Viewing the contents of a bin file
+
+A new CLI subcommand makes this easy without writing Python.  Once you have
+an initialised dataset you can inspect any field file with:
+
+```bash
+uv run -m featureSQL.cli view /path/to/output/dir/features/aapl/open.day.bin --store_type fs
+```
+
+By default the command prints the starting date index and the shape of the
+array, followed by each raw float value.  If the dataset contains a calendar
+file (``calendars/day.txt``) the subcommand will automatically find it and
+show the corresponding date for each value.  You can also supply an explicit
+calendar path:
+
+```bash
+uv run -m featureSQL.cli view path/to/bin/file --calendar_file path/to/calendars/day.txt --store_type fs
+```
+
+Internally the helper still uses `numpy.fromfile` so the Python snippet
+below remains available if you prefer to inspect the file manually.
+
+```python
+import numpy as np
+# if you have a helper for converting codes to filenames, import it here
+from featureSQL.dump_bin import code_to_fname
+
+# point to a particular field file (e.g. open.day.bin) for one symbol
+bin_path = "/path/to/output/dir/features/aapl/open.day.bin"
+arr = np.fromfile(bin_path, dtype="<f")
+
+# first value is the date offset, the rest are data values
+date_index = int(arr[0])
+values = arr[1:]
+print(date_index, values.shape)
+```
+
+Alternatively, many tools provide helpers once a dataset is loaded; you can
+query the field of a symbol and receive a NumPy array.  Use whatever API
+your application or library supplies – the underlying files remain the same.
+
+```pythonfeatureSQL
+# pseudo-code using a generic loader
+data = my_loader.load("/path/to/output/dir")
+print(data['AAPL']['open'])
+```
+
+If you just want to look at the calendar, it’s a text file under
+`calendars/day.txt` with one date per line.

featuresql-0.1.0/README.md

@@ -0,0 +1,258 @@
+This repository is now packaged as the **featureSQL** Python module.  You can
+install it in one of two ways:
+
+1. **From PyPI:**
+
+   ```bash
+   pip install featureSQL
+   ```
+
+   This makes the `featureSQL` console script available on your PATH and lets
+   you `import featureSQL` from any Python program.
+
+2. **From source (development mode):**
+
+   ```bash
+   cd /path/to/featureSQL
+   pip install -e .
+   ```
+
+   This installs the package in editable mode, so local edits are reflected
+   immediately without reinstalling.  Useful when working on the project.
+
+After installation you can still use the original CLI helpers directly by
+importing:
+
+```python
+from featureSQL.cli import Run
+```
+
+and invoking `Run().download(...)` or `featureSQL` at the shell.  The remainder
+of this document is a how‑to reference that you can include in your own
+projects; it covers the three common workflows you asked about.
+
+---
+
+## 1. Download a list of symbols (OCHLVF) to CSV
+
+```bash
+# 1. prepare a text file listing tickers, one per line
+cat > .testsymbols.txt <<'EOF'
+AAPL
+AMZN
+GOOG
+TSLA
+EOF
+
+# 2. run the collector, writing CSVs into a directory (they’ll land in
+#    the `feature-csv` subfolder of whatever you pass to --data_path)
+uv run -m featureSQL.cli download \
+    --region US \
+    --start 2026-01-01 \
+    --end   2026-02-28 \
+    --symbols_file ./.testsymbols.txt \
+    --data_path ./source \
+    --store_type fs   # Output to local fs (default). Change to 'gcs' and set data_path to bucket name for GCS.
+```
+
+> **GCS setup:** when using `--store_type gcs` you must configure two
+> environment variables before running any commands or tests:
+>
+> ```bash
+> export GCS_SC_JSON='{"type":"service_account", …}'    # credentials
+> export GCS_BUCKET_NAME='your-bucket-name'               # target bucket
+> ```
+>
+> The unit tests will skip network‑dependent scenarios if `GCS_BUCKET_NAME`
+> is unset, and they mock the client in other cases.  However, any manual
+> invocation against a real bucket requires both variables to be set.
+
+*Files created*: `/path/to/target/csv/dir/feature-csv/AAPL.csv`,
+`feature-csv/AMZN.csv`, … (i.e. `feature-csv` under the directory you
+passed with `--data_path`).  Each CSV contains the usual Open/Close/High/Low/
+Volume/AdjClose data plus `symbol`/`date` columns.
+
+---
+
+## 2. Convert a directory of CSVs into a binary dataset
+
+This is the “dump all” step: it reads **all** CSV files in `data_path` and builds the calendar, instruments list and feature bins in the standard binary layout.
+
+```bash
+uv run -m featureSQL.dump_bin dump_all \
+    --data_path ./source/feature-csv \
+    --dump_dir   ./source/ \
+    --exclude_fields symbol,date \
+    --store_type fs         # don’t try to treat metadata as floats.
+# (./source/ is /path/to/output/dir/)
+```
+
+After the command finishes you’ll have a structure like:
+
+```
+/path/to/output/dir/
+  calendars/day.txt          # trading dates
+  instruments/all.txt        # ticker start/end dates
+  features/<ticker>/<field>.day.bin …
+```
+
+This directory can be consumed by any tool that understands the same layout – it’s a full **initialised** dataset.
+
+> 💡 run `dump_all` only once per collection.  To add new data later, use `dump_update` (see next section).
+
+---
+
+## 3. Download symbols and produce the binary dataset in one go
+
+The built‑in CLI helper (exposed via the `featureSQL` script or as
+`python -m featureSQL.cli`) can drive both phases and even run ad‑hoc
+queries against a binary dataset:
+
+```bash
+uv run -m featureSQL.cli download \
+    --region US \
+    --start 2026-01-01 \
+    --end   2026-02-28 \
+    --symbols AMZN,GOOG,TSLA \
+    --data_path source  \
+    --out_format bin \
+    --store_type fs
+```
+
+This will
+
+1. fetch OCHLVF data from Yahoo and save raw CSVs under
+   `--{data_path}/feature-csv` (or `./source` if you didn’t pass
+   `--data_path`),
+2. run the binary dumper. On the **first** invocation the helper uses
+   `DumpDataAll` to initialise the dataset; subsequent runs use
+   `DumpDataUpdate` to append new days.  The target directory is the same as
+   `--data_path` and you can still provide `exclude_fields="symbol,date"`.
+
+The first run must be preceded by an empty or non‑existent output directory
+(and/or you can manually run `dump_all` as shown above); thereafter the
+same command will **append** new days to the existing bins.
+
+## 4. Query the binary dataset using SQL
+
+Once you have an initialised dataset you can issue SQL against it.  The
+``query`` subcommand lazily loads only the symbols mentioned in the query
+and keeps a small LRU cache in memory.
+
+```bash
+uv run -m featureSQL.cli query \
+    --data_path source \
+    --max_symbols 100 \
+    --max_memory 2000000000 \
+    --store_type fs \
+    "select date, open, close, high, low, volume, adjclose from AAPL where volume > 1000000"
+```
+
+Joins work transparently as long as both tables have been dumped:
+
+```bash
+uv run -m featureSQL.cli query --data_path source --store_type fs \
+    "select a.open, n.close from AAPL a join NVDA n on a.date = n.date"
+```
+
+(The cache flags are optional; omit them to use unlimited resources.)
+---
+
+### Notes & tips
+
+- **Symbols list:** `--symbols` accepts a comma string, Python list/tuple, or
+  `--symbols_file` path.  If you provide a file that exists but contains no
+  tickers the downloader will now do nothing (previous behaviour fell through
+  to downloading the entire US universe).
+- **Reloading tickers:** use `--reload_symbols` to refresh the cached symbol list.
+- **Alternate flows:** to skip the binning step, omit `--out_format` or set it
+  to `yahoo` (the default).
+
+#### Maintaining binary dataset integrity
+
+Because the dumper appends new days based on the existing calendar, you
+must take care when fetching overlapping or out‑of‑order date ranges:
+
+1. **Always use a single start date that is at or before any previously
+   downloaded data.**  For a continuous collection you can simply run:  
+   ```bash
+   uv run -m featureSQL.cli download \
+       --region US \
+       --start 2025-01-01 \
+       --end   $(date +%Y-%m-%d) \
+       --symbols TSLA \
+       --data_path source --out_format bin
+   ```
+   The collector will skip existing CSVs and the dumper will append only the
+   new days, keeping the existing bins and calendar consistent.
+
+2. **If you need to back‑fill a gap or change the start date earlier:**
+   delete the old bin files (or the entire `source/features` tree) and run
+   with `--out_format bin` again (or manually use
+   `uv run -m featureSQL.dump_bin dump_all`) so that `DumpDataAll` recomputes the
+   calendar from scratch.  The update mode never rewrites the date index,
+   so appending older data without rebuilding will cause the printed dates to
+   be incorrect.
+
+3. **Automate detection if desired.**
+   You can extend the downloader to inspect the last date in existing CSVs
+   and request only missing days, and/or modify the dumper to warn when the
+   incoming data’s maximum date does not exceed the current calendar end.
+
+Following these practices prevents “wrong” date offsets from appearing when
+you use `uv run -m featureSQL.cli view …` (or simply `featureSQL view …`), and ensures the binary dataset remains a
+faithful time series.
+
+Feel free to copy‑paste these examples into your own docs or scripts!
+
+---
+
+## Viewing the contents of a bin file
+
+A new CLI subcommand makes this easy without writing Python.  Once you have
+an initialised dataset you can inspect any field file with:
+
+```bash
+uv run -m featureSQL.cli view /path/to/output/dir/features/aapl/open.day.bin --store_type fs
+```
+
+By default the command prints the starting date index and the shape of the
+array, followed by each raw float value.  If the dataset contains a calendar
+file (``calendars/day.txt``) the subcommand will automatically find it and
+show the corresponding date for each value.  You can also supply an explicit
+calendar path:
+
+```bash
+uv run -m featureSQL.cli view path/to/bin/file --calendar_file path/to/calendars/day.txt --store_type fs
+```
+
+Internally the helper still uses `numpy.fromfile` so the Python snippet
+below remains available if you prefer to inspect the file manually.
+
+```python
+import numpy as np
+# if you have a helper for converting codes to filenames, import it here
+from featureSQL.dump_bin import code_to_fname
+
+# point to a particular field file (e.g. open.day.bin) for one symbol
+bin_path = "/path/to/output/dir/features/aapl/open.day.bin"
+arr = np.fromfile(bin_path, dtype="<f")
+
+# first value is the date offset, the rest are data values
+date_index = int(arr[0])
+values = arr[1:]
+print(date_index, values.shape)
+```
+
+Alternatively, many tools provide helpers once a dataset is loaded; you can
+query the field of a symbol and receive a NumPy array.  Use whatever API
+your application or library supplies – the underlying files remain the same.
+
+```pythonfeatureSQL
+# pseudo-code using a generic loader
+data = my_loader.load("/path/to/output/dir")
+print(data['AAPL']['open'])
+```
+
+If you just want to look at the calendar, it’s a text file under
+`calendars/day.txt` with one date per line.

featuresql-0.1.0/featureSQL/__init__.py

@@ -0,0 +1,30 @@
+"""Top-level package for featureSQL.
+
+This file exposes the public API and maintains the version.
+"""
+
+__version__ = "0.1.0"
+
+from .cli import Run  # expose for convenience
+from .dump_bin import DumpDataAll, DumpDataUpdate
+from .yahoo import (
+    get_calendar_list,
+    get_us_stock_symbols,
+    get_hs_stock_symbols,
+    YahooCollectorUS,
+    YahooNormalize,
+)
+from .utils import deco_retry
+
+__all__ = [
+    "Run",
+    "DumpDataAll",
+    "DumpDataUpdate",
+    "get_calendar_list",
+    "get_us_stock_symbols",
+    "get_hs_stock_symbols",
+    "YahooCollectorUS",
+    "YahooNormalize",
+    "deco_retry",
+    "__version__",
+]