cotmetrics 0.1.0__tar.gz

cotmetrics-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Matt Spinola
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cotmetrics-0.1.0/PKG-INFO

@@ -0,0 +1,75 @@
+Metadata-Version: 2.4
+Name: cotmetrics
+Version: 0.1.0
+Summary: Turn raw COT data into positioning metrics and trading signals (positioning index, concentration/clustering/position-size, reversal signals).
+Author-email: "Matt S." <cotanalyzer@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/mspinola/cotmetrics
+Project-URL: Repository, https://github.com/mspinola/cotmetrics
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=1.5
+Requires-Dist: numpy
+Requires-Dist: pyarrow>=10
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: pytz
+Requires-Dist: requests>=2.28
+Requires-Dist: xlrd>=2.0
+Requires-Dist: cotdata>=0.1.0
+Provides-Extra: options
+Requires-Dist: yfinance; extra == "options"
+Provides-Extra: scheduler
+Requires-Dist: schedule; extra == "scheduler"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: ruff==0.15.22; extra == "dev"
+Requires-Dist: tomli; python_version < "3.11" and extra == "dev"
+Dynamic: license-file
+
+# cotmetrics
+
+Turn raw COT (Commitments of Traders) data into positioning metrics and trading
+signals — the positioning index, concentration / clustering / position-size,
+reversal signals, and the `CotIndexer` that assembles per-instrument weekly panels.
+
+Split out of `cot-analyzer` so the metrics layer installs without the Dash/Plotly
+dashboard stack. Reads prices/COT from the shared [`cotdata`](../cotdata) store.
+
+## Install (workspace, editable)
+
+```bash
+pip install -e ../cotdata -e .[options,scheduler,dev]
+export COTDATA_STORE=~/code/cotdata_store     # shared data store
+export COTMETRICS_CACHE=~/.cache/cotmetrics    # derived per-instrument parquet cache
+```
+
+## Use
+
+```python
+import cotmetrics                                   # flat metric fns
+from cotmetrics.indexer import cotIndexer, boot_options_update
+from cotmetrics.signals import append_trading_signals
+cotmetrics.calculate_cot_index(...)
+```
+
+`import cotmetrics` is side-effect-free. Constructing the indexer
+(`from cotmetrics.indexer import cotIndexer`) loads the store; the daily options
+fetch runs only when you call `boot_options_update()` explicitly.
+
+## Config / paths
+
+| env | meaning | default |
+|-----|---------|---------|
+| `COTDATA_STORE` | shared price/COT store (from cotdata) | required |
+| `COTMETRICS_CACHE` | derived per-instrument parquet cache | `~/.cache/cotmetrics` |
+| `COTMETRICS_PARAMS` | instrument/params config | packaged **sample** `params.yaml` |
+| `COTMETRICS_DATA` | legacy raw_cot_data.parquet + real_test_data exports | `<cache>/../cotmetrics_data` |
+
+The packaged `params.yaml` is a small **generic sample** (a handful of well-known
+symbols, untuned 52-week lookbacks) so cotmetrics runs out of the box. For a real
+instrument universe and any tuned parameters, set `COTMETRICS_PARAMS` to your own
+config file.

cotmetrics-0.1.0/README.md

@@ -0,0 +1,43 @@
+# cotmetrics
+
+Turn raw COT (Commitments of Traders) data into positioning metrics and trading
+signals — the positioning index, concentration / clustering / position-size,
+reversal signals, and the `CotIndexer` that assembles per-instrument weekly panels.
+
+Split out of `cot-analyzer` so the metrics layer installs without the Dash/Plotly
+dashboard stack. Reads prices/COT from the shared [`cotdata`](../cotdata) store.
+
+## Install (workspace, editable)
+
+```bash
+pip install -e ../cotdata -e .[options,scheduler,dev]
+export COTDATA_STORE=~/code/cotdata_store     # shared data store
+export COTMETRICS_CACHE=~/.cache/cotmetrics    # derived per-instrument parquet cache
+```
+
+## Use
+
+```python
+import cotmetrics                                   # flat metric fns
+from cotmetrics.indexer import cotIndexer, boot_options_update
+from cotmetrics.signals import append_trading_signals
+cotmetrics.calculate_cot_index(...)
+```
+
+`import cotmetrics` is side-effect-free. Constructing the indexer
+(`from cotmetrics.indexer import cotIndexer`) loads the store; the daily options
+fetch runs only when you call `boot_options_update()` explicitly.
+
+## Config / paths
+
+| env | meaning | default |
+|-----|---------|---------|
+| `COTDATA_STORE` | shared price/COT store (from cotdata) | required |
+| `COTMETRICS_CACHE` | derived per-instrument parquet cache | `~/.cache/cotmetrics` |
+| `COTMETRICS_PARAMS` | instrument/params config | packaged **sample** `params.yaml` |
+| `COTMETRICS_DATA` | legacy raw_cot_data.parquet + real_test_data exports | `<cache>/../cotmetrics_data` |
+
+The packaged `params.yaml` is a small **generic sample** (a handful of well-known
+symbols, untuned 52-week lookbacks) so cotmetrics runs out of the box. For a real
+instrument universe and any tuned parameters, set `COTMETRICS_PARAMS` to your own
+config file.

cotmetrics-0.1.0/pyproject.toml

@@ -0,0 +1,52 @@
+[build-system]
+requires = ["setuptools>=64", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "cotmetrics"
+version = "0.1.0"
+description = "Turn raw COT data into positioning metrics and trading signals (positioning index, concentration/clustering/position-size, reversal signals)."
+readme = "README.md"
+license = { text = "MIT" }
+authors = [
+    {name = "Matt S.", email = "cotanalyzer@gmail.com"}
+]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+requires-python = ">=3.9"
+dependencies = [
+    "pandas>=1.5",
+    "numpy",
+    "pyarrow>=10",
+    "PyYAML>=6.0",
+    "pytz",
+    "requests>=2.28",       # legacy CFTC ETL (cotmetrics.etl)
+    "xlrd>=2.0",            # legacy CFTC dea_fut_xls are .xls
+    "cotdata>=0.1.0",       # shared data layer; editable in the workspace: -e ../cotdata
+]
+
+[project.optional-dependencies]
+options = ["yfinance"]      # cotmetrics.options_data (max-pain snapshots)
+scheduler = ["schedule"]    # cotmetrics.pipelines.etl_scheduler
+dev = ["pytest>=7", "ruff==0.15.22", 'tomli; python_version < "3.11"']  # tomli for scripts/check_dep_floors.py on <3.11
+
+[project.urls]
+Homepage = "https://github.com/mspinola/cotmetrics"
+Repository = "https://github.com/mspinola/cotmetrics"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+cotmetrics = ["*.yaml"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py39"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I"]
+ignore = ["E501"]

cotmetrics-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

cotmetrics-0.1.0/src/cotmetrics/CotDatabase.py

@@ -0,0 +1,220 @@
+import os
+import sqlite3
+from datetime import datetime, timezone
+from zoneinfo import ZoneInfo
+
+import pandas as pd
+
+import cotmetrics.utils as utils
+
+
+class CotDatabase:
+    """Class to manage COT data"""
+    def __init__(self, db_name=None):
+        if db_name is None:
+            # Resolve absolutely to cot-analyzer/data/cot_data.db
+            base_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
+            self.db_name = os.path.join(base_dir, 'data', 'cot_data.db')
+        else:
+            self.db_name = db_name
+
+        # Ensure directories exist
+        os.makedirs(os.path.dirname(self.db_name), exist_ok=True)
+
+        self.setup_database()
+
+    def setup_database(self):
+        """Create the database and the necessary table if it doesn't exist."""
+        conn = sqlite3.connect(self.db_name)
+        c = conn.cursor()
+        c.execute('''
+            CREATE TABLE IF NOT EXISTS zip_files (
+                year INTEGER PRIMARY KEY,
+                last_modified TEXT
+            )
+        ''')
+
+        # Add visitor logs table
+        c.execute('''
+            CREATE TABLE IF NOT EXISTS visitor_logs (
+                id INTEGER PRIMARY KEY AUTOINCREMENT,
+                timestamp TEXT,
+                ip_address TEXT,
+                path TEXT,
+                user_agent TEXT,
+                city TEXT,
+                country TEXT
+            )
+        ''')
+
+        # Add ml_predictions_v2 table
+        c.execute('''
+            CREATE TABLE IF NOT EXISTS ml_predictions_v2 (
+                symbol TEXT,
+                report_date TEXT,
+                prob_success REAL,
+                meta_side INTEGER,
+                expectancy REAL,
+                atr_mult_tp REAL,
+                atr_mult_sl REAL,
+                updated_at TEXT,
+                PRIMARY KEY (symbol, report_date)
+            )
+        ''')
+
+        # Safely add new columns to existing table
+        for col, col_type in [('expectancy', 'REAL'), ('atr_mult_tp', 'REAL'), ('atr_mult_sl', 'REAL')]:
+            try:
+                c.execute(f'ALTER TABLE ml_predictions_v2 ADD COLUMN {col} {col_type}')
+            except sqlite3.OperationalError:
+                pass
+        conn.commit()
+        conn.close()
+
+    def log_visit(self, ip, path, ua, city="Unknown", country="Unknown"):
+        """Records a new visitor event to the database."""
+        conn = sqlite3.connect(self.db_name)
+        c = conn.cursor()
+        now = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
+        c.execute('''
+            INSERT INTO visitor_logs (timestamp, ip_address, path, user_agent, city, country)
+            VALUES (?, ?, ?, ?, ?, ?)
+        ''', (now, ip, path, ua, city, country))
+        conn.commit()
+        conn.close()
+
+    def get_visitor_stats(self):
+        """Retrieves recent logs for the admin dashboard."""
+        conn = sqlite3.connect(self.db_name)
+        df = pd.read_sql_query("SELECT * FROM visitor_logs ORDER BY id DESC LIMIT 500", conn)
+        conn.close()
+        return df
+
+    def update_zip_file(self, year, last_modified):
+        """Update the last modified date of the zip file in the database."""
+        conn = sqlite3.connect(self.db_name)
+        c = conn.cursor()
+        c.execute('''
+            INSERT INTO zip_files (year, last_modified) VALUES (?, ?)
+            ON CONFLICT(year) DO UPDATE SET last_modified = ?
+        ''', (year, last_modified, last_modified))
+        conn.commit()
+        conn.close()
+        utils.get_cot_logger().warning(f"Updated latest zipfile time in DB {year} {last_modified}")
+
+    def get_zipfile_last_modified_time(self, year):
+        conn = sqlite3.connect(self.db_name)
+        c = conn.cursor()
+        c.execute('SELECT last_modified FROM zip_files WHERE year = ?', (year,))
+        row = c.fetchone()
+        conn.close()
+
+        result = None
+        if row:
+            try:
+                result = datetime.strptime(row[0], '%a, %d %b %Y %H:%M:%S %Z')
+            except Exception as e:
+                utils.get_cot_logger().debug(f"First attempt exception in date format {result}. {e}")
+                try:
+                    utils.get_cot_logger().debug(f"2nd attempt on {row[0]}")
+                    result = datetime.strptime(row[0], '%Y-%m-%d %H:%M:%S')
+                    utils.get_cot_logger().debug(f"2nd time worked {result}")
+                except Exception as e:
+                    utils.get_cot_logger().error(f"2nd time exception in date format {result}. {e}")
+                    return None
+        return result
+
+    def latest_update_timestamp(self):
+        tz_aware = "Unknown"
+        year = str(datetime.now().year)
+        result = self.get_zipfile_last_modified_time(year)
+        if result is not None:
+            tz_aware = result.replace(tzinfo=timezone.utc)
+            tz_aware = tz_aware.astimezone(ZoneInfo("America/New_York"))
+            tz_aware = tz_aware.strftime("%Y-%m-%d %H:%M:%S %Z")
+            if tz_aware is None:
+                tz_aware = "Unknown"
+        return tz_aware
+
+    def save_predictions(self, symbol, report_date, prob_success, meta_side, expectancy=None, atr_mult_tp=None, atr_mult_sl=None):
+        conn = sqlite3.connect(self.db_name)
+        c = conn.cursor()
+        now = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
+        c.execute('''
+            INSERT INTO ml_predictions_v2 (
+                symbol, report_date, prob_success, meta_side, expectancy, atr_mult_tp, atr_mult_sl, updated_at
+            )
+            VALUES (?, ?, ?, ?, ?, ?, ?, ?)
+            ON CONFLICT(symbol, report_date) DO UPDATE SET
+                prob_success = excluded.prob_success,
+                meta_side = excluded.meta_side,
+                expectancy = excluded.expectancy,
+                atr_mult_tp = excluded.atr_mult_tp,
+                atr_mult_sl = excluded.atr_mult_sl,
+                updated_at = excluded.updated_at
+        ''', (symbol, report_date, prob_success, meta_side, expectancy, atr_mult_tp, atr_mult_sl, now))
+        conn.commit()
+        conn.close()
+
+    def get_latest_prediction(self, symbol):
+        conn = sqlite3.connect(self.db_name)
+        c = conn.cursor()
+        c.execute('''
+            SELECT report_date, prob_success, meta_side, updated_at
+            FROM ml_predictions_v2
+            WHERE symbol = ?
+            ORDER BY report_date DESC LIMIT 1
+        ''', (symbol,))
+        row = c.fetchone()
+        conn.close()
+        if row:
+            return {
+                "report_date": row[0],
+                "prob_success": row[1],
+                "meta_side": row[2],
+                "updated_at": row[3]
+            }
+        return None
+
+    def get_prediction(self, symbol, report_date):
+        conn = sqlite3.connect(self.db_name)
+        c = conn.cursor()
+        c.execute('''
+            SELECT report_date, prob_success, meta_side, updated_at
+            FROM ml_predictions_v2
+            WHERE symbol = ? AND report_date = ?
+        ''', (symbol, report_date))
+        row = c.fetchone()
+        conn.close()
+        if row:
+            return {
+                "report_date": row[0],
+                "prob_success": row[1],
+                "meta_side": row[2],
+                "updated_at": row[3]
+            }
+        return None
+
+    def get_all_predictions(self, symbol):
+        conn = sqlite3.connect(self.db_name)
+        c = conn.cursor()
+        c.execute('''
+            SELECT report_date, prob_success, meta_side, updated_at
+            FROM ml_predictions_v2
+            WHERE symbol = ?
+            ORDER BY report_date ASC
+        ''', (symbol,))
+        rows = c.fetchall()
+        conn.close()
+
+        predictions = []
+        for row in rows:
+            predictions.append({
+                "report_date": row[0],
+                "prob_success": row[1],
+                "meta_side": row[2],
+                "updated_at": row[3]
+            })
+        return predictions
+
+