mt5cli 0.5.1__tar.gz → 0.5.3__tar.gz

{mt5cli-0.5.1 → mt5cli-0.5.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mt5cli
-Version: 0.5.1
+Version: 0.5.3
 Summary: Command-line tool for MetaTrader 5
 Project-URL: Repository, https://github.com/dceoy/mt5cli.git
 Author-email: dceoy <dceoy@users.noreply.github.com>
@@ -160,7 +160,10 @@ update_history_with_config(
 - **Rate view resolution**: use `resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases. Both accept `None` (or a missing path) and return deterministic default names unless `require_existing=True`.
 - **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
 - **Multi-series rate loading**: use `build_rate_targets()` to build neutral `RateTarget(symbol, timeframe)` pairs, `resolve_rate_tables()` to map them to table/view names (pass `require_existing=True` for strict resolution), and `load_rate_series_from_sqlite()` to load them into a mapping keyed by `(symbol, integer timeframe)`. The loader requires existing managed views unless `explicit_tables` is supplied, and rejects duplicate `(symbol, timeframe)` targets.
-- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping.
+- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only `pdmt5.Mt5TradingError` / `pdmt5.Mt5RuntimeError` and re-raises once `retry_count` is exhausted.
+- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code.
+- **Throttled history updates**: use `ThrottledHistoryUpdater` to wrap `update_history()` with a minimum `interval_seconds` between successful runs (monotonic clock). Call `should_update()` / `update(client, symbols)` from an application loop; errors propagate by default, or pass `suppress_errors=True` to swallow recoverable `Mt5*Error`/`sqlite3.Error` and let the caller decide logging.
+- **Granularity-keyed rate loading**: `load_rate_series_by_granularity()` builds targets with `build_rate_targets()`, loads them with `load_rate_series_from_sqlite()`, and returns a mapping keyed by `(symbol | None, granularity_name)` such as `("EURUSD", "M1")` to reduce downstream boilerplate.
 - **MT5 session helper**: use the `mt5_session()` context manager to attach to (or, when `Mt5Config.path` is set, launch) an MT5 terminal, log in, and yield a connected `Mt5CliClient` that shuts down on exit.
 - **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
 - **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.

{mt5cli-0.5.1 → mt5cli-0.5.3}/README.md

@@ -136,7 +136,10 @@ update_history_with_config(
 - **Rate view resolution**: use `resolve_rate_view_name()` / `resolve_rate_view_names()` to map symbols and granularities to existing SQLite compatibility views without creating databases. Both accept `None` (or a missing path) and return deterministic default names unless `require_existing=True`.
 - **Rate view loading**: use `load_rate_data()` / `load_rate_data_from_connection()` to load a SQLite rate table or view into a `DatetimeIndex` DataFrame.
 - **Multi-series rate loading**: use `build_rate_targets()` to build neutral `RateTarget(symbol, timeframe)` pairs, `resolve_rate_tables()` to map them to table/view names (pass `require_existing=True` for strict resolution), and `load_rate_series_from_sqlite()` to load them into a mapping keyed by `(symbol, integer timeframe)`. The loader requires existing managed views unless `explicit_tables` is supplied, and rejects duplicate `(symbol, timeframe)` targets.
-- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping.
+- **Multi-account latest rates**: use `collect_latest_rates_for_accounts()` with `AccountSpec` to read the latest bars for several account groups, merged into a `(symbol, integer timeframe)` mapping. For long-running pollers, `collect_latest_rates_for_accounts_with_retries()` adds bounded exponential backoff that retries only `pdmt5.Mt5TradingError` / `pdmt5.Mt5RuntimeError` and re-raises once `retry_count` is exhausted.
+- **Credential resolution**: use `resolve_account_spec()` / `resolve_account_specs()` to merge explicit override values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders (via `substitute_env_placeholders()`), raising `ValueError` for missing variables. This keeps secrets out of plan/config files without coupling to any strategy code.
+- **Throttled history updates**: use `ThrottledHistoryUpdater` to wrap `update_history()` with a minimum `interval_seconds` between successful runs (monotonic clock). Call `should_update()` / `update(client, symbols)` from an application loop; errors propagate by default, or pass `suppress_errors=True` to swallow recoverable `Mt5*Error`/`sqlite3.Error` and let the caller decide logging.
+- **Granularity-keyed rate loading**: `load_rate_series_by_granularity()` builds targets with `build_rate_targets()`, loads them with `load_rate_series_from_sqlite()`, and returns a mapping keyed by `(symbol | None, granularity_name)` such as `("EURUSD", "M1")` to reduce downstream boilerplate.
 - **MT5 session helper**: use the `mt5_session()` context manager to attach to (or, when `Mt5Config.path` is set, launch) an MT5 terminal, log in, and yield a connected `Mt5CliClient` that shuts down on exit.
 - **SQLite export helpers**: use `export_dataframe_to_sqlite()` for append mode, optional index export, and post-write deduplication by key columns.
 - **Recent ticks and margins**: `recent_ticks()` and `minimum_margins()` SDK helpers (and matching CLI commands) cover common downstream read-only queries.

{mt5cli-0.5.1 → mt5cli-0.5.3}/docs/api/history.md

@@ -133,8 +133,8 @@ The `update_history` SDK path uses the same base tables and optional
 ### Rate view resolution
 
 Downstream tools can resolve mt5cli-managed compatibility view names from an
-existing SQLite history database without creating files or guessing legacy
-naming schemes:
+existing SQLite history database without creating files or guessing naming
+schemes:
 
 ```python
 from pathlib import Path
@@ -215,3 +215,15 @@ frame = series["EURUSD", 1]  # keyed by (symbol, integer timeframe)
   requires existing managed `rate_*` compatibility views and raises
   `ValueError` when they are missing. Duplicate `(symbol, timeframe)` targets
   are rejected.
+- `load_rate_series_by_granularity()` is a thin wrapper that builds the targets,
+  loads the series, and rekeys the result by granularity name to avoid
+  converting integer timeframes downstream:
+
+  ```python
+  from mt5cli import load_rate_series_by_granularity
+
+  series = load_rate_series_by_granularity(
+      "history.db", ["EURUSD"], ["M1", "H1"], count=1000
+  )
+  frame = series["EURUSD", "M1"]  # keyed by (symbol | None, granularity_name)
+  ```

mt5cli-0.5.3/docs/api/sdk.md

@@ -0,0 +1,83 @@
+# SDK Module
+
+::: mt5cli.sdk
+
+## Resilient multi-account orchestration
+
+The SDK ships strategy-agnostic helpers for building long-running collectors on
+top of the read-only client. None of them depend on a particular trading
+application.
+
+### Retrying transient rate collection
+
+`collect_latest_rates_for_accounts_with_retries()` wraps
+`collect_latest_rates_for_accounts()` with bounded exponential backoff. Only
+`pdmt5.Mt5TradingError` and `pdmt5.Mt5RuntimeError` are retried; the final
+failure is re-raised once `retry_count` is exhausted.
+
+```python
+from mt5cli import AccountSpec, collect_latest_rates_for_accounts_with_retries
+
+accounts = [AccountSpec(symbols=["EURUSD"], login=12345)]
+rates = collect_latest_rates_for_accounts_with_retries(
+    accounts,
+    ["M1", "H1"],
+    count=500,
+    retry_count=3,
+    backoff_base=2,  # sleeps 2s, 4s, 8s between attempts
+)
+```
+
+### Resolving credentials and `${ENV_VAR}` placeholders
+
+`resolve_account_spec()` / `resolve_account_specs()` merge explicit override
+values over `AccountSpec` fields and expand `${ENV_VAR}` placeholders, keeping
+secrets out of plan/config files. A missing environment variable raises
+`ValueError`.
+
+```python
+import os
+
+from mt5cli import AccountSpec, resolve_account_specs
+
+os.environ["MT5_LOGIN"] = "12345"
+os.environ["MT5_PASSWORD"] = "secret"
+accounts = [
+    AccountSpec(symbols=["EURUSD"], login="${MT5_LOGIN}", password="${MT5_PASSWORD}")
+]
+
+resolved = resolve_account_specs(accounts, server="Broker-Demo")
+# resolved[0].login == "12345", resolved[0].server == "Broker-Demo"
+```
+
+### Throttled incremental history updates
+
+`ThrottledHistoryUpdater` wraps `update_history()` with a minimum interval
+between successful runs (using a monotonic clock), so an application loop can
+call it every iteration without over-fetching.
+
+```python
+from pdmt5 import Mt5Config, Mt5DataClient
+
+from mt5cli import Dataset, ThrottledHistoryUpdater
+
+updater = ThrottledHistoryUpdater(
+    output="history.db",
+    datasets={Dataset.rates},
+    timeframes=["M1"],
+    interval_seconds=60,  # <= 0 updates on every call
+)
+
+client = Mt5DataClient(config=Mt5Config(login=12345))
+client.initialize_and_login_mt5()
+try:
+    while True:
+        updater.update(client, ["EURUSD", "GBPUSD"])  # no-op until 60s elapse
+        # ... do other work; break when shutting down ...
+finally:
+    client.shutdown()
+```
+
+By default `Mt5TradingError`, `Mt5RuntimeError`, and `sqlite3.Error` propagate so
+the caller controls logging; pass `suppress_errors=True` to swallow them and
+return `False` without advancing the throttle.

{mt5cli-0.5.1 → mt5cli-0.5.3}/mt5cli/__init__.py

@@ -8,6 +8,7 @@ from .history import (
     build_rate_view_name,
     load_rate_data,
     load_rate_data_from_connection,
+    load_rate_series_by_granularity,
     load_rate_series_from_sqlite,
     resolve_history_datasets,
     resolve_history_tick_flags,
@@ -19,11 +20,13 @@ from .history import (
 from .sdk import (
     AccountSpec,
     Mt5CliClient,
+    ThrottledHistoryUpdater,
     account_info,
     build_config,
     collect_history,
     collect_latest_rates,
     collect_latest_rates_for_accounts,
+    collect_latest_rates_for_accounts_with_retries,
     copy_rates_from,
     copy_rates_from_pos,
     copy_rates_range,
@@ -42,6 +45,9 @@ from .sdk import (
     positions,
     recent_history_deals,
     recent_ticks,
+    resolve_account_spec,
+    resolve_account_specs,
+    substitute_env_placeholders,
     symbol_info,
     symbol_info_tick,
     symbols,
@@ -75,6 +81,7 @@ __all__ = [
     "IfExists",
     "Mt5CliClient",
     "RateTarget",
+    "ThrottledHistoryUpdater",
     "account_info",
     "build_config",
     "build_rate_targets",
@@ -82,6 +89,7 @@ __all__ = [
     "collect_history",
     "collect_latest_rates",
     "collect_latest_rates_for_accounts",
+    "collect_latest_rates_for_accounts_with_retries",
     "copy_rates_from",
     "copy_rates_from_pos",
     "copy_rates_range",
@@ -96,6 +104,7 @@ __all__ = [
     "latest_rates",
     "load_rate_data",
     "load_rate_data_from_connection",
+    "load_rate_series_by_granularity",
     "load_rate_series_from_sqlite",
     "market_book",
     "minimum_margins",
@@ -110,12 +119,15 @@ __all__ = [
     "positions",
     "recent_history_deals",
     "recent_ticks",
+    "resolve_account_spec",
+    "resolve_account_specs",
     "resolve_history_datasets",
     "resolve_history_tick_flags",
     "resolve_history_timeframes",
     "resolve_rate_tables",
     "resolve_rate_view_name",
     "resolve_rate_view_names",
+    "substitute_env_placeholders",
     "symbol_info",
     "symbol_info_tick",
     "symbols",

{mt5cli-0.5.1 → mt5cli-0.5.3}/mt5cli/history.py

@@ -21,7 +21,7 @@ from .utils import (
 )
 
 if TYPE_CHECKING:
-    from collections.abc import Callable, Sequence
+    from collections.abc import Callable, Mapping, Sequence
 
     from pdmt5 import Mt5DataClient
 
@@ -706,6 +706,55 @@ def load_rate_series_from_sqlite(
             conn.close()
 
 
+def load_rate_series_by_granularity(
+    conn_or_path: SqliteConnOrPath,
+    symbols: Sequence[str],
+    granularities: Sequence[int | str],
+    count: int,
+    *,
+    explicit_tables: Sequence[str] | None = None,
+    allow_missing_symbol: bool = False,
+) -> dict[tuple[str | None, str], pd.DataFrame]:
+    """Load rate series keyed by symbol and string granularity name.
+
+    Builds targets with :func:`build_rate_targets` and loads them with
+    :func:`load_rate_series_from_sqlite`, then rekeys the result by granularity
+    name (for example ``M1``) instead of the integer timeframe to reduce
+    downstream boilerplate.
+
+    Args:
+        conn_or_path: SQLite database path or open connection.
+        symbols: MT5 symbol names. May be empty when ``allow_missing_symbol``.
+        granularities: MT5 timeframes as integers or names (for example ``M1``).
+        count: Number of most recent rows to load per series.
+        explicit_tables: Optional explicit table or view names matching the
+            built targets in row-major order. Required when symbols are omitted.
+        allow_missing_symbol: When True and ``symbols`` is empty, build targets
+            with ``symbol=None`` for each granularity instead of raising.
+
+    Returns:
+        Mapping keyed by ``(symbol | None, granularity_name)`` to each rate
+        DataFrame. Propagates ``ValueError`` (via :func:`build_rate_targets` and
+        :func:`load_rate_series_from_sqlite`) when inputs are empty or invalid,
+        table resolution fails, or duplicate targets are present.
+    """
+    targets = build_rate_targets(
+        symbols,
+        granularities,
+        allow_missing_symbol=allow_missing_symbol,
+    )
+    series = load_rate_series_from_sqlite(
+        conn_or_path,
+        targets,
+        count,
+        explicit_tables=explicit_tables,
+    )
+    return {
+        (symbol, resolve_granularity_name(timeframe)): frame
+        for (symbol, timeframe), frame in series.items()
+    }
+
+
 def get_table_columns(conn: sqlite3.Connection, table: str) -> set[str]:
     """Return existing SQLite columns for a table."""
     quoted_table = quote_sqlite_identifier(table)
@@ -989,7 +1038,20 @@ def drop_duplicates_in_table(
     )
 
 
-DedupScope = tuple[str, tuple[object, ...]]
+@dataclass(frozen=True)
+class DedupScope:
+    """Scoped deduplication predicate and the columns it references.
+
+    Attributes:
+        where: SQL predicate appended to the duplicate-removal query.
+        params: Parameters bound to the scope predicate.
+        required_columns: Columns that must be present in the written table for
+            the scope to run.
+    """
+
+    where: str
+    params: tuple[object, ...]
+    required_columns: frozenset[str]
 
 
 def _record_dedup_scope(
@@ -997,17 +1059,25 @@ def _record_dedup_scope(
     dataset: Dataset,
     scope_where: str,
     scope_params: tuple[object, ...],
+    required_columns: frozenset[str],
 ) -> None:
-    dedup_scopes.setdefault(dataset, []).append((scope_where, scope_params))
+    dedup_scopes.setdefault(dataset, []).append(
+        DedupScope(scope_where, scope_params, required_columns),
+    )
 
 
 def deduplicate_history_tables(
     conn: sqlite3.Connection,
     written_columns: dict[Dataset, set[str]],
     written_tables: set[Dataset],
-    dedup_scopes: dict[Dataset, list[DedupScope]] | None = None,
+    dedup_scopes: Mapping[Dataset, Sequence[DedupScope]] | None = None,
 ) -> None:
-    """Deduplicate appended history tables by stable identifiers."""
+    """Deduplicate appended history tables by stable identifiers.
+
+    Scopes whose required columns are not present in the written table are
+    skipped. If all scopes for a dataset are skipped, the table receives one
+    unscoped deduplication pass instead.
+    """
     cursor = conn.cursor()
     for dataset in written_tables:
         columns = written_columns.get(dataset, set())
@@ -1026,16 +1096,19 @@ def deduplicate_history_tables(
                 table,
             )
             continue
-        scopes = dedup_scopes.get(dataset, []) if dedup_scopes else []
+        raw_scopes: Sequence[DedupScope] = (
+            dedup_scopes.get(dataset, ()) if dedup_scopes else ()
+        )
+        scopes = [scope for scope in raw_scopes if scope.required_columns <= columns]
         if scopes:
-            for scope_where, scope_params in scopes:
+            for scope in scopes:
                 drop_duplicates_in_table(
                     cursor,
                     table,
                     list(keys),
                     keep="last",
-                    scope_where=scope_where,
-                    scope_params=scope_params,
+                    scope_where=scope.where,
+                    scope_params=scope.params,
                 )
             continue
         drop_duplicates_in_table(cursor, table, list(keys), keep="last")
@@ -1402,6 +1475,7 @@ def _write_incremental_rates(
                     Dataset.rates,
                     "symbol = ? AND timeframe = ? AND time >= ?",
                     (symbol, timeframe, start_date),
+                    frozenset({"symbol", "timeframe", "time"}),
                 )
 
 
@@ -1440,6 +1514,7 @@ def _write_incremental_ticks(
                 Dataset.ticks,
                 "symbol = ? AND time >= ?",
                 (symbol, start_date),
+                frozenset({"symbol", "time"}),
             )
 
 
@@ -1478,6 +1553,7 @@ def _write_incremental_history_orders(
                 Dataset.history_orders,
                 "symbol = ? AND time >= ?",
                 (symbol, start_date),
+                frozenset({"symbol", "time"}),
             )
 
 
@@ -1531,6 +1607,7 @@ def _write_incremental_history_deals(
                         Dataset.history_deals,
                         "symbol = ? AND time >= ?",
                         (symbol, start_by_symbol[symbol, None]),
+                        frozenset({"symbol", "time"}),
                     )
             if "type" in columns:
                 _record_dedup_scope(
@@ -1538,6 +1615,7 @@ def _write_incremental_history_deals(
                     Dataset.history_deals,
                     f"type NOT IN {_TRADE_DEAL_TYPES_SQL} AND time >= ?",
                     (account_event_start,),
+                    frozenset({"type", "time"}),
                 )
             if "type" not in columns and "symbol" in columns:
                 _record_dedup_scope(
@@ -1545,6 +1623,7 @@ def _write_incremental_history_deals(
                     Dataset.history_deals,
                     "(symbol IS NULL OR symbol = '') AND time >= ?",
                     (account_event_start,),
+                    frozenset({"symbol", "time"}),
                 )
         return
     start_by_symbol = load_incremental_start_datetimes(
@@ -1572,6 +1651,7 @@ def _write_incremental_history_deals(
                 Dataset.history_deals,
                 "symbol = ? AND time >= ?",
                 (symbol, start_date),
+                frozenset({"symbol", "time"}),
             )