mt5cli 1.2.3__tar.gz → 1.3.0__tar.gz

{mt5cli-1.2.3 → mt5cli-1.3.0}/.github/workflows/ci.yml

@@ -59,10 +59,10 @@ jobs:
       contents: read
     runs-on: windows-latest
     steps:
-      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10  # v6.0.2
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v6.0.2
         with:
           persist-credentials: false
-      - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78  # v7.6.0
+      - uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990  # v8.3.2
       - name: Check CLI help with minimum Click and Typer versions
         run: uv run --with click==8.4.0 --with typer==0.26.0 mt5cli --help
   python-docs-deploy:

{mt5cli-1.2.3 → mt5cli-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mt5cli
-Version: 1.2.3
+Version: 1.3.0
 Summary: Generic MT5 data and execution infrastructure for Python applications
 Project-URL: Repository, https://github.com/dceoy/mt5cli.git
 Author-email: dceoy <dceoy@users.noreply.github.com>
@@ -16,7 +16,6 @@ Classifier: Programming Language :: Python
 Classifier: Programming Language :: Python :: 3
 Classifier: Topic :: Office/Business :: Financial :: Investment
 Requires-Python: <3.14,>=3.11
-Requires-Dist: click>=8.4
 Requires-Dist: pandas>=2.2.2
 Requires-Dist: pdmt5>=1.2.0
 Requires-Dist: typer>=0.26.0

{mt5cli-1.2.3 → mt5cli-1.3.0}/docs/api/public-contract.md

@@ -102,7 +102,10 @@ strategy entries, exits, Kelly sizing, or signal logic.
 | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
 | `get_account_snapshot`, `get_symbol_snapshot`, `get_tick_snapshot`, `get_positions_frame`                                      | Normalized account/symbol/tick/position views                     |
 | `extract_tick_price`                                                                                                           | Positive finite bid/ask extraction from tick mappings             |
-| `estimate_server_clock_offset_seconds`                                                                                         | Broker server clock offset from true UTC, via the latest tick     |
+| `TickClockNormalizer`                                                                                                          | Connection-scoped, copied-tick-validated UTC tick normalization   |
+| `TickClockCalibration`                                                                                                         | Server-clock calibration diagnostics (offset, evidence, status)   |
+| `NormalizedTickSnapshot`                                                                                                       | Typed latest-tick contract with raw vs validated UTC timestamps   |
+| `CalibrationStatus`, `ClockStatus`                                                                                             | Literal status types for clock calibration and normalization      |
 | `detect_position_side`                                                                                                         | Net long / short / flat from open positions                       |
 | `calculate_spread_ratio`                                                                                                       | Relative bid-ask spread                                           |
 | `calculate_margin_and_volume`, `calculate_volume_by_margin`, `calculate_new_position_margin_ratio`                             | Margin budget and volume sizing                                   |

{mt5cli-1.2.3 → mt5cli-1.3.0}/docs/api/trading.md

@@ -215,14 +215,105 @@ Any broker-specific offset handling therefore requires evidence from the live
 data source in use and must be applied separately. Do not subtract a presumed
 UTC+2 or UTC+3 offset from timestamps that are already UTC.
 
-`estimate_server_clock_offset_seconds()` reads the latest tick for a symbol
-and returns the broker's clock offset from true UTC, rounded to the nearest
-half hour, or `None` when no valid tick time is available. Use it to validate a
-suspected offset for the live `symbol_info_tick()` source before applying the
-result in downstream time-window calculations.
-
 No offset is applied automatically by history retrieval helpers.
 
+## UTC normalization for live ticks
+
+`TickClockNormalizer` provides the stable API for obtaining a latest tick with
+a **validated** UTC timestamp, so strategy applications never compare a raw
+server-labeled tick time against `datetime.now(UTC)` themselves.
+
+```python
+from mt5cli import TickClockNormalizer, mt5_session
+
+with mt5_session() as client:
+    normalizer = TickClockNormalizer(client, ["SING30", "EURUSD"])
+    snapshot = normalizer.get_normalized_tick_snapshot("SING30")
+    if snapshot["clock_status"] == "calibrated":
+        freshness = now_utc - snapshot["time_utc"]  # safe wall-clock compare
+```
+
+A normalized snapshot keeps the raw/UTC distinction explicit:
+
+```python
+{
+    "symbol": "SING30",
+    "bid": ...,
+    "ask": ...,
+    "last": ...,
+    "volume": ...,
+    "raw_time": 1717243200,  # numeric MT5 epoch, as labeled by broker
+    "time_utc": datetime(..., tzinfo=UTC),  # None unless calibrated
+    "server_clock_offset_seconds": 10800.0,  # None unless calibrated
+    "clock_status": "calibrated",  # or "uncalibrated"
+}
+```
+
+### Calibration evidence
+
+The offset is never inferred from `latest_tick_time - now` alone. Each
+calibration sample fetches the live `symbol_info_tick()` value **and** recent
+UTC-labeled `copy_ticks_range()` data, then searches every recent copied row
+(not just the newest) for one that matches the live tick on its shared
+`bid`/`ask`/`last`/`volume` fields (with at least one agreeing positive price
+field); the newest matching row sets the resulting offset, which must align
+to a 30-minute increment within the realistic UTC-14..UTC+14 range.
+`time_msc` is preferred over second-resolution `time` on both sides when
+available.
+
+An offset is accepted only after `min_agreeing_samples` (default 2)
+**distinct** matched tick events agree on the same rounded offset — repeated
+observations of one unchanged tick never count twice, so an illiquid symbol
+cannot self-confirm. Evidence can come from multiple samples of one actively
+updating symbol (paced by `sample_interval_seconds`) or across several
+symbols passed to the constructor. Any two matched samples that disagree
+abort the calibration (`offset_disagreement`).
+
+### Failure modes and fail-closed behavior
+
+When calibration cannot be established safely — closed market or weekend
+(`no_copied_ticks`), price disagreement or a non-half-hour delta
+(`no_matching_event`), a single matched event only (`insufficient_agreement`),
+missing live data (`no_live_tick`), or an unrealistic delta
+(`implausible_offset`) — snapshots fail closed: `clock_status` is
+`"uncalibrated"`, `time_utc` and `server_clock_offset_seconds` are `None`,
+and the raw fields remain available. A raw timestamp is never silently
+treated as UTC. `normalizer.calibration` exposes the full
+`TickClockCalibration` diagnostics: status, offset, distinct sample count,
+evidence symbols, and last calibration time (`to_dict()` for serialization).
+
+### Caching and revalidation
+
+Calibration is a property of the broker server clock, so it is cached on the
+normalizer (keep one instance per MT5 connection/account) and reused across
+symbols and calls. A cached offset is recomputed in three cases:
+
+- After `max_calibration_age_seconds` (default 6 hours, bounding
+  DST-transition staleness), unconditionally.
+- Immediately when a normalized timestamp lands more than two minutes in the
+  future — evidence that the broker offset grew, e.g. a UTC+2 to UTC+3
+  transition.
+- Periodically, at most once per `revalidation_interval_seconds` (default 5
+  minutes), a single confirming sample is taken against the still-cached
+  offset; a disagreement forces full recalibration. This is what catches a
+  broker offset _decrease_ (e.g. UTC+3 to UTC+2): the resulting normalized
+  time looks stale rather than future, which a future-skew check alone
+  cannot distinguish from ordinary quiet-market staleness.
+
+If recalibration still cannot validate the timestamp, the snapshot fails
+closed. Failed calibrations are recorded for diagnostics but never reused;
+retries are bounded to at most once per `failed_calibration_retry_seconds`
+(default 30 seconds) rather than on every call, so a closed or illiquid
+market does not trigger a full resample on every snapshot request.
+
+A genuinely stale tick (weekend, illiquid symbol) under a valid calibration
+normalizes to its true **past** UTC instant — staleness is preserved, never
+corrected to the present.
+
+mt5cli owns MT5 timestamp normalization only. Freshness thresholds,
+entry-blocking rules, and other strategy policy belong in downstream
+applications.
+
 ## Migration from application-local helpers
 
 | Application-local concern                                | mt5cli replacement                                                     |
@@ -234,7 +325,7 @@ No offset is applied automatically by history retrieval helpers.
 | Local order or position margin estimation                | `estimate_order_margin()`, `calculate_positions_margin()`              |
 | Local closed-bar fetch from a session                    | `fetch_latest_closed_rates()` or `fetch_latest_closed_rates_indexed()` |
 | Local recent deal history fetch from a session           | `client.recent_history_deals()`                                        |
-| Local broker server clock offset measurement             | `estimate_server_clock_offset_seconds()`                               |
+| Local broker server clock offset measurement             | `TickClockNormalizer`                                                  |
 | Local SL/TP price derivation                             | `determine_order_limits()`                                             |
 | Throttled SQLite history loop with ad-hoc error handling | `ThrottledHistoryUpdater(suppress_errors=True)`                        |
 

{mt5cli-1.2.3 → mt5cli-1.3.0}/mt5cli/__init__.py

@@ -40,8 +40,11 @@ from .marketdata import (
 )
 from .observability import update_observability, update_observability_with_config
 from .trading import (
+    CalibrationStatus,
+    ClockStatus,
     ExecutionStatus,
     MarginVolume,
+    NormalizedTickSnapshot,
     OrderExecutionResult,
     OrderFillingMode,
     OrderLimits,
@@ -49,6 +52,8 @@ from .trading import (
     OrderTimeMode,
     PositionSide,
     ProjectionMode,
+    TickClockCalibration,
+    TickClockNormalizer,
     calculate_account_projected_margin_ratio,
     calculate_margin_and_volume,
     calculate_new_position_margin_ratio,
@@ -65,7 +70,6 @@ from .trading import (
     determine_order_limits,
     ensure_symbol_selected,
     estimate_order_margin,
-    estimate_server_clock_offset_seconds,
     extract_tick_price,
     fetch_latest_closed_rates_indexed,
     get_account_snapshot,

{mt5cli-1.2.3 → mt5cli-1.3.0}/mt5cli/cli.py

@@ -6,11 +6,10 @@ import json
 import logging
 from contextlib import closing
 from dataclasses import dataclass
-from datetime import datetime
+from datetime import datetime  # noqa: TC003
 from pathlib import Path  # noqa: TC003
 from typing import TYPE_CHECKING, Annotated, Any, cast
 
-import click
 import pandas as pd
 import typer
 
@@ -46,126 +45,6 @@ if TYPE_CHECKING:
 
 logger = logging.getLogger(__name__)
 
-# ---------------------------------------------------------------------------
-# Click parameter types
-# ---------------------------------------------------------------------------
-
-
-class _DateTimeType(click.ParamType[datetime]):
-    """Click parameter type for ISO 8601 datetime strings."""
-
-    name = "DATETIME"
-
-    def convert(
-        self,
-        value: object,
-        param: click.Parameter | None = None,
-        ctx: click.Context | None = None,
-    ) -> datetime:
-        """Convert a string value to a timezone-aware datetime.
-
-        Args:
-            value: Raw value from the command line.
-            param: Click parameter instance.
-            ctx: Click context.
-
-        Returns:
-            Parsed datetime.
-        """
-        if isinstance(value, datetime):
-            return value
-        try:
-            return parse_datetime(str(value))
-        except ValueError as exc:
-            self.fail(str(exc), param, ctx)
-
-
-class _TimeframeType(click.ParamType[int]):
-    """Click parameter type for MT5 timeframe values."""
-
-    name = "TIMEFRAME"
-
-    def convert(
-        self,
-        value: object,
-        param: click.Parameter | None = None,
-        ctx: click.Context | None = None,
-    ) -> int:
-        """Convert a string or integer value to a timeframe integer.
-
-        Args:
-            value: Raw value from the command line.
-            param: Click parameter instance.
-            ctx: Click context.
-
-        Returns:
-            Integer timeframe value.
-        """
-        try:
-            return parse_timeframe(value)
-        except ValueError as exc:
-            self.fail(str(exc), param, ctx)
-
-
-class _TickFlagsType(click.ParamType[int]):
-    """Click parameter type for MT5 tick copy flags."""
-
-    name = "FLAGS"
-
-    def convert(
-        self,
-        value: object,
-        param: click.Parameter | None = None,
-        ctx: click.Context | None = None,
-    ) -> int:
-        """Convert a string or integer value to a tick flags integer.
-
-        Args:
-            value: Raw value from the command line.
-            param: Click parameter instance.
-            ctx: Click context.
-
-        Returns:
-            Integer tick flag value.
-        """
-        try:
-            return parse_tick_flags(value)
-        except ValueError as exc:
-            self.fail(str(exc), param, ctx)
-
-
-class _RequestType(click.ParamType[dict[str, Any]]):
-    """Click parameter type for JSON order requests."""
-
-    name = "REQUEST"
-
-    def convert(
-        self,
-        value: object,
-        param: click.Parameter | None = None,
-        ctx: click.Context | None = None,
-    ) -> dict[str, Any]:
-        """Convert a raw CLI value to an order request dictionary.
-
-        Args:
-            value: Raw value from the command line.
-            param: Click parameter instance.
-            ctx: Click context.
-
-        Returns:
-            Parsed request dictionary.
-        """
-        try:
-            return parse_request(str(value))
-        except ValueError as exc:
-            self.fail(str(exc), param, ctx)
-
-
-DATETIME_TYPE = _DateTimeType()
-TIMEFRAME_TYPE = _TimeframeType()
-TICK_FLAGS_TYPE = _TickFlagsType()
-REQUEST_TYPE = _RequestType()
-
 
 def _parse_datetime_parameter(value: str) -> datetime:
     try:

{mt5cli-1.2.3 → mt5cli-1.3.0}/mt5cli/client.py

@@ -355,12 +355,14 @@ def _shutdown_client(client: Mt5DataClient, *, raise_on_error: bool) -> None:
     failure is raised as the stable normalized exception. When False (an
     initialization or session-body exception is already propagating), the
     failure is logged so the primary exception passes through unchanged.
-    """  # noqa: DOC501
+
+    """
     try:
         client.shutdown()
     except Exception as exc:
         if raise_on_error:
-            raise normalize_mt5_exception(exc) from exc
+            normalized = normalize_mt5_exception(exc)
+            raise normalized from exc
         logger.warning("MT5 shutdown failed during cleanup", exc_info=True)
 
 
@@ -385,7 +387,7 @@ def _connected_client(
     Yields:
         Connected ``Mt5DataClient`` instance.
 
-    """  # noqa: DOC501
+    """
     client = (
         Mt5DataClient(config=config)
         if retry_count is None
@@ -395,7 +397,8 @@ def _connected_client(
         client.initialize_and_login_mt5()
     except Exception as exc:
         _shutdown_client(client, raise_on_error=False)
-        raise normalize_mt5_exception(exc) from exc
+        normalized = normalize_mt5_exception(exc)
+        raise normalized from exc
     try:
         yield client
     except BaseException:
@@ -420,12 +423,14 @@ def _run_with_client(
 
     Returns:
         Value returned by ``fetch_fn``.
-    """  # noqa: DOC501
+
+    """
     with _connected_client(config, retry_count=retry_count) as client:
         try:
             return fetch_fn(client)
         except Mt5RuntimeError as exc:
-            raise normalize_mt5_exception(exc) from exc
+            normalized = normalize_mt5_exception(exc)
+            raise normalized from exc
 
 
 class _BaseMT5Client:
@@ -480,7 +485,8 @@ class _BaseMT5Client:
 
         Returns:
             This client instance.
-        """  # noqa: DOC501
+
+        """
         if self._client is not None:
             return self
         client = Mt5DataClient(config=self._config, retry_count=self._retry_count)
@@ -488,7 +494,8 @@ class _BaseMT5Client:
             client.initialize_and_login_mt5()
         except Exception as exc:
             _shutdown_client(client, raise_on_error=False)
-            raise normalize_mt5_exception(exc) from exc
+            normalized = normalize_mt5_exception(exc)
+            raise normalized from exc
         self._client = client
         self._owns_client = True  # only set when this method created the client
         return self

{mt5cli-1.2.3 → mt5cli-1.3.0}/mt5cli/contract.py

@@ -112,10 +112,15 @@ STABLE_SDK_EXPORTS: frozenset[str] = frozenset({
     "OrderTimeMode",
     "PositionSide",
     "ProjectionMode",
+    "CalibrationStatus",
+    "ClockStatus",
     "ExecutionStatus",
     "MarginVolume",
+    "NormalizedTickSnapshot",
     "OrderExecutionResult",
     "OrderLimits",
+    "TickClockCalibration",
+    "TickClockNormalizer",
     "RateTarget",
     "ThrottledHistoryUpdater",
     "build_config",
@@ -141,7 +146,6 @@ STABLE_SDK_EXPORTS: frozenset[str] = frozenset({
     "drop_forming_rate_bar",
     "ensure_symbol_selected",
     "estimate_order_margin",
-    "estimate_server_clock_offset_seconds",
     "extract_tick_price",
     "fetch_latest_closed_rates",
     "fetch_latest_closed_rates_indexed",

{mt5cli-1.2.3 → mt5cli-1.3.0}/mt5cli/history.py

@@ -51,6 +51,10 @@ DEFAULT_HISTORY_DATASETS: frozenset[Dataset] = frozenset({
     Dataset.history_orders,
     Dataset.history_deals,
 })
+_EXPECTED_EMPTY_HISTORY_DATASETS: frozenset[Dataset] = frozenset({
+    Dataset.history_orders,
+    Dataset.history_deals,
+})
 
 _HISTORY_DEDUP_KEYS: dict[Dataset, tuple[tuple[str, ...], ...]] = {
     Dataset.rates: DEDUP_KEYS[DataKind.rates],
@@ -1341,11 +1345,20 @@ def get_incremental_start_datetime(
     return starts[symbol, timeframe]
 
 
+def _empty_dataset_log_level(dataset: Dataset) -> int:
+    """Return the log level for a schema-less dataset result."""
+    if dataset in _EXPECTED_EMPTY_HISTORY_DATASETS:
+        return logging.INFO
+    return logging.WARNING
+
+
 def append_dataframe(
     conn: sqlite3.Connection,
     frame: pd.DataFrame,
     table_name: str,
     if_exists: IfExists,
+    *,
+    empty_log_level: int = logging.WARNING,
 ) -> bool:
     """Append a DataFrame to SQLite when it has a schema.
 
@@ -1353,7 +1366,11 @@ def append_dataframe(
         True if a table was written, False if the frame had no columns.
     """
     if len(frame.columns) == 0:
-        logger.warning("Skipping %s: dataset returned no columns", table_name)
+        logger.log(
+            empty_log_level,
+            "Skipping %s: dataset returned no columns",
+            table_name,
+        )
         return False
     writable = _canonicalize_sqlite_time_columns(frame)
     writable.to_sql(  # type: ignore[reportUnknownMemberType]
@@ -1409,7 +1426,13 @@ def write_streamed_frame(
         True if the dataset table exists after this write attempt.
     """
     write_mode = IfExists.APPEND if table_exists else if_exists
-    if append_dataframe(conn, frame, dataset.table_name, write_mode):
+    if append_dataframe(
+        conn,
+        frame,
+        dataset.table_name,
+        write_mode,
+        empty_log_level=_empty_dataset_log_level(dataset),
+    ):
         record_written_columns(written_columns, dataset, frame)
         return True
     return table_exists
@@ -1751,7 +1774,8 @@ def _stream_symbol_frames(
     for sym in symbols:
         frame = fetch_frame(sym)
         if len(frame.columns) == 0:
-            logger.warning(
+            logger.log(
+                _empty_dataset_log_level(dataset),
                 "Skipping %s for symbol=%s: dataset returned no columns",
                 dataset.table_name,
                 sym,
@@ -2243,8 +2267,8 @@ def _finalize_incremental_writes(
             Dataset.history_deals not in written_columns
             and Dataset.history_deals not in written_tables
         ):
-            logger.warning(
-                "with_views ignored: history_deals table was not available",
+            logger.info(
+                "Skipping history-deal views: no history_deals data was available",
             )
 
 
@@ -2755,11 +2779,7 @@ class ThrottledHistoryUpdater:
             When ``suppress_errors`` is False, recoverable update failures
             propagate to the caller.
 
-        Raises:
-            Mt5ConnectionError: When ``suppress_errors`` is False and the
-                update fails with a recoverable MT5, SQLite, validation, or
-                I/O error.
-        """  # noqa: DOC502
+        """
         if not self.should_update():
             return False
         try:
@@ -2852,9 +2872,13 @@ def collect_history(
                 conn,
                 written_columns[Dataset.history_deals],
             )
+        elif with_views and Dataset.history_deals in selected:
+            logger.info(
+                "Skipping history-deal views: no history_deals data was collected",
+            )
         elif with_views:
             logger.warning(
-                "--with-views ignored: history_deals table was not written",
+                "--with-views requires the history_deals dataset",
             )
     logger.info(
         "Collected %s for %d symbol(s) into %s",