mt5cli 1.2.4__tar.gz → 1.3.1__tar.gz

{mt5cli-1.2.4 → mt5cli-1.3.1}/.github/workflows/ci.yml

@@ -62,7 +62,7 @@ jobs:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0  # v6.0.2
         with:
           persist-credentials: false
-      - uses: astral-sh/setup-uv@f98e06938123ccabd21905ea5d0069192241f9f1  # v8.3.1
+      - uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990  # v8.3.2
       - name: Check CLI help with minimum Click and Typer versions
         run: uv run --with click==8.4.0 --with typer==0.26.0 mt5cli --help
   python-docs-deploy:

{mt5cli-1.2.4 → mt5cli-1.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mt5cli
-Version: 1.2.4
+Version: 1.3.1
 Summary: Generic MT5 data and execution infrastructure for Python applications
 Project-URL: Repository, https://github.com/dceoy/mt5cli.git
 Author-email: dceoy <dceoy@users.noreply.github.com>

{mt5cli-1.2.4 → mt5cli-1.3.1}/docs/api/public-contract.md

@@ -102,7 +102,10 @@ strategy entries, exits, Kelly sizing, or signal logic.
 | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
 | `get_account_snapshot`, `get_symbol_snapshot`, `get_tick_snapshot`, `get_positions_frame`                                      | Normalized account/symbol/tick/position views                     |
 | `extract_tick_price`                                                                                                           | Positive finite bid/ask extraction from tick mappings             |
-| `estimate_server_clock_offset_seconds`                                                                                         | Broker server clock offset from true UTC, via the latest tick     |
+| `TickClockNormalizer`                                                                                                          | Connection-scoped, copied-tick-validated UTC tick normalization   |
+| `TickClockCalibration`                                                                                                         | Server-clock calibration diagnostics (offset, evidence, status)   |
+| `NormalizedTickSnapshot`                                                                                                       | Typed latest-tick contract with raw vs validated UTC timestamps   |
+| `CalibrationStatus`, `ClockStatus`                                                                                             | Literal status types for clock calibration and normalization      |
 | `detect_position_side`                                                                                                         | Net long / short / flat from open positions                       |
 | `calculate_spread_ratio`                                                                                                       | Relative bid-ask spread                                           |
 | `calculate_margin_and_volume`, `calculate_volume_by_margin`, `calculate_new_position_margin_ratio`                             | Margin budget and volume sizing                                   |

{mt5cli-1.2.4 → mt5cli-1.3.1}/docs/api/trading.md

@@ -215,14 +215,115 @@ Any broker-specific offset handling therefore requires evidence from the live
 data source in use and must be applied separately. Do not subtract a presumed
 UTC+2 or UTC+3 offset from timestamps that are already UTC.
 
-`estimate_server_clock_offset_seconds()` reads the latest tick for a symbol
-and returns the broker's clock offset from true UTC, rounded to the nearest
-half hour, or `None` when no valid tick time is available. Use it to validate a
-suspected offset for the live `symbol_info_tick()` source before applying the
-result in downstream time-window calculations.
-
 No offset is applied automatically by history retrieval helpers.
 
+## UTC normalization for live ticks
+
+`TickClockNormalizer` provides the stable API for obtaining a latest tick with
+a **validated** UTC timestamp, so strategy applications never compare a raw
+server-labeled tick time against `datetime.now(UTC)` themselves.
+
+```python
+from mt5cli import TickClockNormalizer, mt5_session
+
+with mt5_session() as client:
+    normalizer = TickClockNormalizer(client, ["SING30", "EURUSD"])
+    snapshot = normalizer.get_normalized_tick_snapshot("SING30")
+    if snapshot["clock_status"] == "calibrated":
+        freshness = now_utc - snapshot["time_utc"]  # safe wall-clock compare
+```
+
+A normalized snapshot keeps the raw/UTC distinction explicit:
+
+```python
+{
+    "symbol": "SING30",
+    "bid": ...,
+    "ask": ...,
+    "last": ...,
+    "volume": ...,
+    "raw_time": 1717243200,  # numeric MT5 epoch, as labeled by broker
+    "time_utc": datetime(..., tzinfo=UTC),  # None unless calibrated
+    "server_clock_offset_seconds": 10800.0,  # None unless calibrated
+    "clock_status": "calibrated",  # or "uncalibrated"
+}
+```
+
+### Calibration evidence
+
+The offset is never inferred from `latest_tick_time - now` alone. Each
+calibration sample fetches the live `symbol_info_tick()` value **and** recent
+UTC-labeled `copy_ticks_range()` data, then searches every recent copied row
+(not just the newest) for one that matches the live tick on its shared
+`bid`/`ask`/`last`/`volume` fields (with at least one agreeing positive price
+field); the newest matching row sets the resulting offset, which must align
+to a 30-minute increment within the realistic UTC-14..UTC+14 range.
+`time_msc` is preferred over second-resolution `time` on both sides when
+available.
+
+An offset is accepted only after `min_agreeing_samples` (default 2)
+**distinct** matched tick events agree on the same rounded offset — repeated
+observations of one unchanged tick never count twice, so an illiquid symbol
+cannot self-confirm. Evidence can come from multiple samples of one actively
+updating symbol (paced by `sample_interval_seconds`) or across several
+symbols passed to the constructor. Any two matched samples that disagree
+abort the calibration (`offset_disagreement`).
+
+The `copy_ticks_range()` query backing each sample extends its upper bound
+`_MAX_FUTURE_SKEW_SECONDS` (2 minutes) past the host clock's current time, in
+addition to covering the full trailing `copied_window_seconds`. A live tick's
+true UTC event is not always at or before the host clock — feed/broker clock
+skew can put it a few seconds ahead — and without that tolerance the matching
+copied row would never be queried, so calibration would always report
+`no_matching_event`. The tolerance is bounded, not open-ended: an event
+further ahead than `_MAX_FUTURE_SKEW_SECONDS` is still excluded from the
+query and cannot be matched.
+
+### Failure modes and fail-closed behavior
+
+When calibration cannot be established safely — closed market or weekend
+(`no_copied_ticks`), price disagreement or a non-half-hour delta
+(`no_matching_event`), a single matched event only (`insufficient_agreement`),
+missing live data (`no_live_tick`), or an unrealistic delta
+(`implausible_offset`) — snapshots fail closed: `clock_status` is
+`"uncalibrated"`, `time_utc` and `server_clock_offset_seconds` are `None`,
+and the raw fields remain available. A raw timestamp is never silently
+treated as UTC. `normalizer.calibration` exposes the full
+`TickClockCalibration` diagnostics: status, offset, distinct sample count,
+evidence symbols, and last calibration time (`to_dict()` for serialization).
+
+### Caching and revalidation
+
+Calibration is a property of the broker server clock, so it is cached on the
+normalizer (keep one instance per MT5 connection/account) and reused across
+symbols and calls. A cached offset is recomputed in three cases:
+
+- After `max_calibration_age_seconds` (default 6 hours, bounding
+  DST-transition staleness), unconditionally.
+- Immediately when a normalized timestamp lands more than two minutes in the
+  future — evidence that the broker offset grew, e.g. a UTC+2 to UTC+3
+  transition.
+- Periodically, at most once per `revalidation_interval_seconds` (default 5
+  minutes), a single confirming sample is taken against the still-cached
+  offset; a disagreement forces full recalibration. This is what catches a
+  broker offset _decrease_ (e.g. UTC+3 to UTC+2): the resulting normalized
+  time looks stale rather than future, which a future-skew check alone
+  cannot distinguish from ordinary quiet-market staleness.
+
+If recalibration still cannot validate the timestamp, the snapshot fails
+closed. Failed calibrations are recorded for diagnostics but never reused;
+retries are bounded to at most once per `failed_calibration_retry_seconds`
+(default 30 seconds) rather than on every call, so a closed or illiquid
+market does not trigger a full resample on every snapshot request.
+
+A genuinely stale tick (weekend, illiquid symbol) under a valid calibration
+normalizes to its true **past** UTC instant — staleness is preserved, never
+corrected to the present.
+
+mt5cli owns MT5 timestamp normalization only. Freshness thresholds,
+entry-blocking rules, and other strategy policy belong in downstream
+applications.
+
 ## Migration from application-local helpers
 
 | Application-local concern                                | mt5cli replacement                                                     |
@@ -234,7 +335,7 @@ No offset is applied automatically by history retrieval helpers.
 | Local order or position margin estimation                | `estimate_order_margin()`, `calculate_positions_margin()`              |
 | Local closed-bar fetch from a session                    | `fetch_latest_closed_rates()` or `fetch_latest_closed_rates_indexed()` |
 | Local recent deal history fetch from a session           | `client.recent_history_deals()`                                        |
-| Local broker server clock offset measurement             | `estimate_server_clock_offset_seconds()`                               |
+| Local broker server clock offset measurement             | `TickClockNormalizer`                                                  |
 | Local SL/TP price derivation                             | `determine_order_limits()`                                             |
 | Throttled SQLite history loop with ad-hoc error handling | `ThrottledHistoryUpdater(suppress_errors=True)`                        |
 

{mt5cli-1.2.4 → mt5cli-1.3.1}/mt5cli/__init__.py

@@ -40,8 +40,11 @@ from .marketdata import (
 )
 from .observability import update_observability, update_observability_with_config
 from .trading import (
+    CalibrationStatus,
+    ClockStatus,
     ExecutionStatus,
     MarginVolume,
+    NormalizedTickSnapshot,
     OrderExecutionResult,
     OrderFillingMode,
     OrderLimits,
@@ -49,6 +52,8 @@ from .trading import (
     OrderTimeMode,
     PositionSide,
     ProjectionMode,
+    TickClockCalibration,
+    TickClockNormalizer,
     calculate_account_projected_margin_ratio,
     calculate_margin_and_volume,
     calculate_new_position_margin_ratio,
@@ -65,7 +70,6 @@ from .trading import (
     determine_order_limits,
     ensure_symbol_selected,
     estimate_order_margin,
-    estimate_server_clock_offset_seconds,
     extract_tick_price,
     fetch_latest_closed_rates_indexed,
     get_account_snapshot,

{mt5cli-1.2.4 → mt5cli-1.3.1}/mt5cli/contract.py

@@ -112,10 +112,15 @@ STABLE_SDK_EXPORTS: frozenset[str] = frozenset({
     "OrderTimeMode",
     "PositionSide",
     "ProjectionMode",
+    "CalibrationStatus",
+    "ClockStatus",
     "ExecutionStatus",
     "MarginVolume",
+    "NormalizedTickSnapshot",
     "OrderExecutionResult",
     "OrderLimits",
+    "TickClockCalibration",
+    "TickClockNormalizer",
     "RateTarget",
     "ThrottledHistoryUpdater",
     "build_config",
@@ -141,7 +146,6 @@ STABLE_SDK_EXPORTS: frozenset[str] = frozenset({
     "drop_forming_rate_bar",
     "ensure_symbol_selected",
     "estimate_order_margin",
-    "estimate_server_clock_offset_seconds",
     "extract_tick_price",
     "fetch_latest_closed_rates",
     "fetch_latest_closed_rates_indexed",

{mt5cli-1.2.4 → mt5cli-1.3.1}/mt5cli/history.py

@@ -51,6 +51,10 @@ DEFAULT_HISTORY_DATASETS: frozenset[Dataset] = frozenset({
     Dataset.history_orders,
     Dataset.history_deals,
 })
+_EXPECTED_EMPTY_HISTORY_DATASETS: frozenset[Dataset] = frozenset({
+    Dataset.history_orders,
+    Dataset.history_deals,
+})
 
 _HISTORY_DEDUP_KEYS: dict[Dataset, tuple[tuple[str, ...], ...]] = {
     Dataset.rates: DEDUP_KEYS[DataKind.rates],
@@ -1341,11 +1345,20 @@ def get_incremental_start_datetime(
     return starts[symbol, timeframe]
 
 
+def _empty_dataset_log_level(dataset: Dataset) -> int:
+    """Return the log level for a schema-less dataset result."""
+    if dataset in _EXPECTED_EMPTY_HISTORY_DATASETS:
+        return logging.INFO
+    return logging.WARNING
+
+
 def append_dataframe(
     conn: sqlite3.Connection,
     frame: pd.DataFrame,
     table_name: str,
     if_exists: IfExists,
+    *,
+    empty_log_level: int = logging.WARNING,
 ) -> bool:
     """Append a DataFrame to SQLite when it has a schema.
 
@@ -1353,7 +1366,11 @@ def append_dataframe(
         True if a table was written, False if the frame had no columns.
     """
     if len(frame.columns) == 0:
-        logger.warning("Skipping %s: dataset returned no columns", table_name)
+        logger.log(
+            empty_log_level,
+            "Skipping %s: dataset returned no columns",
+            table_name,
+        )
         return False
     writable = _canonicalize_sqlite_time_columns(frame)
     writable.to_sql(  # type: ignore[reportUnknownMemberType]
@@ -1409,7 +1426,13 @@ def write_streamed_frame(
         True if the dataset table exists after this write attempt.
     """
     write_mode = IfExists.APPEND if table_exists else if_exists
-    if append_dataframe(conn, frame, dataset.table_name, write_mode):
+    if append_dataframe(
+        conn,
+        frame,
+        dataset.table_name,
+        write_mode,
+        empty_log_level=_empty_dataset_log_level(dataset),
+    ):
         record_written_columns(written_columns, dataset, frame)
         return True
     return table_exists
@@ -1751,7 +1774,8 @@ def _stream_symbol_frames(
     for sym in symbols:
         frame = fetch_frame(sym)
         if len(frame.columns) == 0:
-            logger.warning(
+            logger.log(
+                _empty_dataset_log_level(dataset),
                 "Skipping %s for symbol=%s: dataset returned no columns",
                 dataset.table_name,
                 sym,
@@ -2243,8 +2267,8 @@ def _finalize_incremental_writes(
             Dataset.history_deals not in written_columns
             and Dataset.history_deals not in written_tables
         ):
-            logger.warning(
-                "with_views ignored: history_deals table was not available",
+            logger.info(
+                "Skipping history-deal views: no history_deals data was available",
             )
 
 
@@ -2848,9 +2872,13 @@ def collect_history(
                 conn,
                 written_columns[Dataset.history_deals],
             )
+        elif with_views and Dataset.history_deals in selected:
+            logger.info(
+                "Skipping history-deal views: no history_deals data was collected",
+            )
         elif with_views:
             logger.warning(
-                "--with-views ignored: history_deals table was not written",
+                "--with-views requires the history_deals dataset",
             )
     logger.info(
         "Collected %s for %d symbol(s) into %s",