polyester-bot 1.1.0 → 1.2.0

package/README.md

@@ -1,6 +1,6 @@
 <div align="center">
 
-<img src="polyester-bot.png" alt="Polyester" width="120" height="120" style="border-radius:50%; object-fit:cover;" />
+<img src="https://raw.githubusercontent.com/natanim-kemal/polyester-bot/main/polyester-bot.png" alt="Polyester" width="120" height="120" style="border-radius:50%; object-fit:cover;" />
 
 # Polyester Gap Hunter
 
@@ -20,15 +20,35 @@ scan pairs → find gap → BUY at the bid → ASK under the offer → CYCLE DON
 ```
 
 - **Gap mode** — buys the bid, asks one tick under the offer (maker on both legs).
-  A gap must clear the `0.20%` round-trip fee, so `GAP_MIN_PCT` defaults to `0.21%`.
+  Fees are read from the exchange (VIP tier) every 30 min; the gap floor follows
+  them (`2 x maker + GAP_MARGIN_PCT`, `0.11%` at VIP3). An explicit `GAP_MIN_PCT`
+  pins it instead.
 - **Volume fallback** — when nothing clears the fee bar, the deepest tightest pair is
-  cycled anyway to keep season volume up.
-- **Bail-out rule** — if more than `BETTER_ASKS_LIMIT` better asks sit under our
-  price for `PERSIST_SECONDS`, the ask is cancelled and re-placed under them
-  (maker exit, even at a loss). It then rests until the rule fires again.
+  cycled anyway to keep season volume up. The candidate with real trade flow wins
+  over the widest dead spread. A saturation ladder cuts the waiting:
+  after `SAT_PENNY_AFTER` consecutive unfilled fallback buys it pennies the bid,
+  after `SAT_TAKER_AFTER` it buys at the ask (taker fee), and the buy timeout
+  shrinks to `FALLBACK_BUY_TIMEOUT`.
+- **Chase rule (both sides)** — if `BETTER_ASKS_LIMIT` better asks (sell) or
+  `BETTER_BIDS_LIMIT` better bids (buy) beat our price for `PERSIST_SECONDS`,
+  the resting order is **amended** one tick more aggressive than them — one
+  atomic call, no unfilled window. A cancel happens only when the amend cannot.
+- **Dead-man switch** — `orders.cancel_all_after` is re-armed every 60 s; if the
+  bot dies or freezes without re-arming, the exchange cancels all our orders itself.
+- **Pre-flight check** — before a buy is placed, `orders.preview_order` asks the
+  exchange whether it would be admissible; only a clear rejection blocks
+  (throttled, never an outage blocking trades).
+- **Websocket books** — every pair's orderbook is streamed live (`orderbook.subscribe`);
+  readers prefer the fresh cache and fall back to REST when it goes quiet.
 - **One position** — never stacks trades; the exception is a `>20%` 24h mover.
 - **Volume tracker** — every cycle adds to `volume_state.json`, milestone alert
   every 10,000 USDT.
+- **Daily equity line** — once a day one `EQUITY` alert: total in USDT (balances ×
+  fresh mids), per-position breakdown, 24 h delta vs yesterday's snapshot
+  (`equity_state.json`).
+- **TP triggers** — `TP_TRIGGER_PCT` arms a full-size server-side take-profit on a held
+  position; the trigger reserves the whole inventory while armed, so the bot cannot
+  sell alongside it. Off by default.
 
 ## Install
 
@@ -97,7 +117,7 @@ polyester-bot
 
 ```
 ├── bot/
-│   ├── monitor.py            # the bot: scanner, executor, bail-out rule
+│   ├── monitor.py            # the bot: scanner, executor, chase rule
 │   ├── setup.py              # guided setup: ask, check, write the .env
 │   ├── polyester_farm.py     # season volume, daily paper-asset claim
 │   ├── paths.py              # where .env and the state files live
@@ -115,12 +135,21 @@ polyester-bot
 │   ├── polyester-claim.service  # daily paper-asset claim
 │   ├── polyester-claim.timer    # 00:10 UTC, persistent
 │   └── deploy.sh                # install code + units, restart
-├── tests/
-│   ├── test_sell_loop.py        # regression test for the cancel storm
+├── tests/                    # 14 suites, all run in CI before any deploy:
+│   ├── test_sell_loop.py        # the re-ask/cancel machine, end to end
 │   ├── test_freeze_detect.py    # detects a paused sandbox from clock skew
 │   ├── test_paths.py            # .env and state file resolution
 │   ├── test_stuck.py            # a filled buy survives a lagging balance
-│   └── test_setup.py            # the setup wizard and account derivation
+│   ├── test_setup.py            # the setup wizard and account derivation
+│   ├── test_fees.py             # fee math, gap floor and entry ladder
+│   ├── test_chase.py            # both sides chase; sell trigger on one better ask
+│   ├── test_amend.py            # chase amends in place, cancel only as fallback
+│   ├── test_tp_trigger.py       # server-side take-profit triggers (opt-in)
+│   ├── test_ws.py               # websocket orderbook feed with REST fallback
+│   ├── test_liquidity.py        # fallback prefers a live tape over a wide dead book
+│   ├── test_equity.py           # daily equity report from balances x mids
+│   ├── test_preflight.py        # admissibility check before a buy is placed
+│   └── test_orphan.py           # orphan-clear runs hourly, not every poll
 ├── .github/workflows/
 │   ├── deploy.yml               # push to main → deploy to the sandbox
 │   └── keepalive.yml            # every 10 min: keep the sandbox awake
@@ -136,13 +165,21 @@ The real file stays on the machine and is gitignored — a deploy never overwrit
 
 | Key | Default | Meaning |
 | --- | --- | --- |
-| `GAP_MIN_PCT` | `0.21` | minimum spread to take in gap mode |
+| `GAP_MIN_PCT` | fee-driven (`0.11`) | minimum spread to take in gap mode |
+| `GAP_MARGIN_PCT` | `0.01` | profit margin added on top of the round-trip fee |
+| `FEES_REFRESH_SECONDS` | `1800` | how often per-pair fees are re-read from the exchange |
 | `DEPTH_MIN_USD` | `50` | minimum depth on both sides |
 | `SCAN_SECONDS` | `10` | how often to scan for a gap |
 | `POLL_SECONDS` | `2` | how often to manage the open position |
-| `PERSIST_SECONDS` | `60` | how long better asks must persist before the bail-out fires |
-| `BETTER_ASKS_LIMIT` | `2` | how many better asks trigger the bail-out |
+| `PERSIST_SECONDS` | `60` | how long better prices must persist before a chase fires |
+| `BETTER_ASKS_LIMIT` | `1` | better asks under our ask that trigger the sell chase |
+| `BETTER_BIDS_LIMIT` | `1` | better bids over our bid that trigger the buy chase |
 | `BUY_TIMEOUT_SECONDS` | `180` | cancel an unfilled buy after this long |
+| `SAT_PENNY_AFTER` | `2` | consecutive fallback buy timeouts before penny bids |
+| `SAT_TAKER_AFTER` | `4` | consecutive fallback buy timeouts before buying at the ask |
+| `FALLBACK_BUY_TIMEOUT` | `60` | fallback buy timeout once the ladder is engaged |
+| `DEADMAN_TIMEOUT` | `300` | auto-cancel window re-armed every 60s; `0` disables |
+| `TP_TRIGGER_PCT` | `0` | arm server-side take-profit on sell entry; `0` = never arm |
 | `FREEZE_SECONDS` | `300` | wall-vs-monotonic clock skew that counts as a paused sandbox |
 | `STALL_SECONDS` | `900` | how long the loop may hang before it counts as a pause |
 | `FALLBACK_*` | see example | volume fallback mode |
@@ -208,44 +245,35 @@ journalctl -u polyester-claim.service
 
 ## Tests
 
-`tests/test_sell_loop.py` replays the sell state machine against a fake exchange.
-It reproduces the bug where the bot cancelled the ask it had just placed — the
-`canceling` flag was never cleared once the order reached a terminal state, so
-every re-ask was cancelled again on the next poll.
-
-```bash
-python tests/test_sell_loop.py
-```
-
-Exits non-zero if the bot cancels more than once per bail-out.
-
-`tests/test_freeze_detect.py` pulls the real freeze-detection code out of
-`bot/monitor.py` and drives it: a normal poll does nothing, the real 1h47m
-pause triggers a re-sync, a stalled loop triggers it too, and an unreachable
-exchange keeps the position instead of forgetting it.
-
-```bash
-python tests/test_freeze_detect.py
-```
-
-`tests/test_paths.py` checks where the bot looks for its `.env` and its state
-files: the working directory wins, the script directory is the fallback, and an
-existing state file is never moved.
+All 14 suites run in CI on every push, before anything is deployed, together with a
+build of the Python package and a smoke test of its entry point.
+`python tests/test_<name>.py` runs any one locally.
 
 ```bash
-python tests/test_paths.py
+for t in tests/test_*.py; do python "$t" || exit 1; done
 ```
 
-`tests/test_stuck.py` replays the bug where a filled buy was invisible because
-the balance lagged behind the order, which froze the bot for hours. It fails on
-the old code and passes on the new.
-
-`tests/test_setup.py` checks the setup wizard: seed formats, the `.env` merge,
-the home copy, and that the bot asks the exchange for its account id instead of
-having one baked in.
-
-All five run in CI on every push, before anything is deployed, together with a
-build of the Python package and a smoke test of its entry point.
+Exits non-zero on the first failing suite. Coverage map:
+
+`test_sell_loop.py` replays the sell state machine against a fake exchange — the
+bug where the bot cancelled the ask it had just placed.
+`test_freeze_detect.py` drives the freeze guard — the real 1h47m pause triggers a
+re-sync, an unreachable exchange keeps the position.
+`test_stuck.py` replays the bug where a filled buy was invisible because the
+balance lagged behind the order.
+`test_setup.py` checks the setup wizard and that the account id is derived, never
+baked in.
+`test_paths.py` checks `.env` and state file resolution.
+`test_fees.py` pins the fee math: the gap floor, the entry ladder, cycle fees.
+`test_chase.py` checks both sides chase one tick past the competitor, and the
+flicking-competitor guard.
+`test_amend.py` checks orders amend atomically and cancel only when unavoidable.
+`test_tp_trigger.py` arms a server-side take-profit when enabled (off by default).
+`test_ws.py` checks the websocket book feed populates the cache and REST falls back.
+`test_liquidity.py` checks the fallback prefers a live tape.
+`test_equity.py` checks the daily equity report and its snapshot file.
+`test_preflight.py` checks the admissibility check blocks a clear rejection only.
+`test_orphan.py` checks the orphan-clear runs hourly, not every poll.
 
 ## Ops
 

package/bot/monitor.py

@@ -63,19 +63,33 @@ _SK = nacl.signing.SigningKey(bytes.fromhex(os.environ.get("POLYESTER_ED25519_SE
 ACCOUNT = os.environ.get("POLYESTER_ACCOUNT_ID", "")
 
 QUOTE = "USDT"
-RT_FEE_PCT = Decimal("0.20")
-GAP_MIN_PCT = Decimal(os.environ.get("GAP_MIN_PCT", "0.21"))
+# Real per-pair fees, read from the exchange every 30 min (observed VIP3:
+# maker 0.05 / taker 0.105). An explicit GAP_MIN_PCT env still pins the floor.
+DEFAULT_MAKER_PCT = Decimal("0.05")
+DEFAULT_TAKER_PCT = Decimal("0.105")
+GAP_MARGIN_PCT = Decimal(os.environ.get("GAP_MARGIN_PCT", "0.01"))
+GAP_MIN_FIXED = (Decimal(os.environ["GAP_MIN_PCT"])
+                 if os.environ.get("GAP_MIN_PCT") else None)
+FEES = {}
+FEES_REFRESH_SECONDS = float(os.environ.get("FEES_REFRESH_SECONDS", "1800"))
 DEPTH_MIN_USD = Decimal(os.environ.get("DEPTH_MIN_USD", "50"))
 FALLBACK_SYMBOL = os.environ.get("FALLBACK_SYMBOL", "").strip()
 FALLBACK_MIN_GAP_PCT = Decimal(os.environ.get("FALLBACK_MIN_GAP_PCT", "0.05"))
 FALLBACK_DEPTH_MIN_USD = Decimal(os.environ.get("FALLBACK_DEPTH_MIN_USD", "1000"))
 FALLBACK_COOLDOWN = float(os.environ.get("FALLBACK_COOLDOWN", "60"))
 FALLBACK_PENNY = os.environ.get("FALLBACK_PENNY", "0") == "1"
+SAT_PENNY_AFTER = int(os.environ.get("SAT_PENNY_AFTER", "2"))
+SAT_TAKER_AFTER = int(os.environ.get("SAT_TAKER_AFTER", "4"))
+FALLBACK_BUY_TIMEOUT = float(os.environ.get("FALLBACK_BUY_TIMEOUT", "60"))
+TP_TRIGGER_PCT = Decimal(os.environ.get("TP_TRIGGER_PCT", "0"))
 SCAN_SECONDS = float(os.environ.get("SCAN_SECONDS", "10"))
 POLL_SECONDS = float(os.environ.get("POLL_SECONDS", "2"))
 PERSIST_SECONDS = float(os.environ.get("PERSIST_SECONDS", "60"))
-BETTER_ASKS_LIMIT = int(os.environ.get("BETTER_ASKS_LIMIT", "2"))
+BETTER_ASKS_LIMIT = int(os.environ.get("BETTER_ASKS_LIMIT", "1"))
+BETTER_BIDS_LIMIT = int(os.environ.get("BETTER_BIDS_LIMIT", "1"))
 BUY_TIMEOUT_SECONDS = float(os.environ.get("BUY_TIMEOUT_SECONDS", "180"))
+DEADMAN_TIMEOUT = int(os.environ.get("DEADMAN_TIMEOUT", "120"))
+DEADMAN_REFRESH = 60
 FEE_BUFFER = Decimal("1.0015")
 FREEZE_SECONDS = float(os.environ.get("FREEZE_SECONDS", "300"))
 STALL_SECONDS = float(os.environ.get("STALL_SECONDS", "900"))
@@ -83,6 +97,80 @@ BALANCE_GRACE = float(os.environ.get("BALANCE_GRACE_SECONDS", "120"))
 HEARTBEAT = "/var/tmp/polyester.heartbeat"
 
 
+def maker_fee_pct(fees, sym):
+    """Maker fee % for one pair, falling back to the observed VIP3 rate."""
+    return (fees.get(sym) or (DEFAULT_MAKER_PCT, DEFAULT_TAKER_PCT))[0]
+
+
+def taker_fee_pct(fees, sym):
+    return (fees.get(sym) or (DEFAULT_MAKER_PCT, DEFAULT_TAKER_PCT))[1]
+
+
+def rt_fee_pct(fees, sym):
+    """Round-trip % for two maker legs."""
+    return 2 * maker_fee_pct(fees, sym)
+
+
+def gap_min_pct(fixed, fees=None, sym=None):
+    """Profitable-gap floor: an explicit GAP_MIN_PCT wins, else 2 x maker + margin."""
+    if fixed is not None:
+        return fixed
+    return 2 * maker_fee_pct(fees or FEES, sym) + GAP_MARGIN_PCT
+
+
+def round_trip_fees(buy_usd, sell_usd, maker_pct):
+    """USDT fees for a maker round trip at maker_pct percent per leg."""
+    return (buy_usd + sell_usd) * maker_pct / 100
+
+
+def entry_price(tag, bid, ask, m, penny, taker):
+    """Where the BUY rests: gap buys always penny, a calm fallback joins the
+    bid, and the saturation ladder pennies then crosses the spread."""
+    if taker:
+        return ask
+    if tag == "GAP" or penny:
+        return min(bid + m["tick"], ask - m["tick"])
+    return bid
+
+
+def buy_timeout_for(tag, sat_n):
+    """Fallbacks on the saturation ladder stop waiting sooner."""
+    if tag != "GAP" and sat_n >= SAT_PENNY_AFTER:
+        return FALLBACK_BUY_TIMEOUT
+    return BUY_TIMEOUT_SECONDS
+
+
+async def deadman_rearm(c):
+    """Re-arm the exchange-side auto-cancel: if the bot stops re-arming,
+    every open order is cancelled once the deadline passes."""
+    try:
+        await c.orders.cancel_all_after(timeout_sec=DEADMAN_TIMEOUT)
+        return True
+    except Exception as e:
+        print("deadman re-arm failed:", repr(e), flush=True)
+        return False
+
+
+async def refresh_fees(c):
+    """Pull the exchange's real per-pair fee rates; keep the gap floor in step."""
+    global GAP_MIN_PCT
+    try:
+        fr = await c.fees.get_spot_fee_rates()
+        for r in fr.fee_rates:
+            FEES[r.symbol] = (pdec(r.maker_fee_rate_percent),
+                              pdec(r.taker_fee_rate_percent))
+        GAP_MIN_PCT = gap_min_pct(GAP_MIN_FIXED)
+        if FEES:
+            s0 = next(iter(FEES))
+            print(f"fees refreshed: {len(FEES)} pairs | maker {FEES[s0][0]}% "
+                  f"taker {FEES[s0][1]}% | gap floor {GAP_MIN_PCT}%", flush=True)
+    except Exception as e:
+        print("fee refresh failed:", repr(e), flush=True)
+
+
+GAP_MIN_PCT = gap_min_pct(GAP_MIN_FIXED)
+
+
 def frozen_seconds(wall, mono, last_wall, last_mono):
     """Seconds the VM spent frozen: wall clock advanced but the guest did not run.
 
@@ -242,20 +330,280 @@ async def order_state(c, sym, cid, scale):
     return (str(o.status).lower(), qdec(o.cum_qty, scale), pdec(o.avg_px))
 
 
+async def chase_buy(c, pos, m, now):
+    """Mirror of the ask bail-out: when competitors outbid our resting BUY
+    for too long, cancel and rebid one tick above the best bid, never into
+    the ask. Resets the timeout clock on every chase."""
+    sym = pos["symbol"]
+    bk = await book(c, sym, depth=10)
+    better = {pdec(b.price) for b in (bk.bids or [])
+              if pdec(b.price) > pos["buy_px"]}
+    track = pos["better"]
+    for p in list(track):
+        if p not in better:
+            del track[p]
+    for p in better:
+        track.setdefault(p, now)
+    aged = sum(1 for p, t0 in track.items() if now - t0 >= PERSIST_SECONDS)
+    if aged < BETTER_BIDS_LIMIT:
+        return
+    best = max(better)
+    new_px = best + m["tick"]
+    top = min((pdec(a.price) for a in (bk.asks or []) if pdec(a.price) > 0),
+              default=None)
+    if top is not None:
+        new_px = min(new_px, top - m["tick"])
+    if new_px <= pos["buy_px"]:
+        return
+    st, cum, avg = await order_state(c, sym, pos["buy_cid"], m["scale"])
+    left = floor_step((pos.get("qty_target") or Decimal(0))
+                      - floor_step(cum, m["step"]), m["step"])
+    if st not in TERMINAL:
+        try:
+            await c.orders.modify(key=ClientOrderId(pos["buy_cid"]),
+                                  symbol=sym, new_price=new_px)
+            pos["buy_px"] = new_px
+            pos["buy_at"] = now
+            pos["better"] = {}
+            notify("🔄", "BID CHASE",
+                   f"{sym} | outbid by {aged} bid(s) held {PERSIST_SECONDS:.0f}s "
+                   f"| amending to {num(new_px)}",
+                   wallet(sym.split("-")[0]))
+            return
+        except Exception as e:
+            print("amend fell back to cancel:", repr(e), flush=True)
+    try:
+        await c.orders.cancel(key=ClientOrderId(pos["buy_cid"]), symbol=sym)
+    except Exception:
+        pass
+    st, cum, avg = await order_state(c, sym, pos["buy_cid"], m["scale"])
+    left = floor_step((pos.get("qty_target") or Decimal(0))
+                      - floor_step(cum, m["step"]), m["step"])
+    cid = None
+    if left > 0 and left >= m["min_qty"] and left * new_px >= m["min_notional"]:
+        cid, _ = await place(c, sym, "buy", left, new_px)
+    if cid:
+        pos["buy_cid"] = cid
+        pos["buy_px"] = new_px
+        pos["buy_at"] = now
+        pos["better"] = {}
+        notify("🔄", "BID CHASE",
+               f"{sym} | outbid by {aged} bid(s) held {PERSIST_SECONDS:.0f}s "
+               f"| re-bidding {num(left)} @ {num(new_px)}",
+               wallet(sym.split("-")[0]))
+    # a failed or unneeded re-place is reconciled by the terminal/timeout paths
+
+
+async def place_tp_trigger(c, pos, m):
+    """Arm a server-side take-profit sell: the exchange dumps the position
+    itself when price rallies TP_TRIGGER_PCT above entry, bot frozen or not."""
+    if TP_TRIGGER_PCT <= 0:
+        return None
+    sym = pos["symbol"]
+    qty = floor_step(pos.get("qty") or Decimal(0), m["step"])
+    if qty <= 0 or qty < m["min_qty"]:
+        return None
+    basis = pos.get("buy_px") or Decimal(0)
+    ref = basis
+    if ref <= 0:
+        try:
+            bk = await book(c, sym, depth=1)
+            ref = pdec(bk.asks[0].price) if bk.asks else Decimal(0)
+        except Exception as e:
+            print("tp trigger skipped: book unreadable:", repr(e), flush=True)
+            return None
+    if ref <= 0:
+        return None
+    tp = ref * (1 + TP_TRIGGER_PCT / 100)
+    if qty * tp < m["min_notional"]:
+        return None
+    cid = "gap1-tp-" + uuid.uuid4().hex[:12]
+    try:
+        res = await c.triggers.create(symbol=sym, trigger_type="take_profit",
+                                      side="sell", qty=format(qty, "f"),
+                                      trigger_price=format(tp, "f"),
+                                      order_type="market",
+                                      client_trigger_id=cid)
+        pos["tp_cid"] = cid
+        pos["tp_tid"] = getattr(res, "trigger_id", None)
+        notify("🎯", "TP ARMED",
+               f"{sym} | exchange sells {num(qty)} @ {num(tp)} "
+               f"| entry {num(ref)}")
+        return res
+    except Exception as e:
+        print("tp trigger failed:", repr(e), flush=True)
+        return None
+
+
+async def cancel_stale_symbol_orders(c, sym):
+    """Cancel our own gap1- orders resting on this symbol. The TP trigger can
+    beat our own ask to the fill; a leftover sell would be unbacked."""
+    try:
+        stale = [o for o in rest_open_orders().get("orders", [])
+                 if str(o.get("clientOrderId", "")).startswith("gap1-")
+                 and o.get("symbol") == sym]
+        for o in stale:
+            try:
+                await c.orders.cancel(key=ClientOrderId(o["clientOrderId"]),
+                                      symbol=sym)
+            except Exception:
+                pass
+        return len(stale)
+    except Exception as e:
+        print("stale order sweep failed:", repr(e), flush=True)
+        return 0
+
+
+async def cancel_tp_trigger(c, pos):
+    """Best-effort disarm once the cycle closes."""
+    tid = pos.get("tp_tid")
+    if tid is not None:
+        try:
+            await c.triggers.cancel(trigger_id=tid)
+        except Exception:
+            pass
+        pos["tp_tid"] = None
+
+
+FALLBACK_CANDIDATES = 4
+
+
+async def pick_best_liquid(c, fb, top=FALLBACK_CANDIDATES):
+    """Among the tightest fallback candidates prefer the one whose tape is
+    alive; an unreadable tape counts as zero flow. Ties keep gap order."""
+    if not fb:
+        return None
+    best, best_n = None, -1
+    for row in fb[:top]:
+        try:
+            tr = await c.market_data.get_trades(symbol=row[1], limit=20)
+            n = len(tr.trades)
+        except Exception as e:
+            print("trade count skip", row[1], repr(e), flush=True)
+            n = 0
+        if n > best_n:
+            best, best_n = row, n
+    return best or fb[0]
+
+
+async def preflight_buy(c, sym, side, qty, price):
+    """Ask the exchange whether the buy would be admissible. Fail-open: only
+    a clear rejection blocks; an outage proceeds."""
+    try:
+        res = await c.orders.preview_order(symbol=sym, side=side,
+                                           order_type="limit", tif="gtc",
+                                           qty=format(qty, "f"),
+                                           price=format(price, "f"))
+        adm = getattr(res, "admissible", None)
+        if adm is None or adm:
+            return True, None
+        rej = getattr(res, "rejection", None)
+        code = getattr(rej, "code", None) or getattr(rej, "message", None) or "rejected"
+        return False, str(code)
+    except Exception as e:
+        print("pre-flight failed (proceeding):", repr(e), flush=True)
+        return True, None
+
+
+WS_BOOKS = {}
+
+
+def ws_book(sym, max_age=15.0):
+    """The last WS book snapshot for sym, if fresh enough to trust."""
+    rec = WS_BOOKS.get(sym)
+    if rec and (time.monotonic() - rec[1]) <= max_age:
+        return rec[0]
+    return None
+
+
+async def book(c, sym, depth=10, max_age=15.0):
+    """Orderbook for one pair: the WS-fed snapshot when fresh, REST otherwise."""
+    cached = ws_book(sym, max_age)
+    if cached is not None:
+        return cached
+    bk = await c.orderbook.get(symbol=sym, depth=depth)
+    WS_BOOKS[sym] = (bk, time.monotonic())
+    return bk
+
+
+async def equity_report(c):
+    """Daily equity line from real balances x fresh mids; keeps its own snapshot."""
+    try:
+        total = Decimal(0)
+        parts = []
+        for row in rest_balances():
+            sym0 = row.get("symbol")
+            amt = Decimal(row.get("trading") or 0)
+            if amt <= 0:
+                continue
+            if sym0 == QUOTE:
+                total += amt
+                continue
+            psym = sym0 + "-" + QUOTE
+            try:
+                bk = await book(c, psym, depth=1)
+                if bk.bids and bk.asks:
+                    mid = (pdec(bk.bids[0].price) + pdec(bk.asks[0].price)) / 2
+                    total += amt * mid
+                    parts.append(f"{sym0} {money(amt * mid)}")
+            except Exception:
+                print("equity skip", sym0, flush=True)
+        prev = None
+        try:
+            with open(state_file("equity_state.json"), encoding="utf-8") as f:
+                prev = Decimal(json.load(f)["equity"])
+        except (OSError, ValueError, KeyError):
+            pass
+        delta = f" | 24h {total - prev:+.2f} USDT" if prev is not None else " | no prior snapshot"
+        notify("📈", "EQUITY",
+               f"total {money(total)} USDT{delta}",
+               " | ".join(parts) or "all in USDT")
+        try:
+            with open(state_file("equity_state.json"), "w",
+                      encoding="utf-8") as f:
+                json.dump({"equity": str(total)}, f)
+        except OSError:
+            pass
+    except Exception as e:
+        print("equity report failed:", repr(e), flush=True)
+
+
+async def run_books_ws(c, ids, stop):
+    """Feed WS_BOOKS continuously, one subscription per pair; REST stays the
+    fallback when a stream goes quiet. ids maps symbol -> symbolId."""
+    async def keep_one(sym, symbol_id):
+        while not stop.is_set():
+            try:
+                sub = await c.orderbook.subscribe(symbol=sym,
+                                                  symbol_id=symbol_id, depth=10)
+                async for data in sub:
+                    if getattr(data, "available", True):
+                        WS_BOOKS[sym] = (data, time.monotonic())
+                    if stop.is_set():
+                        break
+            except Exception as e:
+                print("ws book feed", sym, "down:", repr(e), flush=True)
+            try:
+                await asyncio.wait_for(stop.wait(), 5.0)
+            except asyncio.TimeoutError:
+                pass
+    await asyncio.gather(*(keep_one(s, i) for s, i in ids.items()))
+
+
 async def scan_books(c, pairs):
     """Every USDT pair with a two-sided book: (gap_pct, sym, bid, ask, bid$, ask$)."""
     rows = []
     for sym in pairs:
         try:
-            book = await c.orderbook.get(symbol=sym, depth=1)
-            if not book.bids or not book.asks:
+            bk = await book(c, sym, depth=1)
+            if not bk.bids or not bk.asks:
                 continue
-            bid, ask = pdec(book.bids[0].price), pdec(book.asks[0].price)
+            bid, ask = pdec(bk.bids[0].price), pdec(bk.asks[0].price)
             if bid <= 0 or ask <= bid:
                 continue
             gap = (ask - bid) / ((ask + bid) / 2) * 100
-            bid_usd = pdec(book.bids[0].qty) * bid
-            ask_usd = pdec(book.asks[0].qty) * ask
+            bid_usd = pdec(bk.bids[0].qty) * bid
+            ask_usd = pdec(bk.asks[0].qty) * ask
             rows.append((gap, sym, bid, ask, bid_usd, ask_usd))
         except Exception as e:
             print("scan skip", sym, repr(e), flush=True)
@@ -267,11 +615,11 @@ async def place_sell(c, pos, meta):
     sym = pos["symbol"]
     base = sym.split("-")[0]
     tick, step, min_not = meta[sym]["tick"], meta[sym]["step"], meta[sym]["min_notional"]
-    book = await c.orderbook.get(symbol=sym, depth=10)
-    if not book.asks:
+    bk = await book(c, sym, depth=10)
+    if not bk.asks:
         return False
-    best_ask = pdec(book.asks[0].price)
-    best_bid = pdec(book.bids[0].price) if book.bids else Decimal(0)
+    best_ask = pdec(bk.asks[0].price)
+    best_bid = pdec(bk.bids[0].price) if bk.bids else Decimal(0)
     spread = ((best_ask - best_bid) / ((best_ask + best_bid) / 2) * 100
               if best_bid else Decimal(0))
     basis = pos.get("buy_px") or Decimal(0)
@@ -295,6 +643,10 @@ async def place_sell(c, pos, meta):
         held = None
     if q <= 0 or q * best_ask < min_not:
         if held and held * best_ask >= min_not:
+            last = pos.get("orphan_noted")
+            if last is not None and time.time() - last < 3600:
+                return False
+            pos["orphan_noted"] = time.time()
             try:
                 await c.orders.cancel_all(symbol=sym,
                                           request_id="gap1-ca-" + uuid.uuid4().hex[:10])
@@ -409,6 +761,9 @@ async def main():
                     "min_qty": Decimal(p.get("minQtyBase") or "0"),
                     "scale": int(p.get("baseQuantityScale") or 8)}
                 for s, p in pairs.items()}
+        ws_stop = asyncio.Event()
+        ws_ids = {s: p["symbolId"] for s, p in pairs.items() if "symbolId" in p}
+        ws_task = asyncio.create_task(run_books_ws(c, ws_ids, ws_stop))
 
         async def resync(tag):
             """Cancel our stale orders, re-read balances.
@@ -457,6 +812,7 @@ async def main():
             return held, True
 
         adopt, boot_ok = await resync("gap1-boot")
+        await refresh_fees(c)
 
         notify("🤖", "GAP HUNTER ONLINE",
                f"gap ≥ {GAP_MIN_PCT}% | depth ≥ {money(DEPTH_MIN_USD)} USDT | "
@@ -472,6 +828,11 @@ async def main():
         next_scan = 0.0
         next_fallback = 0.0
         next_block_alert = 0.0
+        sat = {"n": 0}
+        next_fees = 0.0
+        fees_seen = False
+        last_deadman = 0.0
+        last_equity_day = ""
 
         vol, next_10k = None, None
         try:
@@ -517,9 +878,11 @@ async def main():
             except OSError:
                 pass
 
-        def cycle_done(sym, sell_px):
+        async def cycle_done(sym, sell_px):
             """Close a cycle from balance truth; alert with balances + volume."""
             nonlocal vol, next_10k, pos, state
+            await cancel_tp_trigger(c, pos)
+            await cancel_stale_symbol_orders(c, sym)
             base_sym = sym.split("-")[0]
             base_held, _ = rest_asset_balance(base_sym)
             qty0 = pos.get("qty0") or pos.get("qty") or Decimal(0)
@@ -528,7 +891,10 @@ async def main():
             basis = pos.get("buy_px") or Decimal(0)
             sell_usd = sell_px * sold
             buy_usd = basis * sold
-            fees = (sell_usd + buy_usd) * Decimal("0.001")
+            mk = maker_fee_pct(FEES, sym)
+            fees = round_trip_fees(buy_usd, sell_usd, mk)
+            if pos.get("buy_fee_pct") and pos["buy_fee_pct"] != mk:
+                fees += buy_usd * (pos["buy_fee_pct"] - mk) / 100
             net = sell_usd - buy_usd - fees
             if sell_px > 0:
                 net_txt = f"net {net:+.2f} USDT"
@@ -579,6 +945,8 @@ async def main():
                            "sell_cid": None, "better": {}, "since": now}
                     state = "selling"
                     carry[sym] = (pos["buy_px"], pos["qty0"])
+                    if sym in meta:
+                        await place_tp_trigger(c, pos, meta[sym])
                     notify("♻️", "ADOPT",
                            f"{sym} {num(qty)} | basis "
                            f"{num(pos['buy_px']) if pos['buy_px'] else 'unknown'}",
@@ -620,7 +988,10 @@ async def main():
                             if FALLBACK_SYMBOL:
                                 fb = [r for r in fb if r[1] == FALLBACK_SYMBOL]
                             if fb:
-                                picked, tag = fb[0], "VOLUME"
+                                if FALLBACK_SYMBOL:
+                                    picked, tag = fb[0], "VOLUME"
+                                else:
+                                    picked, tag = await pick_best_liquid(c, fb), "VOLUME"
                                 next_fallback = now + FALLBACK_COOLDOWN
                         if picked:
                             gap, sym, bid, ask, bid_usd, ask_usd = picked
@@ -660,35 +1031,56 @@ async def main():
                         if picked:
                             gap, sym, bid, ask, bid_usd, ask_usd = picked
                             m = meta[sym]
-                            price = bid if (tag == "VOLUME" and not FALLBACK_PENNY) \
-                                else min(bid + m["tick"], ask - m["tick"])
+                            if tag == "GAP":
+                                sat["n"] = 0
+                                penny, take = False, False
+                            else:
+                                penny = FALLBACK_PENNY or sat["n"] >= SAT_PENNY_AFTER
+                                take = sat["n"] >= SAT_TAKER_AFTER
+                            price = entry_price(tag, bid, ask, m, penny, take)
+                            buy_fee = (taker_fee_pct(FEES, sym) if take
+                                       else maker_fee_pct(FEES, sym))
+                            rt = maker_fee_pct(FEES, sym) + buy_fee
                             usdt = next((Decimal(r["available"]) for r in rest_balances()
                                          if r.get("symbol") == QUOTE), Decimal(0))
                             q = floor_step(usdt / (price * FEE_BUFFER), m["step"])
                             if q > 0 and q >= m["min_qty"] and q * price >= m["min_notional"]:
-                                cid, err = await place(c, sym, "buy", q, price)
+                                pv_ok, pv_why = await preflight_buy(c, sym, "buy", q, price)
+                                cid = None
+                                if not pv_ok:
+                                    if now >= next_block_alert:
+                                        next_block_alert = now + 600
+                                        notify("⚠️", "SKIPPED",
+                                               f"{sym} | {tag} | pre-flight: {pv_why}",
+                                               wallet(sym.split("-")[0]))
+                                else:
+                                    cid, err = await place(c, sym, "buy", q, price)
                                 if cid:
                                     pos = {"symbol": sym, "buy_cid": cid, "buy_px": price,
                                            "buy_at": now, "qty": None,
+                                           "qty_target": q, "tag": tag,
+                                           "buy_fee_pct": buy_fee,
                                            "sell_cid": None, "better": {},
                                            "join_ask": tag == "VOLUME"}
                                     state = "buying"
                                     if tag == "GAP":
                                         notify("🟢", "BUY",
                                                f"{sym} | gap {gap:.2f}% "
-                                               f"(net +{gap - RT_FEE_PCT:.2f}%) | "
+                                               f"(net +{gap - rt:.2f}%) | "
                                                f"{num(q)} @ {num(price)} | "
                                                f"depth {money(bid_usd)}/{money(ask_usd)}",
-                                               f"expected +{q * price * (gap - RT_FEE_PCT) / 100:.2f}"
+                                               f"expected +{q * price * (gap - rt) / 100:.2f}"
                                                f" USDT | {wallet(sym.split('-')[0])}")
                                     else:
-                                        loss = q * price * (RT_FEE_PCT - gap) / 100
+                                        loss = q * price * (rt - gap) / 100
+                                        stage = ("taker ask" if take else
+                                                 "penny bid" if penny else "bid")
                                         notify("🟡", "VOLUME BUY",
                                                f"{sym} | gap {gap:.3f}% "
-                                               f"(fee {RT_FEE_PCT}%) | "
+                                               f"(fee {rt}%) | "
                                                f"{num(q)} @ {num(price)} | "
                                                f"volume {money(q * price * 2)} USDT",
-                                               f"expected -{loss:.2f} USDT "
+                                               f"{stage} | expected -{loss:.2f} USDT "
                                                f"| {wallet(sym.split('-')[0])}")
                             else:
                                 notify("⚠️", "SKIPPED",
@@ -708,9 +1100,11 @@ async def main():
                             pos["buy_px"] = avg if avg > 0 else pos["buy_px"]
                             pos["since"] = now
                             carry[sym] = (pos["buy_px"], pos["qty0"])
+                            sat["n"] = 0
                             notify("✅", "BUY FILLED",
                                    f"{sym} | {num(pos['qty'])} @ {num(pos['buy_px'])}",
                                    wallet(sym.split("-")[0]))
+                            await place_tp_trigger(c, pos, m)
                             state = "selling"
                         else:
                             pos["qty"] = held_qty
@@ -718,31 +1112,38 @@ async def main():
                                    f"{sym} | order ended {st}",
                                    wallet(sym.split("-")[0]))
                             state = "idle"
-                    elif now - pos["buy_at"] > BUY_TIMEOUT_SECONDS:
-                        try:
-                            await c.orders.cancel(key=ClientOrderId(pos["buy_cid"]),
-                                                  symbol=sym)
-                        except Exception:
-                            pass
-                        st, cum, avg = await order_state(c, sym, pos["buy_cid"],
-                                                         m["scale"])
-                        notify("⚠️", "BUY TIMEOUT",
-                               f"{sym} | no fill after {BUY_TIMEOUT_SECONDS:.0f}s "
-                               f"| cancelled",
-                               wallet(sym.split("-")[0]))
-                        pos["qty"] = floor_step(await base_balance(c, sym, m), m["step"])
-                        if pos["qty"] > 0 or cum > 0:
-                            pos["qty"] = pos["qty"] or floor_step(cum, m["step"])
-                            pos["qty0"] = pos["qty"]
-                            pos["buy_px"] = avg if avg > 0 else pos["buy_px"]
-                            pos["since"] = now
-                            carry[sym] = (pos["buy_px"], pos["qty0"])
-                            notify("✅", "BUY FILLED",
-                                   f"{sym} | {num(pos['qty'])} @ {num(pos['buy_px'])}",
-                                   wallet(sym.split("-")[0]))
-                            state = "selling"
-                        else:
-                            state = "idle"
+                    else:
+                        await chase_buy(c, pos, m, now)
+                        tmo = buy_timeout_for(pos.get("tag", "GAP"), sat["n"])
+                        if now - pos["buy_at"] > tmo:
+                            try:
+                                await c.orders.cancel(key=ClientOrderId(pos["buy_cid"]),
+                                                      symbol=sym)
+                            except Exception:
+                                pass
+                            st, cum, avg = await order_state(c, sym, pos["buy_cid"],
+                                                             m["scale"])
+                            notify("⚠️", "BUY TIMEOUT",
+                                   f"{sym} | no fill after {tmo:.0f}s "
+                                   f"| cancelled",
+                                   f"sat {sat['n']} | {wallet(sym.split('-')[0])}")
+                            pos["qty"] = floor_step(await base_balance(c, sym, m), m["step"])
+                            if pos["qty"] > 0 or cum > 0:
+                                pos["qty"] = pos["qty"] or floor_step(cum, m["step"])
+                                pos["qty0"] = pos["qty"]
+                                pos["buy_px"] = avg if avg > 0 else pos["buy_px"]
+                                pos["since"] = now
+                                carry[sym] = (pos["buy_px"], pos["qty0"])
+                                sat["n"] = 0
+                                notify("✅", "BUY FILLED",
+                                       f"{sym} | {num(pos['qty'])} @ {num(pos['buy_px'])}",
+                                       wallet(sym.split("-")[0]))
+                                await place_tp_trigger(c, pos, m)
+                                state = "selling"
+                            else:
+                                state = "idle"
+                                if pos.get("tag") == "VOLUME":
+                                    sat["n"] += 1
 
                 elif state == "selling":
                     sym = pos["symbol"]
@@ -755,12 +1156,12 @@ async def main():
                         if now - pos.get("since", 0) < BALANCE_GRACE:
                             pass
                         else:
-                            cycle_done(sym, price_est)
+                            await cycle_done(sym, price_est)
                     elif held < m["min_qty"] or (price_est > 0 and
                                                held * price_est < m["min_notional"]):
                         if held > 0:
                             dust[sym] = held
-                        cycle_done(sym, price_est)
+                        await cycle_done(sym, price_est)
                     elif pos["sell_cid"] is None:
                         pos["qty"] = min(pos.get("qty") or held, held)
                         placed = await place_sell(c, pos, meta)
@@ -793,8 +1194,8 @@ async def main():
                                 pos["sell_cid"] = None
                                 pos["canceling"] = False
                         else:
-                            book = await c.orderbook.get(symbol=sym, depth=10)
-                            better = {pdec(a.price) for a in (book.asks or [])
+                            bk = await book(c, sym, depth=10)
+                            better = {pdec(a.price) for a in (bk.asks or [])
                                       if pdec(a.price) < pos["ask_px"]}
                             track = pos["better"]
                             for p in list(track):
@@ -804,22 +1205,67 @@ async def main():
                                 track.setdefault(p, now)
                             aged = sum(1 for p, t0 in track.items()
                                        if now - t0 >= PERSIST_SECONDS)
-                            if aged > BETTER_ASKS_LIMIT:
-                                try:
-                                    await c.orders.cancel(key=ClientOrderId(pos["sell_cid"]),
-                                                          symbol=sym)
-                                except Exception:
-                                    pass
-                                notify("⚠️", "ASK BAIL-OUT",
-                                       f"{sym} | {aged} better asks held "
-                                       f"{PERSIST_SECONDS:.0f}s | re-asking under them",
-                                       wallet(sym.split("-")[0]))
-                                pos["canceling"] = True
-                                pos["bail"] = True
+                            if aged >= BETTER_ASKS_LIMIT:
+                                asks2 = [pdec(a.price) for a in (bk.asks or [])
+                                         if pdec(a.price) > 0]
+                                new_px = min(asks2) - m["tick"] if asks2 else None
+                                amended = False
+                                if new_px is not None and new_px != pos["ask_px"]:
+                                    try:
+                                        await c.orders.modify(
+                                            key=ClientOrderId(pos["sell_cid"]),
+                                            symbol=sym, new_price=new_px)
+                                        pos["ask_px"] = new_px
+                                        pos["better"] = {}
+                                        pos["bail"] = True
+                                        notify("🔄", "ASK CHASE",
+                                               f"{sym} | {aged} better ask(s) held "
+                                               f"{PERSIST_SECONDS:.0f}s | "
+                                               f"amending to {num(new_px)}",
+                                               wallet(sym.split("-")[0]))
+                                        amended = True
+                                    except Exception as e:
+                                        print("amend fell back to cancel:", repr(e),
+                                              flush=True)
+                                if not amended and new_px != pos["ask_px"]:
+                                    try:
+                                        await c.orders.cancel(key=ClientOrderId(pos["sell_cid"]),
+                                                              symbol=sym)
+                                    except Exception:
+                                        pass
+                                    notify("⚠️", "ASK BAIL-OUT",
+                                           f"{sym} | {aged} better asks held "
+                                           f"{PERSIST_SECONDS:.0f}s | re-asking under them",
+                                           wallet(sym.split("-")[0]))
+                                    pos["canceling"] = True
+                                    pos["bail"] = True
+                if now >= next_fees:
+                    next_fees = now + FEES_REFRESH_SECONDS
+                    await refresh_fees(c)
+                    if not fees_seen and FEES:
+                        fees_seen = True
+                        s0 = next(iter(FEES))
+                        notify("💱", "FEES LIVE",
+                               f"VIP fee {FEES[s0][0]}% maker / "
+                               f"{FEES[s0][1]}% taker",
+                               f"gap floor {GAP_MIN_PCT}% | {len(FEES)} pairs "
+                               f"| {wallet()}")
+                if DEADMAN_TIMEOUT > 0 and now - last_deadman >= DEADMAN_REFRESH:
+                    last_deadman = now
+                    await deadman_rearm(c)
+                today = time.strftime("%Y-%m-%d", time.gmtime())
+                if today != last_equity_day:
+                    last_equity_day = today
+                    await equity_report(c)
             except Exception as e:
                 print("loop error:", repr(e), flush=True)
             await asyncio.sleep(POLL_SECONDS)
     finally:
+        ws_stop.set()
+        try:
+            await asyncio.wait_for(asyncio.gather(ws_task), 3)
+        except Exception:
+            ws_task.cancel()
         await c.aclose()
 
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "polyester-bot",
-  "version": "1.1.0",
+  "version": "1.2.0",
   "description": "Polyester testnet gap hunter: buys orderbook gaps, asks them back, alerts on Telegram",
   "keywords": [
     "polyester",
@@ -34,8 +34,7 @@
     "pyproject.toml",
     "bot/*.py",
     "LICENSE",
-    "README.md",
-    "polyester-bot.png"
+    "README.md"
   ],
   "engines": {
     "node": ">=14"

package/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "polyester-bot"
-version = "1.1.0"
+version = "1.2.0"
 description = "Polyester testnet gap hunter: buys orderbook gaps, asks them back, alerts on Telegram"
 readme = "README.md"
 requires-python = ">=3.11"