PyFT8 2.8.0__tar.gz → 2.10.0__tar.gz

{pyft8-2.8.0 → pyft8-2.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: PyFT8
-Version: 2.8.0
+Version: 2.10.0
 Summary: FT8 Decoding and Encoding in Python with test/loopback code
 Author-email: G1OJS <g1ojs@yahoo.com>
 License-Expression: GPL-3.0-or-later
@@ -43,6 +43,7 @@ If you're interested in how this works, maybe have a look at [MiniPyFT8](https:/
  - Messages overlaid on waterfall signals that produce them
  - Worked-before info and fine grid locators / distance and bearing in the message boxes
  - List of stations hearing your transmissions on the selected band
+ - Marker on message if 2nd call is hearing you
  - Band activity in your level 4 square live updated next to band select buttons
  - Number of remote stations hearing your Tx, number of remote Txs that you're hearing, plus the same info for the 'best' station in your level 4 square
  - Data used for the above is cached to disk so is not lost when restarting the program
@@ -79,12 +80,48 @@ Once installed, you can use the following commands to run it. Otherwise, please
 | Launch configured GUI|pyft8 -i "Keyword1, Keyword2" -o "Keyword1, Keyword2" -c {config folder}| Config folder stores PyFT8.ini (your callsign, grid, buttons) and PyFT8.adi log file. Run this once to create default PyFT8.ini file.|
 
 ### Rig control 
-I've included the Python code that I use with my Icom IC-7100 in the file 'rigctrl.py', and believe I've moved sufficient 'specification' for the rig protocol into the .ini file so that you can paste in your own rig specification (see for e.g. the Omnirig .ini file for your rig) and get it working with PyFT8 controlling PTT and frequency. I designed this code to drop the serial connection when it's not required, so that the rig's serial port can be accessed by other software at the same time.
+I've included the Python code that I use with my Icom IC-7100 in the file 'rigctrl.py', and believe I've moved sufficient 'specification' for the rig protocol into the .ini file so that you can paste in your own rig specification (see for e.g. the Omnirig .ini file for your rig) and get it working with PyFT8 controlling PTT and frequency. I designed this code to drop the serial connection when it's not required, so that the rig's serial port can be accessed by other software at the same time (assuming that the other software returns the favour and doesn't permanently hog the serial port).
 
 I've also included a basic Hamlib interface which launches rigctld and uses that to control the rig. To use this, make sure that the hamlib section of the ini file is populated; this will then take precedence over the direct CAT control section.
 
 Alternatively, you can run PyFT8 without rig control; if there is no rig found, PyFT8 defaults to running without a rig connected. In this case, you need to provide your own PTT method and note that the band buttons will only set the information used for logging QSOs to the PyFT8.adi file. Or you can use PyFT8 as Rx-only.
 
+## Customising
+Even if you don't want to edit the code, there's quite a bit that you can change in the .ini file. I haven't written a user guide yet, so here's my own .ini file for reference. I use CAT control so my hamlib block is disabled by adding a _ to the name of the block (I keep it there for testing only). I need to add some more switches really to catch up with recent additions.
+
+```
+[station]
+call = G1OJS
+grid = IO90******
+[gui]
+loc = km_deg
+wb = Y
+[rig]
+port = COM4
+baud_rate = 9600
+set_freq_command = FEFE88E0.05.0000000000.FD 
+set_freq_value = 5|5|vfBcdLU|1|0 
+ptt_on_command = FEFE88E0.1C00.01.FD
+ptt_off_command = FEFE88E0.1C00.00.FD
+[_hamlib_rig]
+rigctld = C:/WSJT/wsjtx/bin/rigctld-wsjtx
+port = COM4
+baud_rate = 9600
+model = 3070
+[pskreporter]
+upload = Y
+[bands]
+160m = 1.840
+80m = 3.573
+60m = 5.357
+40m = 7.074
+20m = 14.074
+15m = 21.074
+10m = 28.074
+6m = 50.313
+2m = 144.174
+```
+
 ## Performance Compared with FT8_lib and WSJT-x
 
 The image below shows the number of decodes from PyFT8, WSJT-x V2.7.0 running in NORM mode, and FT8_lib, using the same 10 minutes of busy 20m audio that is used to test ft8_lib. 

pyft8-2.10.0/PyFT8/databases.py

@@ -0,0 +1,311 @@
+from PyFT8.pskreporter import PSKR_MQTT_listener
+import threading, time, os, pickle
+
+call_hashes = {}
+def add_call_hashes(call):
+    global call_hashes
+    chars = " 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ/"
+    call_padded = (call + "          ")[:11]
+    hashes = []
+    for m in [10,12,22]:
+        x = 0
+        for c in call_padded:
+            x = 38*x + chars.find(c)
+            x = x & ((int(1) << 64) - 1)
+        x = x & ((1 << 64) - 1)
+        x = x * 47055833459
+        x = x & ((1 << 64) - 1)
+        x = x >> (64 - m)
+        hashes.append(x)
+        call_hashes[(x, m)] = call
+    return hashes
+
+def grid_to_latlong(grid, centre = True):
+    lat, lon = -90, -180
+    grid = grid.upper()
+    if centre:
+        grid = grid + "LL44LL44LL44"[len(grid):]
+    mults = [20, 2, 2/24, 0.2/24, 0.2/(24*24), 0.02/(24*24)]
+    grid = grid[:2*len(mults)]
+    pairs = [grid[i:i+2] for i in range(0,len(grid),2)]
+    for i, p in enumerate(pairs):
+        zero = [ord('A'),ord('0')][i % 2]
+        lon += mults[i] * (ord(p[0]) - zero)
+        lat += mults[i] * (ord(p[1]) - zero) / 2
+    return (lat, lon)
+        
+def grids_to_dist_brg(sq1, sq2, units):
+    from numpy import sin, cos, asin, atan2, sqrt, radians, degrees
+    ll1, ll2 = grid_to_latlong(sq1), grid_to_latlong(sq2)
+    lats = [radians(ll1[0]), radians(ll2[0])]
+    dlat, dlon = radians(ll2[0] - ll1[0]), radians(ll2[1] - ll1[1])
+    s_lats, c_lats = sin(lats), cos(lats)
+    a = sin(dlat/2)**2 + c_lats[0] * c_lats[1] * sin(dlon/2)**2
+    r = 6371 * 2 * asin(sqrt(a))
+    b = atan2(c_lats[1] * sin(dlon), c_lats[0] * s_lats[1] - s_lats[0] * c_lats[1] * cos(dlon))
+    b *= (1.0 if 'km' in units else 0.621371)
+    return (r, degrees(b) % 360)
+
+class DiskDict:
+    def __init__(self, file):
+        self.lock = threading.Lock()
+        self.file = file
+        self.data = {}
+        self.load()
+        threading.Thread(target = self._autosave, daemon = True).start()
+
+    def _autosave(self, autosave_period = 15):
+        while True:
+            time.sleep(autosave_period)
+            self.save()
+
+    def load(self):
+        with self.lock:        
+            if(os.path.exists(self.file)):
+                with open(f"{self.file}","rb") as f:
+                    self.data = pickle.load(f)
+
+    def save(self):
+        with self.lock:
+            tmp_file = f"{self.file}.tmp"
+            with open(tmp_file, "wb") as f:
+                pickle.dump(self.data, f)
+                f.flush()
+                os.fsync(f.fileno())
+            os.replace(tmp_file, self.file)
+
+class History:
+    def __init__(self, config_folder, my_call, home_square, pskr_refresh_mins, parse_all_file):
+        self.pskr_refresh_mins = pskr_refresh_mins
+        self.config_folder = config_folder
+        self.my_call = my_call
+        self.home_square = home_square
+        self.home_square_lev4 = home_square[:4]
+        self.dist_brg_cache = {}
+        self.hearing_me = DiskDict(f"{self.config_folder}/hearing_me.pkl")   # all-time record of hearing me
+        self.heard_by_me = DiskDict(f"{self.config_folder}/heard_by_me.pkl") # all-time record of heard by me
+        self.hearing_me_new = {}
+        self.heard_by_me_new = {}
+        self.call_to_grid = DiskDict(f"{self.config_folder}/call_to_grid.pkl") # all time cache call -> fine locator
+        self.band_TxRx_homecall_report_times = DiskDict(f"{self.config_folder}/report_times.pkl") # last 20 mins data -> per band tx/rx & current band detail
+        self.home_activity = {}
+        self.home_most_remotes = {}
+        self.lock = threading.Lock()
+        if parse_all_file:
+            self.load_all_file(f"{self.config_folder}/ALL.txt")
+            self.hearing_me.save()
+            self.heard_by_me.save()
+            print("All file parsed and saved to hearing_me / heard_by_me files")
+        else:
+            mqtt = PSKR_MQTT_listener(self.home_square_lev4, self.add_mqtt_spot)
+            threading.Thread(target = self.count_activity, daemon = True).start()
+
+    def band_from_MHz(self, fMHz): # rewrite this to use the band button defs from ini file
+        f = int(fMHz)
+        if f > 0:
+            freqs = [1,3,5,7,10,14,18,21,24,28,50,144,433]
+            idx = freqs.index(f) if f in freqs else -1
+            if idx > -1:
+                return ['160m','80m','60m','40m','30m','20m','17m','15m','12m','10m','6m','2m','70cm'][idx]
+
+    def load_from_wb(self, log_cache):
+        for key in log_cache:
+            key_parts = key.split('_')
+            if len(key_parts) > 1:
+                c, b, m = key_parts
+                if m == 'FT8':
+                    self.add_myspots_record(self.hearing_me.data, None, b, c, 0, 0)
+                    self.add_myspots_record(self.heard_by_me.data, None, b, c, 0, 0)
+
+    def load_all_file(self, all_file):
+        recs = self.parse_all_txt(all_file)
+        if not any(recs): return
+        for r in recs:
+            if r['md'] == 'FT8':
+                band = self.band_from_MHz(r['fMHz'])
+                if band is not None:
+                    TxRx = 'Tx' if (r['TxRx'] == 'Tx' or r['call_b'] == self.my_call) else 'Rx'
+                    call = r['call_b'] if TxRx == 'Rx' else r['call_a']
+                    if len(call) > 3:
+                        data = self.heard_by_me.data if TxRx == 'Rx' else self.hearing_me.data
+                        self.add_myspots_record(data, None, band, call, 0, 0)
+
+    def parse_all_txt(self, all_file):
+        rows, recs = None, []
+        if os.path.exists(all_file):
+            with open(all_file, 'r') as f:
+                rows = f.readlines()
+        if rows is not None:
+            for r in rows:
+                fields = r.strip().split()
+                if len(fields) > 8:
+                    call_a_pos = 7 if fields[7] != '~' else 8
+                    recs.append({'fMHz':float(fields[1]), 'TxRx':fields[2], 'md':fields[3],
+                                 'call_a':fields[call_a_pos], 'call_b':fields[call_a_pos + 1]} )
+        if not any(recs):
+            print("Didn't find any records in an ALL.txt file in the config folder")
+        return recs
+
+    def write_all_txt_row(self, cyclestart_string, fMHz, TxRx, mode, snr, dt, fHz, msg):
+        all_file = f"{self.config_folder}/ALL.txt"
+        filemode = 'w' if not os.path.exists(all_file) else 'a'
+        row = f"{cyclestart_string} {fMHz:8.3f} {TxRx} {mode} {snr:+03d} {dt:4.1f} {fHz:4.0f} {msg}"
+        with open(all_file, filemode) as f:
+            f.write(f"{row}\n")
+
+    def add_mqtt_spot(self, d):
+        tnow = int(time.time())
+        sc, rc = (d['sc'], d['sl']), (d['rc'], d['rl'])
+        for iTxRx, call_grid in enumerate([sc, rc]):
+            call, grid = call_grid
+            self.store_best_grid(call, grid)
+            if self.home_square_lev4 in grid:
+                self.add_homespots_record((d['b'], iTxRx, call), tnow)
+        if d['sc'] == self.my_call:
+            self.add_myspots_record(self.hearing_me.data, self.hearing_me_new, d['b'], d['rc'], tnow, d['rp'])
+        if d['rc'] == self.my_call:
+            self.add_myspots_record(self.heard_by_me.data, self.heard_by_me_new, d['b'], d['sc'], tnow, d['rp'])
+
+    def store_best_grid(self, call, grid):
+        if call.startswith('<'): return
+        existing_grid = self.call_to_grid.data.get(call, '')
+        if len(grid) > len(existing_grid):
+            self.call_to_grid.data[call] = grid
+        
+    def add_homespots_record(self, key, t):
+        self.band_TxRx_homecall_report_times.data.setdefault(key, [])
+        self.band_TxRx_homecall_report_times.data[key].append(int(t))
+
+    def add_myspots_record(self, historic_data, new_alert_data, band, call, t, rp):
+        self._update_new_alert(band, call, historic_data, new_alert_data)
+        historic_data.setdefault(band, {})
+        if call in historic_data[band]:
+            if t < historic_data[band][call]['t']:
+                return
+        historic_data[band][call] = {'t': int(t),'rp':int(rp)}
+
+    def _update_new_alert(self, band, call, historic_data, new_alert_data):
+        if new_alert_data is None:
+            return
+        new = band not in historic_data
+        if not new:
+            new = call not in historic_data[band]
+        if new:
+            new_alert_data.setdefault(band, [])
+            new_alert_data[band].append(call)
+
+    def is_in_new_alert(self, band, call, new_alert_data):
+        result = False
+        if band in new_alert_data:
+            result = call in new_alert_data[band]
+        return result
+
+    def is_hearing_me(self, band, call, since_seconds):
+        result = False
+        if band in self.hearing_me.data:
+            if call in self.hearing_me.data[band]:
+                result = int(self.hearing_me.data[band][call]['t']) > since_seconds
+        return result        
+      
+    def count_activity(self):
+        import numpy as np
+        while True:
+            time.sleep(5)
+            self.home_activity = {}
+            self.home_most_remotes = {}
+            with self.lock:
+                # clear counters for each band
+                for b in self.home_activity:
+                    self.home_activity[b] = [0, 0]
+                for b in self.home_most_remotes:
+                    self.home_most_remotes[b] = [('',0), ('',0)]
+
+                # keep only the remote spots that happened in the self.pskr_refresh_mins window
+                for band_TxRx_homecall in self.band_TxRx_homecall_report_times.data:
+                    band_TxRx_homecall_report_times = self.band_TxRx_homecall_report_times.data[band_TxRx_homecall]
+                    band_TxRx_homecall_report_times = [t for t in band_TxRx_homecall_report_times if (time.time() - t) < 60*self.pskr_refresh_mins]
+                    self.band_TxRx_homecall_report_times.data[band_TxRx_homecall] = band_TxRx_homecall_report_times
+
+                # count number of local Tx and Rx, and identify the local Tx and Rx with most remote spots
+                for band_TxRx_homecall in self.band_TxRx_homecall_report_times.data:
+                    band_TxRx_homecall_report_times = self.band_TxRx_homecall_report_times.data[band_TxRx_homecall]
+                    if len(band_TxRx_homecall_report_times):
+                        b, iTxRx, c = band_TxRx_homecall
+                        self.home_activity.setdefault(b, [0, 0])
+                        self.home_activity[b][iTxRx] +=1
+                        self.home_most_remotes.setdefault(b, [('',0), ('',0)])
+                        nremotes = len(band_TxRx_homecall_report_times)
+                        current_winner = self.home_most_remotes[b][iTxRx]
+                        if nremotes > current_winner[1]:
+                            if c != self.my_call:
+                                self.home_most_remotes[b][iTxRx] = (c, nremotes)
+
+    def get_spot_counts(self, band, call):
+        tx_reports = self.band_TxRx_homecall_report_times.data.get((band, 0, call), [])
+        rx_reports = self.band_TxRx_homecall_report_times.data.get((band, 1, call), [])
+        n_spotting = len(tx_reports) if tx_reports else 0
+        n_spotted = len(rx_reports) if rx_reports else 0
+        return n_spotted, n_spotting
+
+    def get_dist_brg(self, grid, units):
+        self.dist_brg_cache.setdefault(grid, grids_to_dist_brg(self.home_square, grid, units))
+        return self.dist_brg_cache[grid]
+
+    def get_geo_text(self, call, units):
+        geo_text = ''
+        grid = self.call_to_grid.data.get(call, False)
+        if grid:
+            loc = grid if units == 'grid' else self.get_dist_brg(grid, units)
+            units_str = '' if units == 'grid' else ('km' if 'km' in units else 'mi')
+            geo_text = f"{int(loc[0]):5d}{units_str} {int(loc[1]):3d}°"
+        return geo_text
+                
+def ensure_file_exists(path, header = None):
+    try:
+        with open(path, "x") as f:
+            if header is not None:
+                f.write(header)
+    except FileExistsError:
+        pass
+      
+class ADIF:
+    def __init__(self, logfile):
+        self.adif_log_file = logfile
+        ensure_file_exists(self.adif_log_file, header = "header <eoh>\n")
+        self.cache = self._build_cache()
+              
+    def log(self, times, band_info, mStation, oStation, rpts):
+        log_dict = {'call':oStation['c'], 'gridsquare':oStation['g'], 'mode':'FT8',
+        'operator':mStation['c'], 'station_callsign':mStation['c'], 'my_gridsquare':mStation['g'], 
+        'rst_sent':rpts['sent'], 'rst_rcvd':rpts['rcvd'], 
+        'qso_date':time.strftime("%Y%m%d", times['time_on']), 'qso_date_off':time.strftime("%Y%m%d", times['time_off']),
+        'time_on':time.strftime("%H%M%S", times['time_on']), 'time_off':time.strftime("%H%M%S", times['time_on']),
+        'band':band_info['b'], 'freq':band_info['fMHz']}
+        with open(self.adif_log_file,'a') as f:
+            for k, v in log_dict.items():
+                v = str(v)
+                f.write(f"<{k}:{len(v)}>{v} ")
+            f.write(f"<eor>\n")
+        cbm = log_dict['call'] + "_" + log_dict['band'] + "_FT8"
+        tm = time.time()
+        self.cache[log_dict['call']] = tm
+        self.cache[cbm] = tm
+
+    def _build_cache(self):
+        import calendar
+        def parse(rec, field):
+            p = rec.find(field)
+            if p > 0:
+                p1, p2 = rec.find(':',p), rec.find('>',p)
+                n = int(rec[p1+1:p2])
+                return rec[p2+1: p2+1+n]
+        cache = {}
+        with open(self.adif_log_file, 'r') as f:
+            for l in f.readlines():
+                if parse(l, 'mode') == "FT8":
+                    c, b, d, t = parse(l, 'call'), parse(l, 'band'), parse(l, 'qso_date'), parse(l, 'time_on')
+                    time_tuple = time.strptime(d+t, "%Y%m%d%H%M%S")
+                    tm = calendar.timegm(time_tuple)
+                    cache[c] = tm
+                    cache[c + "_"+b+"_FT8"] = tm
+        return cache

{pyft8-2.8.0 → pyft8-2.10.0}/PyFT8/gui.py

@@ -14,7 +14,7 @@ HOVERCOLOR = 'darkgreen'
 ACTIVE_BUTTON_COLOR = 'cyan'
 INACTIVE_BUTTON_COLOR = '#edeef0'
 MAX_FONT_SIZE_MAIN = 10
-L = {'pmargin':0.04, 'sidebar_width': 0.16, 'banner_height':0.1, 'vsep1':0.01, 'hsep1':0.02}
+L = {'pmargin':0.04, 'sidebar_width': 0.17, 'banner_height':0.1, 'vsep1':0.01, 'hsep1':0.02}
 
 # ================== WATERFALL ======================================================
 
@@ -31,7 +31,7 @@ class Scrollbox:
         self.lineartists = []
         for i in range(self.nlines):
             self.lineartists.append(self.ax.text(0.03,1 - self.line_height * (i+1),
-                            '', color = MAIN_TEXT_COLOR, fontsize = self.fontsize))
+                            '', color = MAIN_TEXT_COLOR, fontsize = self.fontsize, clip_on = True))
             if monospace:
                 self.lineartists[-1].set_fontfamily('monospace')
         self.ax.set_xticks([])
@@ -67,7 +67,7 @@ class Msg_box:
         self.onclick = onclick 
         rect = Rectangle((tbin, fbin), width=w, height=h, alpha=0.6, edgecolor='lime', lw=2)
         self.patch = ax.add_patch(rect)
-        self.text_inst = ax.text(tbin, fbin+2, '', fontsize='small', fontweight='bold' )
+        self.text_inst = ax.text(tbin, fbin+2, '', fontsize='small', fontweight='bold', clip_on = True )
         self.cid = fig.canvas.mpl_connect('button_press_event', self._onclick)
         self.expire = 0 
 
@@ -109,7 +109,7 @@ class ButtonBox:
         self.info_axs.set_xticks([])
         self.info_axs.set_yticks([])
         self.info_axs.set_facecolor(TEXT_BACKGROUND_COLOR)
-        self.label2 = self.info_axs.text(0.03, 0.5, '', color = INFO_TEXT_COLOR, verticalalignment = 'center')
+        self.label2 = self.info_axs.text(0.03, 0.5, '', color = INFO_TEXT_COLOR, verticalalignment = 'center', clip_on = True)
         self.btn_widg = Button(self.btn_axs, btn_label, color = BUTTONCOLOR, hovercolor = HOVERCOLOR)
         self.label = self.btn_widg.label
         self.label.set_color(MAIN_TEXT_COLOR)

pyft8-2.8.0/PyFT8/pskr_upload.py → pyft8-2.10.0/PyFT8/pskreporter.py

@@ -1,11 +1,36 @@
-import socket
-import struct
-import time
-import threading
-import random
+import paho.mqtt.client as mqtt
+from ast import literal_eval
+import socket, struct, random
+import threading, time
 
 MAX_REPORTS = 90
 
+class PSKR_MQTT_listener:
+    def __init__(self, home_square, on_spot):
+        self.home_square = home_square
+        self.on_spot = on_spot
+        mqttc = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
+        mqttc.on_connect = self.on_connect
+        mqttc.on_message = self.on_message
+        try:
+            mqttc.connect("mqtt.pskreporter.info", 1883, 60)
+        except:
+            print("[MQTT] connection error")
+        threading.Thread(target = mqttc.loop_forever, daemon = True).start()
+
+    def on_connect(self, client, userdata, flags, reason_code, properties):
+        #pskr/filter/v2/{band}/{mode}/{sendercall}/{receivercall}/{senderlocator}/{receiverlocator}/{sendercouniTxRxy}/{receivercouniTxRxy}
+        print(f"[MQTT] Requesting mqtt feed for {self.home_square}")
+        client.subscribe(f"pskr/filter/v2/+/FT8/+/+/{self.home_square}/#")
+        client.subscribe(f"pskr/filter/v2/+/FT8/+/+/+/{self.home_square}/#")
+
+    def on_message(self, client, userdata, msg):
+        try:
+            d = literal_eval(msg.payload.decode())
+        except:
+            return
+        self.on_spot(d)
+
 class PSKR_upload:
     # https://pskreporter.info/pskdev.html
     # https://pskreporter.info/cgi-bin/psk-analysis.pl