pmemstat 2.3.22__tar.gz → 2.3.23__tar.gz

{pmemstat-2.3.22 → pmemstat-2.3.23}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pmemstat
-Version: 2.3.22
+Version: 2.3.23
 Summary: Report Proportional Memory on Linux
 Project-URL: Homepage, https://github.com/joedefen/pmemstat
 Project-URL: Bug Tracker, https://github.com/joedefen/pmemstat/issues
@@ -17,15 +17,15 @@ Requires-Dist: requests
 Description-Content-Type: text/markdown
 
 > **Quick Start**: from the CLI
-> * **If `python3 -V` shows v3.11 or later, install using `pipx`**:
->   * `python3 -m pip install --user pipx # if pipx not installed`
->   * `python3 -m pipx ensurepath # if needed (restart terminal)`
->   * `pipx upgrade pmemstat || pipx install pmemstat # to install/upgrade`
-> * **Else for python3.10 and lesser versions, install using `pip`**:
->   * `python3 -m pip install --user --upgrade pmemstat`
+> * **If `python3 -V` shows v3.11 or later, install with `pipx`**:
+>   * install `pipx` per your distro's guidance
+>   * then: `pipx upgrade pmemstat || pipx install pmemstat`
+> * **Else for python3.10 and lesser versions, install with `pip`**:
+>   * install `pip` per your distro's guidance
+>   * then: `python3 -m pip install --user --upgrade pmemstat`
 > * **To run**:
->   * `pmemstat # to run`
->   * Type "?" within pmemstat, show help screen.
+>   * simply run: `pmemstat`
+>   * type "?" within `pmemstat` to show help screen.
 
 
 # pmemstat - Proportional Memory Status
@@ -106,26 +106,7 @@ Explanation of some options and arguments:
 
 
 ## Example Usage with Explanation of Output
-```
-10:38:17 Tot=31.3G Avail=21.8G Oth=2.4G Tmp=326.7M Dirty=1.2M PIDs: 202/202
- cpu_pct   pswap   other    data  ptotal   key/info (exe by mem)
-    37.9       0   1,670   5,193   6,862 T 202x --TOTALS in MB --
-──────────────────────────────────────────────────────────────────────────────
-     2.1       0     452   1,830   2,282   41x chrome
-    18.5       0       2     827     830   2x python->main.py
-     0.1       0      87     470     557   8x code
-     0.7       0      63     393     456   1x plasmashell
-     0.7       0     106     281     389   7x Code
-     0.2       0      52     176     227   2x app
-     0.0       0      17     200     217   1x DiscoverNotifier
-     0.0       0      80     114     194   1x app.asar
-     0.0       0      94      76     170   3x signal-desktop
-     0.0       0       4     160     165   2x python->lsp_server.py
-     2.9       0      29     101     129   1x kwin_x11
-     0.0       0      94       9     102   2x Signal
-    12.8       0     590     555   1,145 O 131x ---- OTHERS ----
-
-```
+![pmemstat](https://github.com/joedefen/pmemstat/blob/main/images/pmemstat.png?raw=true)
 
 In the default refreshed window loop, we see
 * a **leader line** with:
@@ -136,7 +117,14 @@ In the default refreshed window loop, we see
        * Features such as BTRFS, ZFS, zRAM, unattached SysV Shared Memory, etc., can cause 'Oth' to be significant.
        * Determining the contributors can be difficult, but start with `sudo slabtop -sc` and feature specific tools (e.g., `zpool list`).
     * how many PIDs are contributing to the report vs the total number of PIDs excluding kernel threads
+* a **second leader line for zRAM** only if zRAM is active with:
+    * percent cpu consumed by kernel processes (not normalized ... if there 4 CPUs, then the total CPU can 400%). This is important because, with zRAM, this often is mostly the swap process.
+    * **zRAM** - the amount of "Used" RAM that is storing zRAM data
+    * **eTot** - the effective Total RAM and the percent of "Tot" RAM (which would typically be above 100%). The bigger **zRAM**, the more accurate this number.
+    * **eUsed** - the effective Used RAM and percent of "Tot" RAM.
+    * **eAvail** - the effective Available RAM and percent of "Tot" RAM.
 * a **header line with the reported fields** including:
+    * **cpu** - percent CPU (again not normalized to 100%)
     * **pswap** - proportional use of swap (per smaps_rollup)
     * **other** - partly sums these categories (shown by the key, `o`):
       * **shSYSV** - proportional use of System V shared memory (per smaps)
@@ -144,8 +132,7 @@ In the default refreshed window loop, we see
       * **stack** - exclusive use of stack memory per smaps
       * **text** - proportional use of memory for text (i.e., the read-only binary code, per smaps)
     * **data** - exclusive use of memory for data (i.e., exclusively used of "heap" memory, per smaps)
-    * **ptotal** - proportional use of memory of all categories (i.e, sum of all columns to the left except **pswap**, per smaps)
-    * (sometimes) **pss** - proportional use of memory (per smaps_rollup)
+    * **ptotal** - proportional use of memory of all categories (i.e, sum of columns to the left); **note**, if zRAM, this includes **pswap**, else not.
     * *empty* - type of the entry which may be:
         * **T** - the grand total
         * **A** - a newly added grouping
@@ -158,7 +145,7 @@ In the default refreshed window loop, we see
 ## Help Screen (in Window Mode, Press '?')
 In window mode, press '?' to enter the help screen which looks like:
 
-![helpscreen example](https://github.com/joedefen/pmemstat/blob/main/images/help-screen_2023-05-15.png?raw=true)
+![helpscreen example](https://github.com/joedefen/pmemstat/blob/main/images/help-screen.png?raw=true)
 
 **Notes:**
 * There are a number of navigation keys (mostly following vim conventions); in the help screen, they apply to help screen; otherwise, they apply to main screen.
@@ -171,7 +158,7 @@ Pressing "K" enter "Kill Mode" where you use the navigation keys to highlight a
 ## Scroll Position (Window Mode)
 
 Sometimes, the horizontal line between the header and scrollable region has a reverse video block (under the "351" in this case):
-![scroll-pos example](https://github.com/joedefen/pmemstat/blob/main/images/scroll-pos_2023-09-05.png?raw=true)
+![scroll-pos example](https://github.com/joedefen/pmemstat/blob/main/images/scroll-pos.png?raw=true)
 
 **Notes:**
 * When there is no block, the scrolled document does not overflow the scrollable region.

{pmemstat-2.3.22 → pmemstat-2.3.23}/README.md

@@ -1,13 +1,13 @@
 > **Quick Start**: from the CLI
-> * **If `python3 -V` shows v3.11 or later, install using `pipx`**:
->   * `python3 -m pip install --user pipx # if pipx not installed`
->   * `python3 -m pipx ensurepath # if needed (restart terminal)`
->   * `pipx upgrade pmemstat || pipx install pmemstat # to install/upgrade`
-> * **Else for python3.10 and lesser versions, install using `pip`**:
->   * `python3 -m pip install --user --upgrade pmemstat`
+> * **If `python3 -V` shows v3.11 or later, install with `pipx`**:
+>   * install `pipx` per your distro's guidance
+>   * then: `pipx upgrade pmemstat || pipx install pmemstat`
+> * **Else for python3.10 and lesser versions, install with `pip`**:
+>   * install `pip` per your distro's guidance
+>   * then: `python3 -m pip install --user --upgrade pmemstat`
 > * **To run**:
->   * `pmemstat # to run`
->   * Type "?" within pmemstat, show help screen.
+>   * simply run: `pmemstat`
+>   * type "?" within `pmemstat` to show help screen.
 
 
 # pmemstat - Proportional Memory Status
@@ -88,26 +88,7 @@ Explanation of some options and arguments:
 
 
 ## Example Usage with Explanation of Output
-```
-10:38:17 Tot=31.3G Avail=21.8G Oth=2.4G Tmp=326.7M Dirty=1.2M PIDs: 202/202
- cpu_pct   pswap   other    data  ptotal   key/info (exe by mem)
-    37.9       0   1,670   5,193   6,862 T 202x --TOTALS in MB --
-──────────────────────────────────────────────────────────────────────────────
-     2.1       0     452   1,830   2,282   41x chrome
-    18.5       0       2     827     830   2x python->main.py
-     0.1       0      87     470     557   8x code
-     0.7       0      63     393     456   1x plasmashell
-     0.7       0     106     281     389   7x Code
-     0.2       0      52     176     227   2x app
-     0.0       0      17     200     217   1x DiscoverNotifier
-     0.0       0      80     114     194   1x app.asar
-     0.0       0      94      76     170   3x signal-desktop
-     0.0       0       4     160     165   2x python->lsp_server.py
-     2.9       0      29     101     129   1x kwin_x11
-     0.0       0      94       9     102   2x Signal
-    12.8       0     590     555   1,145 O 131x ---- OTHERS ----
-
-```
+![pmemstat](https://github.com/joedefen/pmemstat/blob/main/images/pmemstat.png?raw=true)
 
 In the default refreshed window loop, we see
 * a **leader line** with:
@@ -118,7 +99,14 @@ In the default refreshed window loop, we see
        * Features such as BTRFS, ZFS, zRAM, unattached SysV Shared Memory, etc., can cause 'Oth' to be significant.
        * Determining the contributors can be difficult, but start with `sudo slabtop -sc` and feature specific tools (e.g., `zpool list`).
     * how many PIDs are contributing to the report vs the total number of PIDs excluding kernel threads
+* a **second leader line for zRAM** only if zRAM is active with:
+    * percent cpu consumed by kernel processes (not normalized ... if there 4 CPUs, then the total CPU can 400%). This is important because, with zRAM, this often is mostly the swap process.
+    * **zRAM** - the amount of "Used" RAM that is storing zRAM data
+    * **eTot** - the effective Total RAM and the percent of "Tot" RAM (which would typically be above 100%). The bigger **zRAM**, the more accurate this number.
+    * **eUsed** - the effective Used RAM and percent of "Tot" RAM.
+    * **eAvail** - the effective Available RAM and percent of "Tot" RAM.
 * a **header line with the reported fields** including:
+    * **cpu** - percent CPU (again not normalized to 100%)
     * **pswap** - proportional use of swap (per smaps_rollup)
     * **other** - partly sums these categories (shown by the key, `o`):
       * **shSYSV** - proportional use of System V shared memory (per smaps)
@@ -126,8 +114,7 @@ In the default refreshed window loop, we see
       * **stack** - exclusive use of stack memory per smaps
       * **text** - proportional use of memory for text (i.e., the read-only binary code, per smaps)
     * **data** - exclusive use of memory for data (i.e., exclusively used of "heap" memory, per smaps)
-    * **ptotal** - proportional use of memory of all categories (i.e, sum of all columns to the left except **pswap**, per smaps)
-    * (sometimes) **pss** - proportional use of memory (per smaps_rollup)
+    * **ptotal** - proportional use of memory of all categories (i.e, sum of columns to the left); **note**, if zRAM, this includes **pswap**, else not.
     * *empty* - type of the entry which may be:
         * **T** - the grand total
         * **A** - a newly added grouping
@@ -140,7 +127,7 @@ In the default refreshed window loop, we see
 ## Help Screen (in Window Mode, Press '?')
 In window mode, press '?' to enter the help screen which looks like:
 
-![helpscreen example](https://github.com/joedefen/pmemstat/blob/main/images/help-screen_2023-05-15.png?raw=true)
+![helpscreen example](https://github.com/joedefen/pmemstat/blob/main/images/help-screen.png?raw=true)
 
 **Notes:**
 * There are a number of navigation keys (mostly following vim conventions); in the help screen, they apply to help screen; otherwise, they apply to main screen.
@@ -153,7 +140,7 @@ Pressing "K" enter "Kill Mode" where you use the navigation keys to highlight a
 ## Scroll Position (Window Mode)
 
 Sometimes, the horizontal line between the header and scrollable region has a reverse video block (under the "351" in this case):
-![scroll-pos example](https://github.com/joedefen/pmemstat/blob/main/images/scroll-pos_2023-09-05.png?raw=true)
+![scroll-pos example](https://github.com/joedefen/pmemstat/blob/main/images/scroll-pos.png?raw=true)
 
 **Notes:**
 * When there is no block, the scrolled document does not overflow the scrollable region.

pmemstat-2.3.23/pmemstat/CpuSmooth.py

@@ -0,0 +1,337 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Class for fetching CPU usage from processes.
+"""
+# pylint: disable=invalid-name,broad-exception-caught,too-many-instance-attributes
+# pylint: disable=import-outside-toplevel,too-many-branches,too-many-locals
+# pylint: disable=consider-using-with,too-many-statements
+import os
+import time
+import re
+import traceback
+import math
+from types import SimpleNamespace
+
+class Term:
+    """ Escape sequences; e.g., see:
+     - https://en.wikipedia.org/wiki/ANSI_escape_code
+     - https://gist.github.com/fnky/458719343aabd01cfb17a3a4f7296797#file-ansi-md
+    """
+    esc = '\x1B'
+    # pylint: disable=missing-function-docstring,multiple-statements
+    @staticmethod
+    def erase_line(): return f'{Term.esc}[2K'
+    @staticmethod
+    def erase_to_eol(): return f'{Term.esc}[0K'
+    @staticmethod
+    def bold(): return f'{Term.esc}[1m'
+    @staticmethod
+    def reverse_video(): return f'{Term.esc}[7m'
+    @staticmethod
+    def normal_video(): return f'{Term.esc}[m'
+    @staticmethod
+    def pos_up(cnt): return f'{Term.esc}[{cnt}F' if cnt > 0 else ''
+    @staticmethod
+    def pos_down(cnt): return f'{Term.esc}[{cnt}E' if cnt > 0 else ''
+    @staticmethod
+    def col(pos): return f'{Term.esc}[{pos}G'
+    @staticmethod
+    def clear_screen(): return f'{Term.esc}[H{Term.esc}[2J{Term.esc}[3J'
+
+class SysStat:
+    """ TBD """
+    singleton = None
+    def __init__(self):
+        self.fh = None
+        self.prev = None
+        self.delta = None
+        self.clock_tick = 100
+        assert not self.singleton, 'cannot instantiate two SysStat'
+        SysStat.singleton = self
+        self._set_clock_tick()
+        self._refresh()
+
+    @staticmethod
+    def get_singleton():
+        """ Return THE SysStat object"""
+        return SysStat.singleton if SysStat.singleton else SysStat()
+
+    def _set_clock_tick(self):
+        if self.clock_tick is None:
+            try:
+                self.clock_tick = os.sysconf(os.sysconf_names['SC_CLK_TCK'])
+            except Exception:
+                pass
+            if self.clock_tick <= 0: # fake it
+                self.clock_tick = 100
+
+    @staticmethod
+    def refresh():
+        """ TBD """
+        return SysStat.get_singleton()._refresh()
+
+    def _refresh(self):
+        """ TBD """
+        if not self.fh:
+            pathname = '/proc/stat'
+            self.fh = open(pathname, "r", encoding='utf-8')
+        self.fh.seek(0)
+
+        ns = SimpleNamespace(mono=time.monotonic(),
+                    cpu_cnt=0, percent=0, ticks=0)
+        delta = SimpleNamespace(**vars(ns))
+
+        for line in self.fh:
+            wds = line.split()
+            keyword = wds[0]
+            if keyword == 'cpu':
+                ns.ticks = int(wds[1])
+                ns.ticks += int(wds[3])
+                # needed for VM to normalize the %cpu per process
+                ns.gross_ticks = sum(int(val) for val in wds[1:])
+            elif keyword.startswith('cpu'):
+                ns.cpu_cnt += 1
+
+        if self.prev:
+            prev = self.prev
+            delta.mono = round(ns.mono - prev.mono, 4)
+            delta.ticks = ns.ticks - prev.ticks
+            delta.cpu_cnt = ns.cpu_cnt
+            delta.max_ticks = math.ceil(self.clock_tick
+                                * delta.mono * ns.cpu_cnt)
+            delta.gross_ticks = ns.gross_ticks - prev.gross_ticks
+            if delta.mono > 0:
+                delta.percent = round(100
+                    * delta.ticks / self.clock_tick / delta.mono, 4)
+        self.prev = ns
+        self.delta = delta
+        return delta
+
+
+class CpuSmooth:
+    """Class that get smoothed CPU percent of given process"""
+    clock_tick = None  # number of clock ticks/sec
+    prev_system_stats = None
+    prev_system_delta = None
+
+    def __init__(self, pid, avg_secs=0, error=False, DB=False):
+        self.pid = pid
+        self.error = error # once an error stop beating head against wall
+        self.DB = DB
+        self.db_info = ''  # describe last update
+        self.avg_secs = avg_secs # smoothing interval
+        self.fh = None
+        self.stat_ns = None # last read status
+        self.percent = 0 # smoothed percent
+        self.hists = []
+        self.nickname = '' # crudely fetched on demand (for test)
+        self.sys_stat = SysStat.get_singleton()
+
+    def __del__(self):
+        if self.fh:
+            try:
+                self.fh.close()
+            except Exception:
+                pass
+
+    def _set_error(self):
+        if self.fh:
+            try:
+                self.fh.close()
+            except Exception:
+                pass
+            self.fh = None
+        self.percent, self.error = 0, True
+        return self.percent
+
+    def get_nickname(self):
+        """ Get the nickname of the process (crude)."""
+        if self.nickname:
+            return self.nickname
+        cmdline_file = f'/proc/{self.pid}/cmdline'
+        arguments, nickname = [], ''
+        try:
+            # pylint: disable=consider-using-with
+            with open(cmdline_file, encoding='utf-8') as fh:
+                for line in fh:
+                    arguments = line.split('\0')
+                    break
+        except Exception:
+            pass
+        if arguments:
+            wds = os.path.basename(arguments[0]).split() + arguments[1:]
+            nickname = re.sub(r'^\W+', '', wds.pop(0))
+            nickname = re.sub(r'\W+$', '', nickname)
+            # DB(0, f'basename={basename} wds={wds}')
+            if nickname in ('python', 'python2', 'python3', 'perl', 'bash', 'ruby',
+                    'sh', 'ksh', 'zsh') and wds:
+                script = os.path.basename(wds[0])
+                # DB(0, f'script={script} wds[0]={wds[0]}')
+                if script != wds[0]:
+                    nickname = f'{nickname}->{script}'
+        if not nickname:
+            ns = self.stat_ns if self.stat_ns else self._get_stat()
+            if ns and ns.exec:
+                nickname = ns.exec
+        self.nickname = nickname
+        return nickname
+
+    def _get_stat(self):
+        if not self.fh:
+            try:
+                # pylint: disable=consider-using-with
+                self.fh = open(f'/proc/{self.pid}/stat', encoding='utf-8')
+            except (PermissionError, FileNotFoundError):
+                return self._set_error()
+        try:
+            self.fh.seek(0)
+            data = self.fh.read().split()
+            self.stat_ns = SimpleNamespace(exec=data[1],
+                             user=int(data[13]), system=int(data[14]),
+                             nthr=int(data[19]))
+        except Exception:
+            self._set_error()
+        return self.stat_ns
+
+    def refresh_cpu(self):
+        """Get the Cpu Number for the PID (if possible)"""
+        def pct(hist0, hist1):
+            delta_ticks = abs(hist0[0] - hist1[0])
+            delta_mono = abs(hist0[1] - hist1[1])
+            delta_gross_ticks = abs(hist0[2] - hist1[2])
+
+            percent = 0
+            cpu_cnt = self.sys_stat.prev.cpu_cnt
+            if delta_gross_ticks > 0:
+                percent = cpu_cnt * 100 * delta_ticks / delta_gross_ticks
+
+            return percent, delta_ticks, delta_mono
+        def pct_str(triple):
+            return f'{triple[0]:7.3f}%,{triple[1]:5d},{triple[2]:7.4f}s'
+#       def adjust_down(delta_ticks):
+#           """ Lower current ticks by amount ... meaning add to all
+#               previous ticks"""
+#           for hist in self.hists[:-1]:
+#               hist[0] += delta_ticks
+
+        if self.error or not self._get_stat():
+            return self.percent
+        ticks = self.stat_ns.user + self.stat_ns.system
+        mono = time.monotonic()
+        gross_ticks = self.sys_stat.prev.gross_ticks
+        self.hists.append([ticks, mono, gross_ticks])
+
+        if len(self.hists) < 2: # takes two to tango
+            return 0
+        floor_mono = mono - self.avg_secs
+        while len(self.hists) > 2 and self.hists[0][1] < floor_mono:
+            self.hists.pop(0)
+        try:
+            _, _, delta_mono = pct(self.hists[-1], self.hists[-2])
+        except:
+            pass
+        if delta_mono <= 0.0:
+            self.hists.pop()
+            return 0
+
+        self.percent, _, _ = pct(self.hists[-1], self.hists[0])
+
+        # print(f'{self.percent}%')
+        if self.DB:
+            deltas = []
+            hists = self.hists
+            for idx in range(len(hists)-1, 0, -1):
+                hist, prev = hists[idx], hists[idx-1]
+                deltas.append(f'{hist[0]-prev[0]}'
+                               + f'/{hist[1]-prev[1]:.2f}'
+                               # + ('' if hist[2] <= 1 else f'#{hist[2]}')
+                               )
+            self.db_info = ' '
+            if len(hists) >= 2:
+                self.db_info = ' '.join([
+                      f'{self.get_nickname()[:16]:>16} {self.pid:>6d}',
+                      pct_str(pct(self.hists[-1], self.hists[-2])),
+                      '//', pct_str(pct(self.hists[-1], self.hists[0])),
+                        ' '.join(deltas)])
+        # if self.nickname == 'firefox':
+            # print(self.nickname, self.hists)
+        return self.percent
+
+if __name__ == '__main__':
+    def main():
+        """Main loop"""
+        import argparse
+        parser = argparse.ArgumentParser()
+        parser.add_argument('-t', '--top', type=int, default=10,
+                help='number of top CPU entries to show')
+        parser.add_argument('-o', '--only', type=str, default='',
+                help='name must have only substring')
+        parser.add_argument('-l', '--loop', type=float, default=1.0, dest='loop_secs',
+                        help='loop interval in secs [dflt=1.0]')
+        opts = parser.parse_args()
+
+        loop_secs = opts.loop_secs if opts.loop_secs >= 0.25 else 0.25
+        loop_secs = opts.loop_secs if opts.loop_secs <= 120.0 else 120.0
+
+        spots = [] # where to put the top items
+        cpus = {}
+        losers = set()
+        pids = set()
+        start_mono = time.monotonic()
+        sys_stat = SysStat.get_singleton()
+        while True:
+            with os.scandir('/proc') as it:
+                for entry in it:
+                    # if re.match(r'^\d+$', entry.name):
+                    if entry.name.isdigit():
+                        pids.add(int(entry.name))
+            old_losers, losers = losers, set()
+            sys_stat.refresh()
+            for pid in pids:
+                if pid in old_losers:
+                    losers.add(pid)
+                    continue
+                cpu = cpus.get(pid, None)
+                if not cpu:
+                    cpu = CpuSmooth(pid=pid, avg_secs=10, DB=True)
+                    if not cpu.error:
+                        # print(f'adding: {pid} {cpu.get_nickname()}')
+                        cpus[pid] = cpu
+                if cpu.error:
+                    losers.add(pid)
+                    continue
+                cpu.refresh_cpu()
+            top_cpus = sorted(cpus.values(), key=lambda x: x.percent, reverse=True)
+            top_cpus = top_cpus[:opts.top]
+            top_cpus = sorted(top_cpus, key=lambda x: x.pid)
+            run_time = time.monotonic()-start_mono
+            total_pct = 0
+            for cpu in cpus.values():
+                total_pct += cpu.percent
+            print(f'--------- {run_time:6.1f}s {total_pct:7.2f}% ----------- ',
+                  f'{sys_stat.delta}' + Term.erase_to_eol())
+            old_spots, spots, todo = spots, [None]*opts.top, set()
+            for cpu in top_cpus:
+                try:
+                    idx = old_spots.index(cpu)
+                    spots[idx] = cpu
+                except Exception:
+                    todo.add(cpu)
+            for idx, cpu in enumerate(spots):
+                if not cpu:
+                    cpu = todo.pop()
+                    spots[idx] = cpu
+                print(cpu.db_info + Term.erase_to_eol())
+
+            time.sleep(loop_secs)
+            up_one = Term.pos_up(1) + '\r'
+            print(up_one * (2+len(top_cpus)))
+
+    try:
+        main()
+    except KeyboardInterrupt:
+        pass
+    except Exception as exce:
+        print("exception:", str(exce))
+        print(traceback.format_exc())

{pmemstat-2.3.22 → pmemstat-2.3.23}/pmemstat/pmemstat.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 """
 Pending Features:
@@ -35,7 +35,9 @@ NOTE: kB is a misnomer ... should be "KB".  Morons.
 # pylint: disable=broad-except,import-outside-toplevel,global-statement
 # pylint: disable=too-many-boolean-expressions,invalid-name
 # pylint: disable=too-many-instance-attributes,too-many-lines
-# pylint: disable=too-many-arguments
+# pylint: disable=too-many-arguments,too-many-branches
+# pylint: disable=too-many-statements,too-many-locals
+# pylint: disable=multiple-statements,too-few-public-methods
 
 
 import os
@@ -43,7 +45,6 @@ import re
 import sys
 import traceback
 import time
-import subprocess
 import curses
 # from curses.textpad import rectangle
 from types import SimpleNamespace
@@ -52,9 +53,11 @@ from datetime import datetime, timedelta
 try:
     from Window import Window, OptionSpinner
     from KillThem import KillThem
-except:
+    from CpuSmooth import CpuSmooth, SysStat
+except Exception:
     from pmemstat.Window import Window, OptionSpinner
     from pmemstat.KillThem import KillThem
+    from pmemstat.CpuSmooth import CpuSmooth, SysStat
 
 # Trace Levels:
 #  0 - forced, temporary debugging (comment it out)
@@ -94,14 +97,128 @@ def human(number):
         number /= 1024
         if number < 999.95 or not suffixes:
             return f'{number:.1f}{suffix}'
+    return '' # impossible, but make pylint happy
 
-######
 ####################################################################################
-######
-######
+###### ZramProjector class
 ####################################################################################
-######
+class ZramProjector:
+    """ Guess the zram numbers """
+    def __init__(self):
+        self.meminfo = None
+        self.e_total = 0
+        self.e_avail = 0
+        self.e_used = 0
+        self.e_max_used = 0
+        self.limit_pct = 80
+        self.devs = {}
+        self.DB = False
+
+    def human_pct(self, number):
+        """ Return the number in human form and as pct total memory. """
+        rv = human(number)
+        if number > 0 and self.meminfo.MemTotal > 0:
+            fraction = number/self.meminfo.MemTotal
+            pct = int(round(100*fraction))
+            rv += f'/{pct}%'
+        return rv
+
+    def _get_zram_stats(self):
+        """ Get only what we want
+         orig_data_size   uncompressed size of data stored in this disk.
+         compr_data_size  compressed size of data stored in this disk
+         mem_used_total   the amount of memory allocated for this disk.
+         mem_limit        max RAM ZRAM can use to store (0 means unlimited)
+         mem_used_max     max RAM zram has consumed to store the data
+        """
+        fields = ('orig_data_size compr_data_size mem_used_total'
+                 + ' mem_limit mem_used_max').split()
+        infos = {}
+        zram_devices = sorted([device for device in os.listdir('/sys/class/block/')
+                    if device.startswith('zram')])
+        for device in zram_devices:
+            pathname = f'/sys/class/block/{device}/mm_stat'
+            if not os.path.exists(pathname):
+                continue # not active
+            with open(pathname, encoding='utf-8') as fh:
+                ns = SimpleNamespace()
+                for line in fh: # all the goodies are on 1st line
+                    nums = line.split()
+                    for idx, field in enumerate(fields):
+                        setattr(ns, field, int(nums[idx]))
+                    break
+                infos[device] = ns
+            for param in ('disksize', ):
+                pathname = f'/sys/class/block/{device}/{param}'
+                with open(pathname, encoding='utf-8') as fh:
+                    for line in fh: # one value, one line
+                        setattr(ns, param, int(line.strip()))
+                        break
+            if self.DB: print(f'DB: {device}: {ns}')
+
+        self.devs = infos
+        return infos
+
+    def compute_effective(self, meminfoKB):
+        """ Compute the effective values """
+        def seed(meminfoKB):
+            nonlocal self
+            meminfo = self.meminfo = SimpleNamespace()
+            for key, value in meminfoKB.items():
+                setattr(meminfo, key, value*1024)
+            # artificial stats
+            meminfo.MemUsed = meminfo.MemTotal - meminfo.MemAvailable
+            meminfo.MemZram=0
+            self.e_total = meminfo.MemTotal
+            self.e_avail = meminfo.MemAvailable
+            self.e_used = self.e_total - self.e_avail
+            self.e_max_used = self.e_used
+            return meminfo
+
+        meminfo = seed(meminfoKB)
+        devs = self._get_zram_stats()
+        if not devs:
+            return
+        stats = None
+        for stat in devs.values():
+            # NOTE: generalization for more than one weak unless all
+            #        are identically configured
+            if not stats:
+                stats = vars(stat).copy()
+                continue
+            for key, value in vars(stat).items():
+                stats[key] += value
+        stats = SimpleNamespace(**stats)
+        self.meminfo.MemZram = stats.mem_used_total 
+
+        e_used = meminfo.MemUsed - stats.mem_used_total + stats.orig_data_size
+        ratio = None
+        if e_used <= self.e_used: # zram hurts. assume not enuf data to project
+            e_used = self.e_used
+            ratio = 3.5 # wild guess for projection
+            # if self.DB: dump_effective(ns, '(unchg)')
+            # return ns
+        self.e_used = e_used
+        limit_pct = self.limit_pct
+        if stats.mem_limit > 0:
+            stats_limit_pct = (stats.limit / meminfo.MemTotal)*100
+            limit_pct = min(1.0, limit_pct, stats_limit_pct)
+            # projected amount of orig used memory that fits in limit
+        if ratio is None: # hopefully about 4 ;-)
+            ratio = stats.orig_data_size / stats.mem_used_total
+        e_max_used = ratio * meminfo.MemTotal * limit_pct/100
+            # now limit by disksize
+        e_max_used = min(e_max_used, stats.disksize)
+            # now add uncompressed memory
+        e_max_used += meminfo.MemTotal - e_max_used/ratio
+        self.e_max_used = e_max_used
+        # self.e_max_used = max(e_max_total, e_used) # TODO: uncomment
+        self.e_avail = e_max_used - e_used
 
+
+####################################################################################
+###### ProcMem class
+####################################################################################
 class ProcMem:
     """Represents the memory map summation for processes and groups.
       - the ProcMem object represents one process (or pid)
@@ -158,8 +275,9 @@ class ProcMem:
         self.alive = True
         self.is_new = True
         self.wanted = True # until proven otherwise
+        self.kernel = False # until proven otherwise
         self.is_changed = False
-        self.whynot = None # populate me with why unwanted
+        self.why_not = None # populate me with why unwanted
         self.smaps_file = f'/proc/{self.pid}/smaps'
         self.rollup_file = f'/proc/{self.pid}/smaps_rollup'
         self.cpu = None
@@ -168,56 +286,9 @@ class ProcMem:
 
     def refresh_cpu(self):
         """Get the Cpu Number for the PID (if possible)"""
-        def init_cpu(error=False):
-            return SimpleNamespace(error=error, fh=None,
-                percent=0, hists=[])
-
-        if ProcMem.clock_tick is None:
-            try:
-                ProcMem.clock_tick = 0
-                rv = subprocess.run(['getconf', 'CLK_TCK'],
-                            capture_output=True, text=True, check=True)
-                ProcMem.clock_tick = int(rv.stdout.strip())
-            except Exception:
-                pass
-            if self.clock_tick <= 0:
-                ProcMem.clock_tick = 100
-
         if not self.cpu:
-            self.cpu = init_cpu()
-            try:
-                # pylint: disable=consider-using-with
-                self.cpu.fh = open(f'/proc/{self.pid}/stat', encoding='utf-8')
-            except (PermissionError, FileNotFoundError):
-                self.cpu = init_cpu(error=True)
-                return
-        if self.cpu.error:
-            return
-        try:
-            self.cpu.fh.seek(0)
-            data = self.cpu.fh.read().split()
-            user, system = int(data[13]), int(data[14])
-        except Exception:
-            self.cpu = init_cpu(error=True)
-            return
-        ticks = user + system
-        mono = time.monotonic()
-
-        if self.cpu.hists: # initialized
-            self.cpu.hists.append((ticks, mono))
-            floor_mono = mono - ProcMem.opts.cpu_avg_secs
-            while len(self.cpu.hists) > 1 and self.cpu.hists[0][1] < floor_mono:
-                self.cpu.hists.pop(0)
-            delta_ticks = ticks - self.cpu.hists[0][0]
-            delta_time = mono - self.cpu.hists[0][1]
-            self.cpu.percent = 0
-            if delta_time > 0 and delta_ticks >= 0:
-                self.cpu.percent = round(100
-                    * delta_ticks / ProcMem.clock_tick / delta_time, 8)
-            # print(f'{self.cpu.percent}%')
-
-        else:
-            self.cpu.hists.append((ticks, mono))
+            self.cpu = CpuSmooth(self.pid, avg_secs= ProcMem.opts.cpu_avg_secs)
+        return self.cpu.refresh_cpu() # sets self.cpu.percent
 
     def get_cmdline(self):
         """Get the command line of the PID."""
@@ -235,8 +306,7 @@ class ProcMem:
                 return
             arguments = line.split('\0')
             if not arguments or not arguments[0]: # kernel process
-                self.wanted = False
-                self.whynot = 'KernelProcess'
+                self.wanted, self.kernel = False, True
                 # print(f'{self.pid}: kernel thread')
                 return
             # DB(0, f'{self.pid}: {arguments}')
@@ -260,8 +330,8 @@ class ProcMem:
             # not really expecting this ... probably a bug
             print(f'  WARNING: skip pid={self.pid} no-basename exc={exc}')
             print(traceback.format_exc())
-            self.wanted = False
-            self.whynot = 'CannotGetCmdline'
+            self.wanted = self.kernel = False
+            self.why_not = 'CannotGetCmdline'
             return
 
         ## print(f'DBDB: {self.pid} {ProcMem.opts.pids}')
@@ -269,7 +339,7 @@ class ProcMem:
         if (ProcMem.opts.pids and str(self.pid) not in ProcMem.opts.pids
                 and self.exebasename not in ProcMem.opts.pids):
             self.wanted = False
-            self.whynot = 'FilteredByArgs'
+            self.why_not = 'FilteredByArgs'
             # print(f'    >>>> unwanted {pid}')
         ## print(f'DBDB: {self.pid} wanted={self.wanted} whynot={self.whynot}')
         self.set_key()
@@ -287,13 +357,13 @@ class ProcMem:
                 lines = fhandle.read().splitlines()
         except (PermissionError, FileNotFoundError) as exc:
             # normal cases: not permitted or this is a race where the pid is terminating
-            self.whynot = f'CannotReadLines({type(exc).__name__})'
+            self.why_not = f'CannotReadLines({type(exc).__name__})'
         except Exception as exc:
             # unexpected cases (probably a bug)
             if not self.opts.window:
                 print(f'ERROR: skip pid={self.pid}',
                       f'no-smaps-or-rollup-lines exc={type(exc).__name__}')
-            self.whynot = f'CannotReadLines({type(exc).__name__})'
+            self.why_not = f'CannotReadLines({type(exc).__name__})'
         return lines
 
     def get_rollup_lines(self):
@@ -308,7 +378,7 @@ class ProcMem:
 
         if not rollup_lines:
             self.wanted = False
-            self.whynot = 'CannotReadRollups'
+            self.why_not = 'CannotReadRollups'
         elif DebugLevel:
             DB(3, f'pid={self.pid} {self.exebasename} #rollup_lines={len(rollup_lines)}')
 
@@ -326,7 +396,7 @@ class ProcMem:
 
         if not smaps_lines:
             self.wanted = False
-            self.whynot = 'CannotReadSmaps'
+            self.why_not = 'CannotReadSmaps'
         else:
             if DebugLevel:
                 DB(1, f'pid={self.pid} {self.exebasename} #smaps_lines={len(smaps_lines)}')
@@ -414,6 +484,8 @@ class ProcMem:
                     summary['ptotal'] += val
                 elif tag == 'SwapPss':
                     summary['pswap'] += val
+                    if self.pmemstat.has_zram():
+                        summary['ptotal'] += val
                 continue
             print(f'ERROR: cannot parse "{line}" [{self.rollup_file}:{idx+1}]')
         summary['pss'] = summary['ptotal'] # for consistency
@@ -487,20 +559,19 @@ class ProcMem:
         """Process one PID"""
         self.alive = True
         self.is_changed = False
-        if not self.whynot and not self.cmdline:
+        if not self.why_not and not self.cmdline:
             self.get_cmdline()
             if not self.cmdline:
                 return
         rollup_lines = []
-        if not self.whynot:
+        if not self.why_not:
             rollup_lines = self.get_rollup_lines()
-        if self.whynot:
-            DB(4, f'pid={self.pid} {self.exebasename} whynot={self.whynot}')
+        if self.why_not:
+            DB(4, f'pid={self.pid} {self.exebasename} whynot={self.why_not}')
             return
         self.is_changed = False
         rollup_summary = self.parse_rollups(rollup_lines)
         if self.opts.cpu:
-            self.refresh_cpu()
             rollup_summary['cpu_pct'] = self.cpu.percent
         group = self.pmemstat.get_group(self.key)
         if not group.alive:
@@ -535,6 +606,11 @@ class PmemStat:
         setattr(opts, 'cpu_avg_secs', 20) # pseudo option
         self.groups_by_line = {}
         self._set_units()
+        self.zram_projector = ZramProjector()
+
+    def has_zram(self):
+        """Have zRAM actual? """
+        return bool(self.zram_projector and self.zram_projector.devs)
 
     def get_sortby(self):
         """Make sort_by sensible."""
@@ -777,37 +853,66 @@ class PmemStat:
         # pylint: disable=too-many-branches
 
         def pr_top_of_report(appKB):
-            nonlocal self, meminfoKB, wanted_prcs, total_pids
+            nonlocal self, meminfoKB, wanted_prcs, total_user_pids, kernel_cpu
+            windowed = bool(self.window)
+            resume = False
             # print timestamp of report
-            if not self.window:
-                leader = f'\n---- {now.strftime("%H:%M:%S")}'
-            else:
-                leader = f'{now.strftime("%H:%M:%S")}'
-                self.emit(leader, to_head=True,
-                          attr=curses.A_BOLD if self.loop_num % 2 else None)
-                leader = ''
-
+            leader = '' if windowed else '--- '
+            leader += f'{now.strftime("%H:%M:%S")}'
+            self.emit(leader, to_head=True, resume=resume,
+                      attr=curses.A_BOLD if self.loop_num % 2 else None)
+            leader = ''
+            resume = True
+
+            used = meminfoKB["MemTotal"] - meminfoKB["MemAvailable"]
             leader += f' Tot={human(meminfoKB["MemTotal"]*1024)}'
+            leader += f' Used={human(used*1024)}'
             leader += f' Avail={human(meminfoKB["MemAvailable"]*1024)}'
             if appKB:
                 other = (meminfoKB["MemTotal"] - appKB
                          - meminfoKB["MemAvailable"] - meminfoKB["Shmem"])
                 leader += f' Oth={human(other*1024)}'
-            leader += f' Tmp={human(meminfoKB["Shmem"]*1024)}'
-            leader += f' Dirty={human(meminfoKB["Dirty"]*1024)}'
-            leader += f' PIDs: {len(wanted_prcs)}/{total_pids} {read_smaps}'
-            self.emit(leader, to_head=True, resume=bool(self.window))
+            leader += f' Sh+Tmp={human(meminfoKB["Shmem"]*1024)}'
+            if len(wanted_prcs) < total_user_pids:
+                leader += f' PIDs={len(wanted_prcs)}/{total_user_pids}'
+            else:
+                leader += f' PIDs={total_user_pids}'
             if self.opts.search:
-                self.emit(' /', to_head=True, resume=True)
+                self.emit(leader +' /', to_head=True, resume=resume)
+                resume = True
                 self.emit(self.opts.search, to_head=True,
-                          resume=True, attr=curses.A_UNDERLINE)
-                self.emit('/', to_head=True, resume=True)
+                          resume=resume, attr=curses.A_UNDERLINE)
+                self.emit('/', to_head=True, resume=resume)
+            else:
+                self.emit(leader, to_head=True, resume=resume)
+
+            # second line only if zRAM
+            if not self.has_zram():
+                return
+            resume = False
+            proj = self.zram_projector
+            if self.opts.cpu:
+                leader = f'{kernel_cpu:8.1f}/ker '
+                self.emit(leader, to_head=True, resume=resume)
+                resume = True
+            self.emit(f' zRAM={human(self.zram_projector.meminfo.MemZram)}',
+                      to_head=True, attr=curses.A_BOLD, resume=resume)
+            resume = True
+            leader = ''
+            leader += f' eTot:{proj.human_pct(proj.e_max_used)}'
+            leader += f' eUsed:{proj.human_pct(proj.e_used)}'
+            leader += f' eAvail:{proj.human_pct(proj.e_avail)}'
+            # leader += f' Dirty={human(meminfoKB["Dirty"]*1024)}'
+            self.emit(leader, to_head=True, resume=resume)
 
             # pylint: disable=too-many-branches
         self.loop_num += 1
         meminfoKB = self.get_meminfo()
-        total_pids = 0
-        allpids = []
+        self.zram_projector.compute_effective(meminfoKB)
+        total_user_pids = 0
+        total_kernel_pids = 0
+        kernel_cpu = total_user_pids
+        all_pids = []
         wanted_prcs = {}
 
         self.prep_new_loop(regroup)
@@ -829,9 +934,10 @@ class PmemStat:
             for entry in it:
                 # if re.match(r'^\d+$', entry.name):
                 if entry.name.isdigit():
-                    allpids.append(entry.name)
+                    all_pids.append(entry.name)
 
-        for pid in allpids:
+        prcs = []
+        for pid in all_pids:
             ## print(f'DBDB pid={pid} self.opts.pids={opts.pids}')
             prc = self.prcs.get(pid, None)
             if not prc:
@@ -839,10 +945,25 @@ class PmemStat:
                 self.prcs[pid] = prc
             else:
                 prc.is_new = False
+            prcs.append(prc)
+
+        # do cpu together that stats are consistent
+        if self.opts.cpu:
+            SysStat.refresh()
+            for prc in prcs:
+                if prc.wanted or prc.kernel:
+                    percent = prc.refresh_cpu()
+                if prc.kernel:
+                    kernel_cpu += percent
+                    total_kernel_pids += 1
+                else:
+                    total_user_pids += 1
+
+        for prc in prcs:
             prc.prc_pid()
+            pid = prc.pid
             ## if str(pid) in opts.pids:
                 ## print(f'DBDB pid={pid} dir={vars(prc)}')
-            total_pids += 0 if prc.whynot == 'KernelProcess' else 1
 
             if prc.wanted:
                 wanted_prcs[pid] = prc
@@ -939,6 +1060,8 @@ class PmemStat:
             if not group.alive and group.o_summary and remainder > 0:
                 remainder -= 1
                 self.pr_summary('x', group.o_summary)
+        if not self.window:
+            self.emit('')
 
     def emit(self, line, to_head=False, attr=None, resume=False):
         """ Emit a line of the report"""
@@ -949,7 +1072,7 @@ class PmemStat:
             else:
                 self.window.add_body(line, attr=attr, resume=resume)
         else:
-            print(line)
+            print(('' if resume else '\n') + line, end='')
 
     def help_screen(self):
         """Populate help screen"""

{pmemstat-2.3.22 → pmemstat-2.3.23}/pyproject.toml

@@ -14,7 +14,8 @@
 #   rm -r dist; python3 -m build; ls dist/.
 #   python3 -m twine upload dist/*
 #     ## Enter __token__ and the saved TOKEN (in bitwarden)
-#   sudo python3 -m pip install pmemstat
+#   pipx upgrade pmemstat || pipx install pmemstat # >= python3.11
+#   --OR-- sudo python3 -m pip install pmemstat # <= python3.10
 #   ## VISIT  https://pypi.org/project/pmemstat and delete old versions
 #
 # TEST Build and test (from project directory):
@@ -30,7 +31,7 @@ requires = ["hatchling"]
 build-backend = "hatchling.build"
 
 [project]
-version = "2.3.22"
+version = "2.3.23"
 name = "pmemstat"
 authors = [
     {name = "Joe Defen", email = "joedef@gmail.com"},