thrash-protect 0.0.0.dev0__tar.gz

thrash_protect-0.0.0.dev0/.github/workflows/ci.yml

@@ -0,0 +1,50 @@
+name: CI
+
+on:
+  push:
+    branches: [master, main]
+  pull_request:
+    branches: [master, main]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install ruff
+        run: pip install ruff
+      - name: Run ruff check
+        run: ruff check .
+      - name: Run ruff format check
+        run: ruff format --check .
+
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # Needed for setuptools-scm
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install pytest pytest-cov
+          pip install -e .
+      - name: Run tests
+        run: pytest tests/ -v --cov=thrash_protect --cov-report=xml
+      - name: Upload coverage
+        uses: codecov/codecov-action@v3
+        if: matrix.python-version == '3.12'
+        with:
+          files: ./coverage.xml
+          fail_ci_if_error: false

thrash_protect-0.0.0.dev0/.github/workflows/release.yml

@@ -0,0 +1,48 @@
+name: Release to PyPI
+
+on:
+  push:
+    tags:
+      - 'v*'
+      - '[0-9]+.*'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # Needed for setuptools-scm
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build dependencies
+        run: pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Upload artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write  # Required for trusted publishing
+    steps:
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

thrash_protect-0.0.0.dev0/.gitignore

@@ -0,0 +1,28 @@
+# Python
+__pycache__/
+*.pyc
+*.pyo
+*.egg
+*.egg-info/
+/build/
+/dist/
+
+# Make-generated files
+ChangeLog.recent
+.tag.*
+/*.tar.gz
+
+# archlinux/Makefile generated
+archlinux/PKGBUILD
+archlinux/.SRCINFO
+archlinux/thrash-protect/
+archlinux/*.tar.gz
+
+# rpm/Makefile generated
+rpm/*.spec.bak
+
+# IDE/editor
+.vscode/
+.idea/
+*.swp
+*~

thrash_protect-0.0.0.dev0/.pre-commit-config.yaml

@@ -0,0 +1,15 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.1.9
+    hooks:
+      - id: ruff
+        args: [--fix, --exit-non-zero-on-fix]
+      - id: ruff-format

thrash_protect-0.0.0.dev0/.travis.yml

@@ -0,0 +1,10 @@
+language: python
+# should work in 2.5 as well, but the test code depends on unittest library from python3.3
+# we're attempting to mock up open() - for some reason it fails on python3.3 and python3.4
+python:
+  - "3.5"
+  - "nightly" # currently points to 3.6-dev
+# command to install dependencies
+#install: "..." ## no dependencies
+# command to run tests
+script: nosetests

thrash_protect-0.0.0.dev0/AUTHORS

@@ -0,0 +1,15 @@
+Thrash-protect was initially written in late 2013 by Tobias Brox while
+working at Redpill-Linpro.
+
+The PRIMARY AUTHORS are (and/or have been):
+
+    * Tobias Brox
+
+Other significant contributors:
+
+    * Bjørn Pettersen - quick code review and constructive criticism
+    * Xavier Martin - done some work on prettify the logging a bit
+    * wyg3958 - code cleanup
+    * Riccardo Balbo - bugfixing, ref https://github.com/tobixen/thrash-protect/pull/7
+    * Unnamed colleague - for setting up swap and not editing the MaxClients setting under apache on a production server running some php application, such that we got downtime due to thrashing - wouldn't have made thrash-protect without this one.
+    * Unnamed employee at an unnamed customer - for bypassing our job-scheduling system and running some very memory-intensive processes directly on some of the computing nodes, repeatedly causing nodes to go down due to thrashing.  (thrash-protect eventually did allow his processes to complete, without downtime or further manual work).

thrash_protect-0.0.0.dev0/ChangeLog

@@ -0,0 +1,150 @@
+2025-12-16  Tobias Brox  <tobias@redpill-linpro.com>
+	* thrash-protect.py: Removed Python 2 compatibility code (Python 3.9+ required now)
+	* thrash-protect.py: Applied ruff linting and formatting, removed unused imports
+	* tests/test_thrash_protect.py: Migrated from nose to pytest
+	* pyproject.toml: New modern build configuration (replaces setup.py)
+	* .github/workflows/ci.yml: Added GitHub Actions for linting and testing
+	* .github/workflows/release.yml: Added automatic PyPI release on tags
+	* .pre-commit-config.yaml: Added pre-commit hooks for ruff
+	* docs/CODE_REVIEW.md: Code review with improvement recommendations
+	* docs/TODO.md: TODO list including SSD default tuning issue
+
+2022-01-31  Tobias <tobias@redpill-linpro.com>
+	* thrash-protect.py: terrible, embarrassing bugfix (revealing a lack of code coverage in tests).  Bumped version to 0.14.3.
+	* Makefile: worked a bit more on rpm releases
+
+2021-09-01  Tobias <tobias@redpill-linpro.com>
+	* trash-protect.py: bugfixing and adding debug code (see changelog-0.14.2.md)
+	* INSTALL.md: information about configuration options
+	* README.md: more information about the job control issue
+
+2020-04-08  Niektory <niektory@gmail.com>
+	* trash-protect.py: OOMScoreProcessSelector should be used before PageFaultingProcessSelector, ref https://github.com/tobixen/thrash-protect/issues/31
+
+2019-08-29  Misc authors  <tobias@redpill-linpro.com>
+        * thrash-protect.py: Seems like I've forgotten to maintain the ChangeLog for more than a year.  Misc tweaks have been done, catching up on feedback and pull requests coming through github.  Bumping to 0.13.
+	* README.md: various documentation tweaks
+
+2018-08-23  Tobias Brox  <tobias@redpill-linpro.com>
+        * README.md: we don't need this file anymore, since we have README.rst
+        * README.rst: merged latest changes from README.md
+        * thrash-protect.py: more information in an error message.  Bumping 0.12.1.
+
+2018-03-21  Tobias Brox  <tobias@redpill-linpro.com>
+        * thrash-protect.py: bugfix in logging, when parent pids are suspended they should also be logged
+        * thrash-protect.py: Trying yet another time to fix the thrash-detecting algorithm; apparently the December bugfix was not well enough tested or thought through.
+        * ChangeLog: updated, preparing for new release 0.12.
+
+2018-02-08  Tobias Brox  <tobias@redpill-linpro.com>
+        * thrash-protect.py: Silly-stupid bugfix for the THRASH_PROTECT_WHITELIST configuration environmental variable
+        * README: doc tweaks
+
+2017-12-08  Tobias Brox  <tobias@redpill-linpro.com>
+	* thrash-protect.py: a bug in the thrash-detecting algorithm; the threshold for detecting thrashing has efficiently been 1 all the time, even though it's defaulted to 512 in the config section.  Fixed the bug and lowered the default to 4, not to change the behaviour too drastically.  This will cause thrash-protect to be less aggressive.
+	* ChangeLog: updated, preparing for new release 0.11.6 ... 0.11.7 even
+	* Makefile: latest changes did not work well for the archlinux install
+
+2017-10-30  Tobias Brox  <tobias@redpill-linpro.com>
+	* README: documentation tweaks and new thoughts
+	* thrash-protect.py: sudo is added to the list of processes where it may be important to suspend the parent process
+
+2017-10-11  Tobias Brox  <tobias@redpill-linpro.com>
+	* Makefile: attempting to get directories right for other distros - credits to @anophelesgreyhoe@github
+
+2017-02-23  Chris Coetzee  <chriscz93@gmail.com>
+	* setup.py: pypi-compatible setup script
+	* thrash-protect.py: moving logics to a main sub - minor refactoring to follow best pypi-practice
+
+2017-02-20  Tobias Brox  <tobias@redpill-linpro.com>
+	* rpm/thrash-protect.spec: bugfixing the rpm build
+
+2017-02-17  Tobias Brox  <tobias@redpill-linpro.com>
+	* rpm/thrash-protect.spec: bugfix, trying to generate an rpm for centos7
+
+2017-02-11  Riccardo Balbo  <riccardo@forkforge.net>
+	* thrash-protect: bugfix: processes with spaces or paranthesis in the name was not properly handled
+
+2016-11-21  Tobias Brox  <tobias@redpill-linpro.com>
+	* puppet/templates/nrpe.cfg.erb: puppet/nrpe tuning; stuck frozen pid list is not allowed to be more than 3s old (was 5s).
+	* archlinux/Makefile: bugfix: "make archlinux" should be indempotent
+	* AUTHOR: added other contributors
+	* README: added info on a theoretical problem that thrash-protect will be suspending processes "unfairly", removed donation section
+
+2016-11-15  Tobias Brox  <tobias@redpill-linpro.com>
+	* thrash-protect.py: the script broke on python 2.6.  Done a quick and dirty workaround, the extended logging information just won't work under older python.  Released 0.11.4 (0.11.2 and 0.11.3 was redundant releases - I've completely forgotten the prerequisites for a "make release")
+
+2016-02-22  Tobias Brox  <tobias@redpill-linpro.com>
+        * archlinux/ The AUR setup has been changed quite a bit, the PKGBUILD was buggy.  Version 0.11.1 only contains changes to the arch linux package management
+
+2016-02-19  Tobias Brox  <tobias@redpill-linpro.com>
+	* thrash-protect.py: misc minor stuff since last release, including some code cleanup, some error handling, better logging; much of it pulled in from other forks (sorry, I'm too lazy to do a proper job creating the missing ChangeLog entries).
+
+2014-12-18  Tobias Brox  <tobias@redpill-linpro.com>
+        * thrash-protect.py: some refactoring, plus now it will STOP a parent bash process before stopping the child, not to disrupt the job control
+        * tests/test_thrash_protect.py: more unit tests and a small functional test
+        * tests/thrash_bot2.py: now with argparse (though, that requires 2.7) and made compatible with python3.
+
+2014-12-09  Tobias Brox  <tobias@redpill-linpro.com>
+        * thrash-protect.py: started work on solving the annoying bash job control interpherence ... if suspending a bash child, make sure the parent process is already suspended.
+	* test/test_thrash_protect: first unit tests are coming up
+
+2014-11-20  Tobias Brox  <tobias@redpill-linpro.com>
+        * thrash-protect.py: major refactoring, cleaning up quite some mess and getting rid of quite some globals
+        * thrash-protect.py: fixed a bug - when thrash_protect kicks in, it will at first almost always freeze a pid and then unfreeze it immediately afterwards without any sleep
+
+2014-03-10  Tobias Brox  <tobias@redpill-linpro.com>
+        * thrash-protect.py: found a silly print statement left behind from earlier debug rounds.
+	* thrash-protect.py: bumped the version (patchlevel)
+
+2014-02-26  Tobias Brox  <tobias@redpill-linpro.com>
+	* README.md - elaborated a bit on the drawbacks/problems observed
+	* thrash-protect.py (thrash_protect): doing a best-effort on running mlockall()
+
+2014-01-14  Tobias Brox  <tobias@redpill-linpro.com>
+	* general: version number bump - there shouldn't have been any 0.6.4, it should have been released as 0.7.
+	* README.md: some minor docfixes
+	* tests/thrash-bot.py: a bit of confusion on what version of this script I actually used when I managed to thrash the system.  At least now it seems to be eating up memory very fast instead of just spinning CPU.
+
+2014-01-13  Tobias Brox  <tobias@redpill-linpro.com>
+	* general: bump of version number to 0.6.4 (post-script: this was a mistake - correct version number should have been 0.7).
+	* tests/: new directory, unfortunately rather empty
+	* tests/thrash-bot.py: improved the thrash-bot, it consumes far less CPU and gobbles far more memory now
+	* thrash-protect.py (general): tweaks and bugfixes to handle the new thrash_bot.py.  unfortunately I added lot more complexity. :-(
+	* thrash-protect.py (general): debug hooks for randomly stopping processes even if there is no thrashing going on
+	* thrash-protect.py (scan_processes): modified from logic to find the process with most pagefaults to "meta-logic" selecting an algorithm for finding which process to stop.
+	* thrash-protect.py (scan_processes_pagefaults): old logics from scan_processes
+	* thrash-protect.py (scan_processes_pagefaults): various bugfixes
+	* thrash-protect.py (scan_processes_pagefaults): added a threshold to make it less likely whitelisted processes will be stopped
+	* thrash-protect.py (scan_processes_oom_score): new function to select the process with highest oom_score
+	* thrash-protect.py (find_last_unfrozen_process): returns the pid of the last unfrozen process, if it's still running
+
+2013-12-26  Tobias Brox  <tobias@redpill-linpro.com>
+	* general: bump of version number to 0.6.3; only "meta"-changes (Makefile, etc)
+	* general: successfully created a .deb-package that has been deployed on ubuntu precise
+	* INSTALL.md: new file - extracted the practical information out from all the blah-blah in README.md
+	* README.md: even more blah-blah - now information on the alternatives to thrash-protect
+	* Makefile: quite some bugfixes and work
+	* debian/*: debianization.
+	* systemv/thrash-protect: bugfix to get correct return code from "status"
+	* puppet/: puppet setup
+
+2013-12-20  Tobias Brox  <tobias@redpill-linpro.com>
+	* general: bumping version number to 0.6
+	* thrash-protect.py: Major fix - changed from using the "major page fault"-counter to using the "page swap in" and "page swap out" counters.  Bidirectional swap or more than 100 pages in/out during a one second interval will trigger process suspension.
+	* thrash-protect.py: Made symmetry between the blacklisting and whitelisting; it's done by weighting instead of "stop this process first" or "don't stop this process".
+	* thrash-protect.py: Minor bugfix - unfrozen processes didn't get logged properly due to a ProcessLookupFailure when running CONT on the session or group id.
+	* puppet/: includes an erb-template for a nrpe config file, though might not work outside our puppet environment (depends on $nagios::nrpe::nagios_plugin_dir to be set)
+
+2013-12-19  Tobias Brox  <tobias@redpill-linpro.com>
+	* thrash-protect.py: made support for --version at python2.7 and up, and split out the main logic into a subroutine.
+	* rpm: work in progress to build rpm files
+	* systemv: work in progress to build a working init-script
+	* archlinux/Makefile: allow for automatic build of PKGBUILD and upload to AUR
+	* Makefile: Working on release-management through Makefile; ideally all the work involved with making a release should be handled by a simple make command.  Pardon for all the stupid mini-releases, need to do quite some testing here.
+	* ChangeLog: Now with a ChangeLog file.  The entries will be used for the commit message when doing releases.
+
+2013-09-06  Tobias Brox  <tobias@redpill-linpro.com>
+	* thrash-protect.py: Got a mostly working prototype here - it has been tested in production, and it seems to do what it promises, and I'm happy with it, maybe the initial plan rewriting it in C is not really needed.
+
+2013-07-29  Tobias Brox <tobias@redpill-linpro.com>
+	* General: initial commit

thrash_protect-0.0.0.dev0/INSTALL.md

@@ -0,0 +1,117 @@
+Installation and usage
+======================
+
+Requirements
+------------
+
+This will only work on linux, it depends on reading stats from the
+/proc directory, it depends on python 3 (python 2.5 or higher should probably work - and an old version of the script was backported to python 2.4).
+
+No other dependencies.
+
+The box or VM running thrash-protect needs to be set up with swap, or
+trash-protect won't do anything useful (even if thrash-like situations
+can happen without swap installed).  A reasonably large swap partition
+is recommended, possibly twice as much swap as physical memory, though
+YMMV, and even a very small swap partition is enough for
+thrash-protect to do useful work.
+
+My original idea was to make a rapid prototype in python, and then
+port it over to C for a smaller memory- and CPU footprint; while
+thrash-protect has successfully been running on single-CPU instances
+with 512M RAM, it's probably best suited on systems with at least 1GB
+RAM and multiple CPUs (or CPU cores) due to the overhead costs.
+
+Compile and Install
+-------------------
+
+As it's in python, no compilation is needed.
+
+"make install" will hopefully do the right thing and install the
+script as a service.
+
+Archlinux users may also install through AUR.  rpm and deb packages
+will be made available on request.  There are some logic in the Makefile for creating such packages, but it's poorly tested.
+
+
+Usage
+-----
+
+The service will need to be started and/or set up to start at boot.
+
+If everything else fails, just run the script as root and do whatever
+is necessary to ensure it will be started again after next reboot.
+
+While it should be possible to adjust configuration through
+environment variables, best practice is probably to run it without any
+configuration.
+
+The System V init file is so far quite redhat-specific and may need
+tuning for usage with other distributions.
+
+Configuration
+-------------
+
+It should be possible to configure the script through environment
+variables, though this is poorly tested - the default configuration
+has mostly been working out for me.  However, the defaults was made in 2013 and may possibly need a bit of tweaking for state-of-the-art equipment.
+
+Configuration environment variables that may need tweaking:
+
+* THRASH_PROTECT_CMD_WHITELIST - a list of processes that you rather don't want thrash-protect to touch (no guarantees - it just adds a weight).  Defaults to "sshd bash xinit X spectrwm screen SCREEN mutt ssh xterm rxvt urxvt Xorg.bin Xorg systemd-journal".  Can most likely be trimmed down, particularly on servers.  On desktop systems you may want to add more processes, depending on your desktop system.
+* THRASH_PROTECT_CMD_BLACKLIST - opposite of whitelist - processes thrash-protect should prioritize to stop.  Defaults to ''.
+* THRASH_PROTECT_CMD_JOBCTRLLIST - processes that may be confused if the child process gets suspended.  Defaults to "bash sudo".  You may want to do some research if you use another shell, run bash under some pseudonym, or have other job control systems or experience problems with other processes.  (See the README for details).
+* THRASH_PROTECT_INTERVAL - thrash protect is set to sleep for 0.5s between each normal iteration, as long as no thrashing is detected.  This default was set in 2013, perhaps it can be tuned down on modern hardware.
+* SWAP_PAGE_THRESHOLD - defaults to 4.  If there is 4 pages swapped in and 4 pages swapped out during the interval, the script will be triggered.  There is also a hard coded constant 10x for single-direction swapping during the interval, so if 40 blocks are swapped in or out, the algorithm will also trigger.  The default was set in 2013, maybe it should be adjusted upwards on swap media with high bandwidth, to prevent thrash-protect from suspending processes when it's not needed.
+* THRASH_PROTECT_UNFREEZE_POP_RATIO - default 5.  TLDR: should probably be lowered on interactive desktops and increased on servers doing only batch processing.  All suspended processes are put in a double ended queue (a double ended queue behaves both as a queue and a stack - so the pid is placed at the end of the queue or at the top of the stack according to how you look at it).  If the host has stopped thrashing, the "fair" thing to do would be to always resume the process at the front of the queue (unfreeze_pop_ratio set to 1), but the most effective thing to do is probably to resume and suspend the same process over and over again (unfreeze_pop_ratio set to MAXINT). When set to five it will pop four processes from the top of the stack before it pulls out one process from the front of the queue.
+* THRASH_PROTECT_BLACKLIST_SCORE_MULTIPLIER - default 16.  A blacklisted job will be 16 times more likely to be picked up for suspension than a non-blacklisted job.
+* THRASH_PROTECT_WHITELIST_SCORE_MULTIPLIER - default 4 times the blacklist score multiplier.  A non-whitelisted job will by default be 64 times more likely to be choosen for suspension than a whitelisted job.
+* THRASH_PROTECT_LOG_USER_DATA_ON_FREEZE - we may log extra process data when freezing processes.  The current code forks up a `ps` subprocess (should be rewritten to just check up /proc/stat).  Since the system may be critically overloaded when we want to freeze a process, it's considered that we probably don't want to do this, so it's defaulted to false.  Note that this is about "hard" logging and the log location is hard coded to /var/log/thrash-protect.log (should probably be consolidated with logging done through the logging module).
+* THRASH_PROTECT_LOG_USER_DATA_ON_UNFREEZE - much the same as the former.  Since the system is probably not critically overloaded when we want to unfreeze a process, it's considered that we probably do want this logging, so default is set to true.
+* THRASH_PROTECT_DEBUG_LOGGING - leave it turned off, or thrash-protect will log a lot to stderr (trough the logging module).
+* THRASH_PROTECT_DEBUG_CHECKSTATE - will log warnings (through the logging module) if processes are in unexpected states, i.e. because two instances of the script is running at the same time.
+* THRASH_PROTECT_DATE_HUMAN_READABLE - the early versions of the script logged timestamps in unix format (long int).  Set to 0 if you prefer such timestamps.
+* THRASH_PROTECT_PGMAJFAULT_SCAN_THRESHOLD - the script maintains a list of processes and amount of "major page faults" every process has done.  This is a bit expensive process hence it's only done when the global major page fault counter has passed some threshold.  Default set to swap_page_threshold*4.  Can probably be left where it is.
+ * THRASH_PROTECT_TEST_MODE - pretend the system is thrashed every now and then, for testing purposes.  This hasn't been exercised for quite some years, should probably be removed.
+
+Monitoring
+----------
+
+thrash-protect may relatively safely live it's own life, users will
+only notice some delays and slowness, and bad situations will
+autorecover (i.e. the resource-consuming process will stop by itself,
+or the kernel will finally run out of swap and the OOM-killer will
+kill the rogue process).
+
+For production servers, thrash-protect should ideally only be latent,
+only occationally stop something very briefly, if it becomes active a
+system administrator should manually inspect the box and deal with the
+situation, and eventually order more memory.
+
+There are three useful ways to monitor:
+
+* Monitoring the number of suspended processes.  This will possibly
+  catch situations where thrash-protect itself has gone haywire,
+  suspending processes but unable to reanimate them.  Unfortunately it
+  may also cause false alarms on systems where processes are being
+  suspended legitimately outside thrash-protect (i.e. due to some
+  sysadmin pressing ^Z).
+
+* Monitoring the /tmp/thrash-protect-frozen-pid-list file.  It should
+  only exist briefly.
+
+* Age of the /tmp/thrash-protect-frozen-pid-list file; if it exists
+  and is old, most likely thrash-protect is not running anymore.
+
+nrpe-scripts and icinga-configuration may be done available on request.
+
+Subdirectories
+--------------
+
+The subdirectories contains various logic for deploying the script:
+
+* archlinux - contains logic for submitting to AUR for Arch Linux
+* systemv - contains a traditional init-script, though it may be rather RedHat-specific as for now
+* systemd - contains a service config file for running the script under systemd
+* upstart - contains the config file for starting up the script under the (Ubuntu) upstart system
+* debian - contains files necessary for "debianization" and creating .deb-packages for ubuntu and debian