badwulf 0.1.0__tar.gz

badwulf-0.1.0/LICENSE

@@ -0,0 +1,181 @@
+The Artistic License 2.0
+
+Copyright (c) 2000-2006, The Perl Foundation.
+
+Everyone is permitted to copy and distribute verbatim copies of this license
+document, but changing it is not allowed.
+
+Preamble
+
+This license establishes the terms under which a given free software Package
+may be copied, modified, distributed, and/or redistributed. The intent is that
+the Copyright Holder maintains some artistic control over the development of
+that Package while still keeping the Package available as open source and free
+software.
+
+You are always permitted to make arrangements wholly outside of this license
+directly with the Copyright Holder of a given Package. If the terms of this
+license do not permit the full use that you propose to make of the Package,
+you should contact the Copyright Holder and seek a different licensing
+arrangement.
+
+Definitions
+
+"Copyright Holder" means the individual(s) or organization(s) named in the
+copyright notice for the entire Package.
+
+"Contributor" means any party that has contributed code or other material to
+the Package, in accordance with the Copyright Holder's procedures.
+
+"You" and "your" means any person who would like to copy, distribute, or
+modify the Package.
+
+"Package" means the collection of files distributed by the Copyright Holder,
+and derivatives of that collection and/or of those files. A given Package may
+consist of either the Standard Version, or a Modified Version.
+
+"Distribute" means providing a copy of the Package or making it accessible to
+anyone else, or in the case of a company or organization, to others outside of
+your company or organization.
+
+"Distributor Fee" means any fee that you charge for Distributing this Package
+or providing support for this Package to another party. It does not mean
+licensing fees.
+
+"Standard Version" refers to the Package if it has not been modified, or has
+been modified only in ways explicitly requested by the Copyright Holder.
+
+"Modified Version" means the Package, if it has been changed, and such changes
+were not explicitly requested by the Copyright Holder.
+
+"Original License" means this Artistic License as Distributed with the
+Standard Version of the Package, in its current version or as it may be
+modified by The Perl Foundation in the future.
+
+"Source" form means the source code, documentation source, and configuration
+files for the Package.
+
+"Compiled" form means the compiled bytecode, object code, binary, or any other
+form resulting from mechanical transformation or translation of the Source
+form.
+
+Permission for Use and Modification Without Distribution
+
+(1) You are permitted to use the Standard Version and create and use Modified
+Versions for any purpose without restriction, provided that you do not
+Distribute the Modified Version.
+
+Permissions for Redistribution of the Standard Version
+
+(2) You may Distribute verbatim copies of the Source form of the Standard
+Version of this Package in any medium without restriction, either gratis or
+for a Distributor Fee, provided that you duplicate all of the original
+copyright notices and associated disclaimers. At your discretion, such
+verbatim copies may or may not include a Compiled form of the Package.
+
+(3) You may apply any bug fixes, portability changes, and other modifications
+made available from the Copyright Holder. The resulting Package will still be
+considered the Standard Version, and as such will be subject to the Original
+License.
+
+Distribution of Modified Versions of the Package as Source
+
+(4) You may Distribute your Modified Version as Source (either gratis or for a
+Distributor Fee, and with or without a Compiled form of the Modified Version)
+provided that you clearly document how it differs from the Standard Version,
+including, but not limited to, documenting any non-standard features,
+executables, or modules, and provided that you do at least ONE of the
+following:
+
+(a) make the Modified Version available to the Copyright Holder of the
+Standard Version, under the Original License, so that the Copyright Holder may
+include your modifications in the Standard Version.
+
+(b) ensure that installation of your Modified Version does not prevent the
+user installing or running the Standard Version. In addition, the Modified
+Version must bear a name that is different from the name of the Standard
+Version.
+
+(c) allow anyone who receives a copy of the Modified Version to make the
+Source form of the Modified Version available to others under
+
+(i) the Original License or
+
+(ii) a license that permits the licensee to freely copy, modify and
+redistribute the Modified Version using the same licensing terms that apply to
+the copy that the licensee received, and requires that the Source form of the
+Modified Version, and of any works derived from it, be made freely available
+in that license fees are prohibited but Distributor Fees are allowed.
+
+Distribution of Compiled Forms of the Standard Version or Modified Versions
+without the Source
+
+(5) You may Distribute Compiled forms of the Standard Version without the
+Source, provided that you include complete instructions on how to get the
+Source of the Standard Version. Such instructions must be valid at the time of
+your distribution. If these instructions, at any time while you are carrying
+out such distribution, become invalid, you must provide new instructions on
+demand or cease further distribution. If you provide valid instructions or
+cease distribution within thirty days after you become aware that the
+instructions are invalid, then you do not forfeit any of your rights under
+this license.
+
+(6) You may Distribute a Modified Version in Compiled form without the Source,
+provided that you comply with Section 4 with respect to the Source of the
+Modified Version.
+
+Aggregating or Linking the Package
+
+(7) You may aggregate the Package (either the Standard Version or Modified
+Version) with other packages and Distribute the resulting aggregation provided
+that you do not charge a licensing fee for the Package. Distributor Fees are
+permitted, and licensing fees for other components in the aggregation are
+permitted. The terms of this license apply to the use and Distribution of the
+Standard or Modified Versions as included in the aggregation.
+
+(8) You are permitted to link Modified and Standard Versions with other works,
+to embed the Package in a larger work of your own, or to build stand-alone
+binary or bytecode versions of applications that include the Package, and
+Distribute the result without restriction, provided the result does not expose
+a direct interface to the Package.
+
+Items That are Not Considered Part of a Modified Version
+
+(9) Works (including, but not limited to, modules and scripts) that merely
+extend or make use of the Package, do not, by themselves, cause the Package to
+be a Modified Version. In addition, such works are not considered parts of the
+Package itself, and are not subject to the terms of this license.
+
+General Provisions
+
+(10) Any use, modification, and distribution of the Standard or Modified
+Versions is governed by this Artistic License. By using, modifying or
+distributing the Package, you accept this license. Do not use, modify, or
+distribute the Package, if you do not accept this license.
+
+(11) If your Modified Version has been derived from a Modified Version made by
+someone other than you, you are nevertheless required to ensure that your
+Modified Version complies with the requirements of this license.
+
+(12) This license does not grant you the right to use any trademark, service
+mark, tradename, or logo of the Copyright Holder.
+
+(13) This license includes the non-exclusive, worldwide, free-of-charge patent
+license to make, have made, use, offer to sell, sell, import and otherwise
+transfer the Package with respect to any patent claims licensable by the
+Copyright Holder that are necessarily infringed by the Package. If you
+institute patent litigation (including a cross-claim or counterclaim) against
+any party alleging that the Package constitutes direct or contributory patent
+infringement, then this Artistic License to you shall terminate on the date
+that such litigation is filed.
+
+(14) Disclaimer of Warranty:
+
+THE PACKAGE IS PROVIDED BY THE COPYRIGHT HOLDER AND CONTRIBUTORS "AS IS'
+AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES. THE IMPLIED WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT ARE
+DISCLAIMED TO THE EXTENT PERMITTED BY YOUR LOCAL LAW. UNLESS REQUIRED BY LAW,
+NO COPYRIGHT HOLDER OR CONTRIBUTOR WILL BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING IN ANY WAY OUT OF THE USE OF THE
+PACKAGE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

badwulf-0.1.0/PKG-INFO

@@ -0,0 +1,39 @@
+Metadata-Version: 2.1
+Name: badwulf
+Version: 0.1.0
+Summary: Minimal manager for Beowulf clusters and scientific data
+Author-email: "Kylie A. Bemis" <k.bemis@northeastern.edu>
+License: Artistic License 2.0
+Project-URL: Homepage, https://github.com/kuwisdelu/badwulf
+Project-URL: Issues, https://github.com/kuwisdelu/badwulf/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: License :: OSI Approved :: Artistic License
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: Unix
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: tomli>=1.1.0; python_version < "3.11"
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+
+# Badwulf 
+
+## Minimal manager for Beowulf clusters and scientific data
+
+The goal of *badwulf* is a provide a minimal command line interface for accessing and managing experimental data on scientific computing servers and Beowulf clusters.
+
+This tool is __not__ intended to replace true cluster management and scheduling software for such as *Slurm*. Instead, *badwulf* is a lightweight package for simplying tasks such as:
+
+- Connecting an port forwarded SSH session to a research server behind a login server
+
+- Transfering files and directories between a local client and a research server
+
+- Managing a simple repository of experimental data and metadata
+
+- Searching experimental metadata for terms and keywords
+
+- Syncing experimental data between a local client and a research server

badwulf-0.1.0/README.md

@@ -0,0 +1,17 @@
+# Badwulf 
+
+## Minimal manager for Beowulf clusters and scientific data
+
+The goal of *badwulf* is a provide a minimal command line interface for accessing and managing experimental data on scientific computing servers and Beowulf clusters.
+
+This tool is __not__ intended to replace true cluster management and scheduling software for such as *Slurm*. Instead, *badwulf* is a lightweight package for simplying tasks such as:
+
+- Connecting an port forwarded SSH session to a research server behind a login server
+
+- Transfering files and directories between a local client and a research server
+
+- Managing a simple repository of experimental data and metadata
+
+- Searching experimental metadata for terms and keywords
+
+- Syncing experimental data between a local client and a research server

badwulf-0.1.0/pyproject.toml

@@ -0,0 +1,45 @@
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "badwulf"
+version = "0.1.0"
+authors = [
+	{name = "Kylie A. Bemis", email = "k.bemis@northeastern.edu"}
+]
+description = "Minimal manager for Beowulf clusters and scientific data"
+readme = "README.md"
+license = {text = "Artistic License 2.0"}
+requires-python = ">=3.10"
+dependencies = [
+	"tomli >= 1.1.0 ; python_version < '3.11'"
+]
+classifiers = [
+	"Development Status :: 3 - Alpha",
+	"Programming Language :: Python :: 3",
+	"Programming Language :: Python :: 3.10",
+	"Programming Language :: Python :: 3.11",
+	"License :: OSI Approved :: Artistic License",
+	"Intended Audience :: Science/Research",
+	"Operating System :: Unix",
+]
+
+[project.urls]
+Homepage = "https://github.com/kuwisdelu/badwulf"
+Issues = "https://github.com/kuwisdelu/badwulf/issues"
+
+[project.optional-dependencies]
+test = ["pytest"]
+
+[external]
+dependencies = [
+	"pkg:generic/ssh",
+	"pkg:generic/rsync",
+]
+
+[tool.pytest.ini_options]
+addopts = [
+	"--import-mode=importlib",
+]

badwulf-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

badwulf-0.1.0/src/badwulf/__init__.py

@@ -0,0 +1,5 @@
+
+from .rssh import rssh
+from .expdb import expdb
+
+__all__ = ["rssh", "expdb"]

badwulf-0.1.0/src/badwulf/cli/__init__.py

@@ -0,0 +1,5 @@
+
+from .clmanager import clmanager
+from .dbmanager import dbmanager
+
+__all__ = ["clmanager", "dbmanager"]

badwulf-0.1.0/src/badwulf/cli/clmanager.py

@@ -0,0 +1,298 @@
+
+import sys
+import platform
+import subprocess
+import argparse
+import datetime
+import importlib.metadata
+from time import sleep
+
+from ..tools import *
+from ..rssh import rssh
+
+class clmanager:
+	"""
+	Command line utility for Beowulf clusters
+	"""
+	
+	def __init__(self,
+		name,
+		nodes,
+		version,
+		date,
+		description,
+		readme = None,
+		program = None,
+		username = None,
+		server = None,
+		server_username = False,
+		port = None):
+		"""
+		Initialize a cluster CLI utility program
+		:param name: The name of the cluster/server
+		:param nodes: A list of nodenames or dict in the form {alias: nodename}
+		:param version: The version of the program
+		:param date: The date of the program's last revision
+		:param description: A description of the program
+		:param readme: The file path of a README.md file
+		:param program: The name of the program (defaults to name)
+		:param username: Your username on the cluster
+		:param server: The gateway server hostname (optional)
+		:param server_username: Your username on the gateway server (optional)
+		:param port: The local port for gateway server SSH forwarding
+		"""
+		self.name = name
+		self.nodes = nodes
+		self.version = version
+		if isinstance(date, datetime.date):
+			self.date = date
+		else:
+			self.date = datetime.date.fromisoformat(date)
+		self.description = description
+		self.readme = readme
+		if program is None:
+			self.program = name.casefold()
+		else:
+			self.program = program
+		self.username = username
+		self.server = server
+		self.server_username = server_username
+		self.port = port
+		self._parser = None
+		self._args = None
+
+	def _add_cluster_args(self, parser):
+		"""
+		Add cluster parameters to a parser.
+		:param parser: The parser to update
+		"""
+		if isinstance(self.nodes, dict):
+			for alias, nodename in self.nodes.items():
+				parser.add_argument(f"-{alias}", action="append_const",
+					help=nodename, dest="nodes", const=nodename)
+		parser.add_argument("-n", "--node", action="append",
+			help=f"{self.name} node", dest="nodes",
+			metavar="NODE")
+		parser.add_argument("-p", "--port", action="store",
+			help="port forwarding", default=self.port)
+		parser.add_argument("-u", "--user", action="store",
+			help=f"{self.name} user", default=self.username)
+		parser.add_argument("-L", "--login", action="store",
+			help="gateway server user", default=self.server_username)
+		parser.add_argument("-S", "--server", action="store",
+			help="gateway server host", default=self.server)
+	
+	def _add_subcommand_run(self, subparsers):
+		"""
+		Add 'run' subcommand to subparsers.
+		:param subparsers: The subparsers to update
+		"""
+		cmd = subparsers.add_parser("run", 
+			help=f"run command (e.g., shell) on a {self.name} node")
+		self._add_cluster_args(cmd)
+		cmd.add_argument("remote_command", action="store",
+			help="command to execute on a Magi node", nargs=argparse.OPTIONAL,
+			metavar="command")
+		cmd.add_argument("remote_args", action="store",
+			help="command arguments", nargs=argparse.REMAINDER,
+			metavar="...")
+	
+	def _add_subcommand_copy_id(self, subparsers):
+		"""
+		Add 'copy-id' subcommand to subparsers.
+		:param subparsers: The subparsers to update
+		"""
+		cmd = subparsers.add_parser("copy-id", 
+			help=f"copy ssh keys to a {self.name} node")
+		self._add_cluster_args(cmd)
+		cmd.add_argument("identity_file", action="store",
+			help="ssh key identity file")
+	
+	def _add_subcommand_push(self, subparsers):
+		"""
+		Add a 'push' subcommand to subparsers.
+		:param subparsers: The subparsers to update
+		"""
+		cmd = subparsers.add_parser("push", 
+			help=f"upload file(s) to {self.name}")
+		self._add_cluster_args(cmd)
+		cmd.add_argument("src", action="store",
+			help="source file/directory")
+		cmd.add_argument("dest", action="store",
+			help="destination file/directory")
+		cmd.add_argument("--ask", action="store_true",
+			help="ask to confirm before uploading files?")
+		cmd.add_argument("--dry-run", action="store_true",
+			help="show what would happen without doing it?")
+	
+	def _add_subcommand_pull(self, subparsers):
+		"""
+		Add a 'pull' subcommand to subparsers.
+		:param subparsers: The subparsers to update
+		"""
+		cmd = subparsers.add_parser("pull", 
+			help=f"download file(s) from {self.name}")
+		self._add_cluster_args(cmd)
+		cmd.add_argument("src", action="store",
+			help="source file/directory")
+		cmd.add_argument("dest", action="store",
+			help="destination file/directory")
+		cmd.add_argument("--ask", action="store_true",
+			help="ask to confirm before downloading files?")
+		cmd.add_argument("--dry-run", action="store_true",
+			help="show what would happen without doing it?")
+	
+	def _add_subcommand_readme(self, subparsers):
+		"""
+		Add 'readme' subcommand to subparsers.
+		:param subparsers: The subparsers to update
+		"""
+		cmd = subparsers.add_parser("readme", 
+			help="display readme")
+		cmd.add_argument("-p", "--pager", action="store",
+			help="program to display readme (default 'glow')")
+		cmd.add_argument("-w", "--width", action="store",
+			help="word-wrap readme at width (default 70)", default=70)
+
+	def _init_parser(self):
+		"""
+		Initialize the argument parser
+		"""
+		parser = argparse.ArgumentParser(self.program,
+			description=self.description)
+		parser.add_argument("-v", "--version", action="store_true",
+			help="display version")
+		subparsers = parser.add_subparsers(dest="cmd")
+		self._add_subcommand_run(subparsers)
+		self._add_subcommand_copy_id(subparsers)
+		self._add_subcommand_push(subparsers)
+		self._add_subcommand_pull(subparsers)
+		if self.readme is not None:
+			self._add_subcommand_readme(subparsers)
+		self._parser = parser
+	
+	def is_node(self):
+		"""
+		Check if the program is running on a cluster node
+		:returns: True if running the a cluster node, False otherwise
+		"""
+		if isinstance(self.nodes, dict):
+			nodes = self.nodes.values()
+		else:
+			nodes = self.nodes
+		return is_known_host(nodes)
+	
+	def resolve_node(self, nodes):
+		"""
+		Get single valid nodename from a list of nodes
+		:param nodes: A list of nodenames
+		"""
+		host = platform.node().replace(".local", "")
+		if nodes is None or len(nodes) != 1:
+			sys.exit(f"{self.program}: error: must specify exactly _one_ {self.name} node")
+		node = nodes[0]
+		if self.is_node():
+			if host == node.casefold():
+				node = "localhost"
+			else:
+				node += ".local"
+		return node
+	
+	def open_ssh(self,
+		node,
+		username = None,
+		server = None,
+		server_username = None,
+		port = None):
+		"""
+		Open SSH connection to a cluster node
+		:param node: The target cluster node
+		:param username: Your username on the cluster
+		:param server: The gateway server hostname (optional)
+		:param server_username: Your username on the gateway server (optional)
+		:param port: Port used for gateway forwarding
+		:returns: An open rssh instance
+		"""
+		if username is None:
+			username = self.username
+		if server is None:
+			server = self.server
+		if server_username is None:
+			server_username = self.server_username
+		if port is None:
+			port = findport()
+		# connect and return the session
+		session = rssh(username, node,
+			server=server,
+			server_username=server_username,
+			port=port,
+			autoconnect=True)
+		return session
+	
+	def parse_args(self):
+		"""
+		Parse command line arguments
+		"""
+		if self._parser is None:
+			self._init_parser()
+		self._args = self._parser.parse_args()
+	
+	def main(self):
+		"""
+		Run the program
+		"""
+		if self._args is None:
+			self.parse_args()
+		args = self._args
+		# version
+		if args.version:
+			description = self.description.splitlines()[0]
+			print(f"{description} version {self.version} (revised {self.date})")
+			print(badwulf_attribution())
+			sys.exit()
+		# open ssh for server commands
+		if args.cmd in ("run", "copy-id", "push", "pull"):
+			con = self.open_ssh(self.resolve_node(args.nodes),
+				username=args.user,
+				server=args.server,
+				server_username=args.login, 
+				port=args.port)
+			sleep(1) # allow time to connect
+		# help
+		if args.cmd is None:
+			self._parser.print_help()
+		# run
+		elif args.cmd == "run":
+			if args.remote_command is None:
+				con.ssh()
+			else:
+				print(f"connecting as {con.username}@{con.destination}")
+				dest = f"{con.username}@{con.hostname}"
+				if con.server is None:
+					cmd = ["ssh", dest]
+				else:
+					cmd = ["ssh", "-o", "NoHostAuthenticationForLocalhost=yes"]
+					cmd += ["-p", str(con.port), dest]
+				cmd.append(args.remote_command)
+				cmd.extend(args.remote_args)
+				subprocess.run(cmd)
+		# copy-id
+		elif args.cmd == "copy-id":
+			con.copy_id(args.identity_file)
+		# push
+		elif args.cmd == "push":
+			con.upload(args.src, args.dest,
+				dry_run=args.dry_run, ask=args.ask)
+		# pull
+		elif args.cmd == "pull":
+			con.download(args.src, args.dest,
+				dry_run=args.dry_run, ask=args.ask)
+		# readme
+		elif args.cmd == "readme":
+			if args.pager is None:
+				cmd = ["glow", "-p", "-w", str(args.width)]
+			else:
+				cmd = [args.pager]
+			cmd += [self.readme]
+			subprocess.run(cmd)
+		sys.exit()