testgres 1.10.0__tar.gz → 1.10.2__tar.gz

testgres-1.10.2/PKG-INFO

@@ -0,0 +1,226 @@
+Metadata-Version: 2.1
+Name: testgres
+Version: 1.10.2
+Summary: Testing utility for PostgreSQL and its extensions
+Home-page: https://github.com/postgrespro/testgres
+Author: Postgres Professional
+Author-email: testgres@postgrespro.ru
+License: PostgreSQL
+Keywords: test,testing,postgresql
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pg8000
+Requires-Dist: port-for>=0.4
+Requires-Dist: six>=1.9.0
+Requires-Dist: psutil
+Requires-Dist: packaging
+
+[![Build Status](https://travis-ci.com/postgrespro/testgres.svg?branch=master)](https://app.travis-ci.com/github/postgrespro/testgres/branches)
+[![codecov](https://codecov.io/gh/postgrespro/testgres/branch/master/graph/badge.svg)](https://codecov.io/gh/postgrespro/testgres)
+[![PyPI version](https://badge.fury.io/py/testgres.svg)](https://badge.fury.io/py/testgres)
+
+[Documentation](https://postgrespro.github.io/testgres/)
+
+# testgres
+
+PostgreSQL testing utility. Both Python 2.7 and 3.3+ are supported.
+
+
+## Installation
+
+To install `testgres`, run:
+
+```
+pip install testgres
+```
+
+We encourage you to use `virtualenv` for your testing environment.
+
+
+## Usage
+
+### Environment
+
+> Note: by default testgres runs `initdb`, `pg_ctl`, `psql` provided by `PATH`.
+
+There are several ways to specify a custom postgres installation:
+
+* export `PG_CONFIG` environment variable pointing to the `pg_config` executable;
+* export `PG_BIN` environment variable pointing to the directory with executable files.
+
+Example:
+
+```bash
+export PG_BIN=$HOME/pg_10/bin
+python my_tests.py
+```
+
+
+### Examples
+
+Here is an example of what you can do with `testgres`:
+
+```python
+# create a node with random name, port, etc
+with testgres.get_new_node() as node:
+
+    # run inidb
+    node.init()
+
+    # start PostgreSQL
+    node.start()
+
+    # execute a query in a default DB
+    print(node.execute('select 1'))
+
+# ... node stops and its files are about to be removed
+```
+
+There are four API methods for runnig queries:
+
+| Command | Description |
+|----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
+| `node.psql(query, ...)` | Runs query via `psql` command and returns tuple `(error code, stdout, stderr)`. |
+| `node.safe_psql(query, ...)` | Same as `psql()` except that it returns only `stdout`. If an error occures during the execution, an exception will be thrown. |
+| `node.execute(query, ...)` | Connects to PostgreSQL using `psycopg2` or `pg8000` (depends on which one is installed in your system) and returns two-dimensional array with data. |
+| `node.connect(dbname, ...)` | Returns connection wrapper (`NodeConnection`) capable of running several queries within a single transaction. |
+
+The last one is the most powerful: you can use `begin(isolation_level)`, `commit()` and `rollback()`:
+```python
+with node.connect() as con:
+    con.begin('serializable')
+    print(con.execute('select %s', 1))
+    con.rollback()
+```
+
+
+### Logging
+
+By default, `cleanup()` removes all temporary files (DB files, logs etc) that were created by testgres' API methods.
+If you'd like to keep logs, execute `configure_testgres(node_cleanup_full=False)` before running any tests.
+
+> Note: context managers (aka `with`) call `stop()` and `cleanup()` automatically.
+
+`testgres` supports [python logging](https://docs.python.org/3.6/library/logging.html),
+which means that you can aggregate logs from several nodes into one file:
+
+```python
+import logging
+
+# write everything to /tmp/testgres.log
+logging.basicConfig(filename='/tmp/testgres.log')
+
+# enable logging, and create two different nodes
+testgres.configure_testgres(use_python_logging=True)
+node1 = testgres.get_new_node().init().start()
+node2 = testgres.get_new_node().init().start()
+
+# execute a few queries
+node1.execute('select 1')
+node2.execute('select 2')
+
+# disable logging
+testgres.configure_testgres(use_python_logging=False)
+```
+
+Look at `tests/test_simple.py` file for a complete example of the logging
+configuration.
+
+
+### Backup & replication
+
+It's quite easy to create a backup and start a new replica:
+
+```python
+with testgres.get_new_node('master') as master:
+    master.init().start()
+
+    # create a backup
+    with master.backup() as backup:
+
+        # create and start a new replica
+        replica = backup.spawn_replica('replica').start()
+
+        # catch up with master node
+        replica.catchup()
+
+        # execute a dummy query
+        print(replica.execute('postgres', 'select 1'))
+```
+
+### Benchmarks
+
+`testgres` is also capable of running benchmarks using `pgbench`:
+
+```python
+with testgres.get_new_node('master') as master:
+    # start a new node
+    master.init().start()
+
+    # initialize default DB and run bench for 10 seconds
+    res = master.pgbench_init(scale=2).pgbench_run(time=10)
+    print(res)
+```
+
+
+### Custom configuration
+
+It's often useful to extend default configuration provided by `testgres`.
+
+`testgres` has `default_conf()` function that helps control some basic
+options. The `append_conf()` function can be used to add custom
+lines to configuration lines:
+
+```python
+ext_conf = "shared_preload_libraries = 'postgres_fdw'"
+
+# initialize a new node
+with testgres.get_new_node().init() as master:
+
+    # ... do something ...
+	
+    # reset main config file
+    master.default_conf(fsync=True,
+                        allow_streaming=True)
+
+    # add a new config line
+    master.append_conf('postgresql.conf', ext_conf)
+```
+
+Note that `default_conf()` is called by `init()` function; both of them overwrite
+the configuration file, which means that they should be called before `append_conf()`.
+
+### Remote mode
+Testgres supports the creation of PostgreSQL nodes on a remote host. This is useful when you want to run distributed tests involving multiple nodes spread across different machines.
+
+To use this feature, you need to use the RemoteOperations class. This feature is only supported with Linux.
+Here is an example of how you might set this up:
+
+```python
+from testgres import ConnectionParams, RemoteOperations, TestgresConfig, get_remote_node
+
+# Set up connection params
+conn_params = ConnectionParams(
+    host='your_host',  # replace with your host
+    username='user_name',  # replace with your username
+    ssh_key='path_to_ssh_key'  # replace with your SSH key path
+)
+os_ops = RemoteOperations(conn_params)
+
+# Add remote testgres config before test
+TestgresConfig.set_os_ops(os_ops=os_ops)
+
+# Proceed with your test
+def test_basic_query(self):
+    with get_remote_node(conn_params=conn_params) as node:
+        node.init().start()
+        res = node.execute('SELECT 1')
+        self.assertEqual(res, [(1,)])
+```
+
+## Authors
+
+[Ildar Musin](https://github.com/zilder)  
+[Dmitry Ivanov](https://github.com/funbringer)  
+[Ildus Kurbangaliev](https://github.com/ildus)  
+[Yury Zhuravlev](https://github.com/stalkerg)  

testgres-1.10.2/setup.py

@@ -0,0 +1,43 @@
+import sys
+
+try:
+    from setuptools import setup
+except ImportError:
+    from distutils.core import setup
+
+# Basic dependencies
+install_requires = [
+    "pg8000",
+    "port-for>=0.4",
+    "six>=1.9.0",
+    "psutil",
+    "packaging"
+]
+
+# Add compatibility enum class
+if sys.version_info < (3, 4):
+    install_requires.append("enum34")
+
+# Add compatibility ipaddress module
+if sys.version_info < (3, 3):
+    install_requires.append("ipaddress")
+
+# Get contents of README file
+with open('README.md', 'r') as f:
+    readme = f.read()
+
+setup(
+    version='1.10.2',
+    name='testgres',
+    packages=['testgres', 'testgres.operations', 'testgres.helpers'],
+    description='Testing utility for PostgreSQL and its extensions',
+    url='https://github.com/postgrespro/testgres',
+    long_description=readme,
+    long_description_content_type='text/markdown',
+    license='PostgreSQL',
+    author='Postgres Professional',
+    author_email='testgres@postgrespro.ru',
+    keywords=['test', 'testing', 'postgresql'],
+    install_requires=install_requires,
+    classifiers=[],
+)

testgres-1.10.2/testgres/backup.py

@@ -0,0 +1,188 @@
+# coding: utf-8
+
+import os
+
+from six import raise_from
+
+from .enums import XLogMethod
+
+from .consts import \
+    DATA_DIR, \
+    TMP_NODE, \
+    TMP_BACKUP, \
+    PG_CONF_FILE, \
+    BACKUP_LOG_FILE
+
+from .exceptions import BackupException
+
+from .utils import \
+    get_bin_path, \
+    execute_utility, \
+    clean_on_error
+
+
+class NodeBackup(object):
+    """
+    Smart object responsible for backups
+    """
+    @property
+    def log_file(self):
+        return os.path.join(self.base_dir, BACKUP_LOG_FILE)
+
+    def __init__(self,
+                 node,
+                 base_dir=None,
+                 username=None,
+                 xlog_method=XLogMethod.fetch,
+                 options=None):
+        """
+        Create a new backup.
+
+        Args:
+            node: :class:`.PostgresNode` we're going to backup.
+            base_dir: where should we store it?
+            username: database user name.
+            xlog_method: none | fetch | stream (see docs)
+        """
+        if not options:
+            options = []
+        self.os_ops = node.os_ops
+        if not node.status():
+            raise BackupException('Node must be running')
+
+        # Check arguments
+        if not isinstance(xlog_method, XLogMethod):
+            try:
+                xlog_method = XLogMethod(xlog_method)
+            except ValueError:
+                msg = 'Invalid xlog_method "{}"'.format(xlog_method)
+                raise BackupException(msg)
+
+        # Set default arguments
+        username = username or self.os_ops.get_user()
+        base_dir = base_dir or self.os_ops.mkdtemp(prefix=TMP_BACKUP)
+
+        # public
+        self.original_node = node
+        self.base_dir = base_dir
+        self.username = username
+
+        # private
+        self._available = True
+
+        data_dir = os.path.join(self.base_dir, DATA_DIR)
+
+        _params = [
+            get_bin_path("pg_basebackup"),
+            "-p", str(node.port),
+            "-h", node.host,
+            "-U", username,
+            "-D", data_dir,
+            "-X", xlog_method.value
+        ]  # yapf: disable
+        _params += options
+        execute_utility(_params, self.log_file)
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, type, value, traceback):
+        self.cleanup()
+
+    def _prepare_dir(self, destroy):
+        """
+        Provide a data directory for a copy of node.
+
+        Args:
+            destroy: should we convert this backup into a node?
+
+        Returns:
+            Path to data directory.
+        """
+
+        if not self._available:
+            raise BackupException('Backup is exhausted')
+
+        # Do we want to use this backup several times?
+        available = not destroy
+
+        if available:
+            dest_base_dir = self.os_ops.mkdtemp(prefix=TMP_NODE)
+
+            data1 = os.path.join(self.base_dir, DATA_DIR)
+            data2 = os.path.join(dest_base_dir, DATA_DIR)
+
+            try:
+                # Copy backup to new data dir
+                self.os_ops.copytree(data1, data2)
+            except Exception as e:
+                raise_from(BackupException('Failed to copy files'), e)
+        else:
+            dest_base_dir = self.base_dir
+
+        # Is this backup exhausted?
+        self._available = available
+
+        # Return path to new node
+        return dest_base_dir
+
+    def spawn_primary(self, name=None, destroy=True):
+        """
+        Create a primary node from a backup.
+
+        Args:
+            name: primary's application name.
+            destroy: should we convert this backup into a node?
+
+        Returns:
+            New instance of :class:`.PostgresNode`.
+        """
+
+        # Prepare a data directory for this node
+        base_dir = self._prepare_dir(destroy)
+
+        # Build a new PostgresNode
+        NodeClass = self.original_node.__class__
+        with clean_on_error(NodeClass(name=name, base_dir=base_dir, conn_params=self.original_node.os_ops.conn_params)) as node:
+
+            # New nodes should always remove dir tree
+            node._should_rm_dirs = True
+
+            # Set a new port
+            node.append_conf(filename=PG_CONF_FILE, line='\n')
+            node.append_conf(filename=PG_CONF_FILE, port=node.port)
+
+            return node
+
+    def spawn_replica(self, name=None, destroy=True, slot=None):
+        """
+        Create a replica of the original node from a backup.
+
+        Args:
+            name: replica's application name.
+            slot: create a replication slot with the specified name.
+            destroy: should we convert this backup into a node?
+
+        Returns:
+            New instance of :class:`.PostgresNode`.
+        """
+
+        # Build a new PostgresNode
+        with clean_on_error(self.spawn_primary(name=name,
+                                               destroy=destroy)) as node:
+
+            # Assign it a master and a recovery file (private magic)
+            node._assign_master(self.original_node)
+            node._create_recovery_conf(username=self.username, slot=slot)
+
+            return node
+
+    def cleanup(self):
+        """
+        Remove all files that belong to this backup.
+        No-op if it's been converted to a PostgresNode (destroy=True).
+        """
+
+        if self._available:
+            self._available = False
+            self.os_ops.rmdirs(self.base_dir, ignore_errors=True)

testgres-1.10.2/testgres/config.py

@@ -0,0 +1,216 @@
+# coding: utf-8
+
+import atexit
+import copy
+import logging
+import os
+import tempfile
+
+from contextlib import contextmanager
+
+from .consts import TMP_CACHE
+from .operations.os_ops import OsOperations
+from .operations.local_ops import LocalOperations
+
+log_level = os.getenv('LOGGING_LEVEL', 'WARNING').upper()
+log_format = os.getenv('LOGGING_FORMAT', '%(asctime)s - %(levelname)s - %(message)s')
+logging.basicConfig(level=log_level, format=log_format)
+
+
+class GlobalConfig(object):
+    """
+    Global configuration object which allows user to override default settings.
+    """
+    # NOTE: attributes must not be callable or begin with __.
+
+    cache_initdb = True
+    """ shall we use cached initdb instance? """
+
+    cached_initdb_unique = False
+    """ shall we give new node a unique system id? """
+
+    cache_pg_config = True
+    """ shall we cache pg_config results? """
+
+    use_python_logging = False
+    """ enable python logging subsystem (see logger.py). """
+
+    error_log_lines = 20
+    """ N of log lines to be shown in exceptions (0=inf). """
+
+    node_cleanup_full = True
+    """ shall we remove EVERYTHING (including logs)? """
+
+    node_cleanup_on_good_exit = True
+    """ remove base_dir on nominal __exit__(). """
+
+    node_cleanup_on_bad_exit = False
+    """ remove base_dir on __exit__() via exception. """
+
+    _cached_initdb_dir = None
+    """ underlying class attribute for cached_initdb_dir property """
+
+    os_ops = LocalOperations()
+    """ OsOperation object that allows work on remote host """
+    @property
+    def cached_initdb_dir(self):
+        """ path to a temp directory for cached initdb. """
+        return self._cached_initdb_dir
+
+    @cached_initdb_dir.setter
+    def cached_initdb_dir(self, value):
+        self._cached_initdb_dir = value
+
+        if value:
+            cached_initdb_dirs.add(value)
+        return testgres_config.cached_initdb_dir
+
+    @property
+    def temp_dir(self):
+        """ path to temp dir containing nodes with default 'base_dir'. """
+        return tempfile.tempdir
+
+    @temp_dir.setter
+    def temp_dir(self, value):
+        tempfile.tempdir = value
+
+    def __init__(self, **options):
+        self.update(options)
+
+    def __setitem__(self, key, value):
+        setattr(self, key, value)
+
+    def __getitem__(self, key):
+        return getattr(self, key)
+
+    def __setattr__(self, name, value):
+        if name not in self.keys():
+            raise TypeError('Unknown option {}'.format(name))
+
+        super(GlobalConfig, self).__setattr__(name, value)
+
+    def keys(self):
+        """
+        Return a list of all available settings.
+        """
+
+        keys = []
+
+        for key in dir(GlobalConfig):
+            if not key.startswith('__') and not callable(self[key]):
+                keys.append(key)
+
+        return keys
+
+    def items(self):
+        """
+        Return setting-value pairs.
+        """
+
+        return ((key, self[key]) for key in self.keys())
+
+    def update(self, config):
+        """
+        Extract setting-value pairs from 'config' and
+        assign those values to corresponding settings
+        of this GlobalConfig object.
+        """
+
+        for key, value in config.items():
+            self[key] = value
+
+        return self
+
+    def copy(self):
+        """
+        Return a copy of this object.
+        """
+
+        return copy.copy(self)
+
+    @staticmethod
+    def set_os_ops(os_ops: OsOperations):
+        testgres_config.os_ops = os_ops
+        testgres_config.cached_initdb_dir = os_ops.mkdtemp(prefix=TMP_CACHE)
+
+
+# cached dirs to be removed
+cached_initdb_dirs = set()
+
+# default config object
+testgres_config = GlobalConfig()
+
+# NOTE: for compatibility
+TestgresConfig = testgres_config
+
+# stack of GlobalConfigs
+config_stack = []
+
+
+@atexit.register
+def _rm_cached_initdb_dirs():
+    for d in cached_initdb_dirs:
+        testgres_config.os_ops.rmdirs(d, ignore_errors=True)
+
+
+def push_config(**options):
+    """
+    Permanently set custom GlobalConfig options and
+    put previous settings on top of the config stack.
+    """
+
+    # push current config to stack
+    config_stack.append(testgres_config.copy())
+
+    return testgres_config.update(options)
+
+
+def pop_config():
+    """
+    Set previous GlobalConfig options from stack.
+    """
+
+    if len(config_stack) == 0:
+        raise IndexError('Reached initial config')
+
+    # restore popped config
+    return testgres_config.update(config_stack.pop())
+
+
+@contextmanager
+def scoped_config(**options):
+    """
+    Temporarily set custom GlobalConfig options for this context.
+    Previous options are pushed to the config stack.
+
+    Example:
+        >>> from .api import get_new_node
+        >>> with scoped_config(cache_initdb=False):
+        ...     # create a new node with fresh initdb
+        ...     with get_new_node().init().start() as node:
+        ...         print(node.execute('select 1'))
+        [(1,)]
+    """
+
+    try:
+        # set a new config with options
+        config = push_config(**options)
+
+        # return it
+        yield config
+    finally:
+        # restore previous config
+        pop_config()
+
+
+def configure_testgres(**options):
+    """
+    Adjust current global options.
+    Look at the GlobalConfig to learn about existing settings.
+    """
+
+    testgres_config.update(options)
+
+
+# NOTE: assign initial cached dir for initdb
+testgres_config.cached_initdb_dir = testgres_config.os_ops.mkdtemp(prefix=TMP_CACHE)