sqlutilpy 0.21.0__tar.gz → 0.23.0__tar.gz

{sqlutilpy-0.21.0/py/sqlutilpy.egg-info → sqlutilpy-0.23.0}/PKG-INFO

@@ -1,20 +1,21 @@
 Metadata-Version: 2.1
 Name: sqlutilpy
-Version: 0.21.0
+Version: 0.23.0
 Summary: Database query code returning numpy arrays
-Home-page: https://github.com/segasai/sqlutilpy
-Author: Sergey Koposov
-Author-email: skoposov@ed.ac.uk
+Author-email: "Sergey E. Koposov" <skoposov@ed.ac.uk>
 License: BSD
+Project-URL: Homepage, https://github.com/segasai/sqlutilpy
 Keywords: numpy postgresql query sql sqlite array
 Classifier: Development Status :: 3 - Alpha
-Classifier: Topic :: Utilities
 Classifier: License :: OSI Approved :: BSD License
+Classifier: Topic :: Utilities
+Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
 Requires-Dist: numpy
-Requires-Dist: psycopg2-binary
+Requires-Dist: psycopg[binary]
 Requires-Dist: numpydoc
+Requires-Dist: numpy
 
 [![Build Status](https://github.com/segasai/sqlutilpy/workflows/Testing/badge.svg)](https://github.com/segasai/sqlutilpy/actions)
 [![Documentation Status](https://readthedocs.org/projects/sqlutilpy/badge/?version=latest)](http://sqlutilpy.readthedocs.io/en/latest/?badge=latest)
@@ -24,7 +25,8 @@ Requires-Dist: numpydoc
 # sqlutilpy
 Python module to query SQL databases and return numpy arrays, upload
 tables and run join queries involving local arrays and the tables in the DB.
-The module only works with PostgreSQL and SQLite databases.
+This module is optimized to be able to deal efficiently with query results with millions of rows.
+The module only works with PostgreSQL, SQLite and DuckDB databases.
 
 The full documentation is available [here](http://sqlutilpy.readthedocs.io/en/latest/)
 
@@ -36,10 +38,11 @@ To install the package you just need to do pip install.
 ```
 pip install sqlutilpy
 ```
-## Authentification
-Throughout this readme, I'll assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
-has been created with the login/password details for Postgresql. If that is not the case, all of the 
-commands given above will also need user='....' and password='...' options
+## Authentication
+
+Throughout this readme, I will assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
+has been created with your login/password details for Postgresql. If that is not the case, all of the 
+commands given below will also need user='....' and password='...' options
 
 ## Connection information
 
@@ -50,6 +53,8 @@ of the connection.
 
 
 ## Querying the database and retrieving the results
+
+This command will run the query and put the columns into variables ra,dec
 ```python
 import sqlutilpy
 ra,dec = squtilpy.get('select ra,dec from mytable', 
@@ -57,10 +62,11 @@ ra,dec = squtilpy.get('select ra,dec from mytable',
                  db='THE_NAME_OF_MY_DB')
 ```
 
-By default sqlutilpy.get executes the result and returns the tuple of 
-results. But you can return the results as dictionary using asDict option.
+By default sqlutilpy.get executes the query and returns the tuple with 
+results. You can return the results as dictionary using asDict option.
 
 ## Uploading your arrays as column in a table
+
 ```python
 x = np.arange(10)                                                   
 y = x**.5                                                           
@@ -76,20 +82,28 @@ you could upload the arrays in the DB and run a query, but local_join function d
 
 ```python
 myid = np.arange(10)
-y = x**.5
+y = np.random.uniform(size=10)
+
 R=sqlutilpy.local_join('''select * from mytmptable as m, 
-           somebigtable as s where s.id=m.myid order by m.myid''',                                                                       'mytmptable',(x,y),('myid','ycol'))
+           somebigtable as s where s.id=m.myid order by m.myid''',                                              
+           'mytmptable',(myid, y),('myid','ycol'))
 ```
-It executes a query as if you arrays where in a mytmptable. ( behind the scenes
-it uploads the data to the db and runs a query)
 
+It executes a query as if you arrays were in mytmptable. (behind the scenes
+it uploads the data to the db and runs a query)
 
 ## Keeping the connection open. 
 
-Often it's benefitial to preserve an open connection. You can do that if you first 
+Often it is beneficial to preserve an open connection to the database. You can do that if you first 
 obtain the connection using sqlutilpy.getConnection() and then provide it directly
 to sqlutil.get() and friends using conn=conn argument
+```python
+conn = sqlutilpy.getConnection(db='mydb', user='meuser', password='something', host='hostname')
+R= sqlutilpy.get('select 1', conn=conn)
+R1= sqlutilpy.get('select 1', conn=conn)
+```
+
+# How to cite the software
 
+If you use this package, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957
 
-# How to cite it
-If you use this, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957

{sqlutilpy-0.21.0 → sqlutilpy-0.23.0}/README.md

@@ -6,7 +6,8 @@
 # sqlutilpy
 Python module to query SQL databases and return numpy arrays, upload
 tables and run join queries involving local arrays and the tables in the DB.
-The module only works with PostgreSQL and SQLite databases.
+This module is optimized to be able to deal efficiently with query results with millions of rows.
+The module only works with PostgreSQL, SQLite and DuckDB databases.
 
 The full documentation is available [here](http://sqlutilpy.readthedocs.io/en/latest/)
 
@@ -18,10 +19,11 @@ To install the package you just need to do pip install.
 ```
 pip install sqlutilpy
 ```
-## Authentification
-Throughout this readme, I'll assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
-has been created with the login/password details for Postgresql. If that is not the case, all of the 
-commands given above will also need user='....' and password='...' options
+## Authentication
+
+Throughout this readme, I will assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
+has been created with your login/password details for Postgresql. If that is not the case, all of the 
+commands given below will also need user='....' and password='...' options
 
 ## Connection information
 
@@ -32,6 +34,8 @@ of the connection.
 
 
 ## Querying the database and retrieving the results
+
+This command will run the query and put the columns into variables ra,dec
 ```python
 import sqlutilpy
 ra,dec = squtilpy.get('select ra,dec from mytable', 
@@ -39,10 +43,11 @@ ra,dec = squtilpy.get('select ra,dec from mytable',
                  db='THE_NAME_OF_MY_DB')
 ```
 
-By default sqlutilpy.get executes the result and returns the tuple of 
-results. But you can return the results as dictionary using asDict option.
+By default sqlutilpy.get executes the query and returns the tuple with 
+results. You can return the results as dictionary using asDict option.
 
 ## Uploading your arrays as column in a table
+
 ```python
 x = np.arange(10)                                                   
 y = x**.5                                                           
@@ -58,20 +63,28 @@ you could upload the arrays in the DB and run a query, but local_join function d
 
 ```python
 myid = np.arange(10)
-y = x**.5
+y = np.random.uniform(size=10)
+
 R=sqlutilpy.local_join('''select * from mytmptable as m, 
-           somebigtable as s where s.id=m.myid order by m.myid''',                                                                       'mytmptable',(x,y),('myid','ycol'))
+           somebigtable as s where s.id=m.myid order by m.myid''',                                              
+           'mytmptable',(myid, y),('myid','ycol'))
 ```
-It executes a query as if you arrays where in a mytmptable. ( behind the scenes
-it uploads the data to the db and runs a query)
 
+It executes a query as if you arrays were in mytmptable. (behind the scenes
+it uploads the data to the db and runs a query)
 
 ## Keeping the connection open. 
 
-Often it's benefitial to preserve an open connection. You can do that if you first 
+Often it is beneficial to preserve an open connection to the database. You can do that if you first 
 obtain the connection using sqlutilpy.getConnection() and then provide it directly
 to sqlutil.get() and friends using conn=conn argument
+```python
+conn = sqlutilpy.getConnection(db='mydb', user='meuser', password='something', host='hostname')
+R= sqlutilpy.get('select 1', conn=conn)
+R1= sqlutilpy.get('select 1', conn=conn)
+```
+
+# How to cite the software
 
+If you use this package, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957
 
-# How to cite it
-If you use this, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957

sqlutilpy-0.23.0/pyproject.toml

@@ -0,0 +1,33 @@
+[project]
+name = "sqlutilpy"
+dynamic = ["version"] 
+description = "Database query code returning numpy arrays"
+authors = [
+    {name = "Sergey E. Koposov", email = "skoposov@ed.ac.uk"},
+]
+dependencies = [
+    "numpy",
+    "psycopg[binary]",
+    "numpydoc",
+    "numpy\n"
+]
+requires-python = ">=3.8"
+readme = "README.md"
+license = {text = "BSD"}
+keywords = ["numpy postgresql query sql sqlite array"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "License :: OSI Approved :: BSD License",
+    "Topic :: Utilities",
+]
+
+[build-system]
+requires = ["setuptools", "wheel", "setuptools-git-versioning"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools-git-versioning]
+enabled = true
+
+[project.urls]
+Homepage = "https://github.com/segasai/sqlutilpy"
+

sqlutilpy-0.23.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

{sqlutilpy-0.21.0/py → sqlutilpy-0.23.0/src}/sqlutilpy/sqlutil.py

@@ -3,13 +3,11 @@
 from __future__ import print_function
 import numpy
 import numpy as np
-import psycopg2
+import psycopg
 import threading
 import collections
 import warnings
 from numpy.core import numeric as sb
-from select import select
-from psycopg2.extensions import POLL_OK, POLL_READ, POLL_WRITE
 
 try:
     import astropy.table as atpy
@@ -22,7 +20,6 @@ except ImportError:
     # pandas is not installed
     pandas = None
 
-from io import BytesIO as StringIO
 import queue
 
 _WAIT_SELECT_TIMEOUT = 10
@@ -37,31 +34,6 @@ class SqlUtilException(Exception):
     pass
 
 
-def __wait_select_inter(conn):
-    """ Make the queries interruptible by Ctrl-C
-
-    Taken from http://initd.org/psycopg/articles/2014/07/20/cancelling-postgresql-statements-python/ # noqa
-    """
-    while True:
-        try:
-            state = conn.poll()
-            if state == POLL_OK:
-                break
-            elif state == POLL_READ:
-                select([conn.fileno()], [], [], _WAIT_SELECT_TIMEOUT)
-            elif state == POLL_WRITE:
-                select([], [conn.fileno()], [], _WAIT_SELECT_TIMEOUT)
-            else:
-                raise conn.OperationalError("bad state from poll: %s" % state)
-        except KeyboardInterrupt:
-            conn.cancel()
-            # the loop will be broken by a server error
-            continue
-
-
-psycopg2.extensions.set_wait_callback(__wait_select_inter)
-
-
 def getConnection(db=None,
                   driver=None,
                   user=None,
@@ -80,7 +52,7 @@ def getConnection(db=None,
         The name of the database (in case of PostgreSQL) or filename in
         case of sqlite db
     driver :  string
-        The db driver (either 'psycopg2' or 'sqlite3')
+        The db driver (either 'psycopg' or 'sqlite3')
     user : string, optional
         Username
     password: string, optional
@@ -99,6 +71,10 @@ def getConnection(db=None,
 
     """
     if driver == 'psycopg2':
+        warnings.warn(
+            'psycopg2 driver is not supported anymore using psycopg instead')
+        driver = 'psycopg'
+    if driver == 'psycopg':
         conn_dict = dict()
         if db is not None:
             conn_dict['dbname'] = db
@@ -110,7 +86,7 @@ def getConnection(db=None,
             conn_dict['user'] = user
         if password is not None:
             conn_dict['password'] = password
-        conn = psycopg2.connect(**conn_dict)
+        conn = psycopg.connect(**conn_dict)
     elif driver == 'sqlite3':
         import sqlite3
         if timeout is None:
@@ -126,6 +102,10 @@ def getCursor(conn, driver=None, preamb=None, notNamed=False):
     Retrieve the database cursor
     """
     if driver == 'psycopg2':
+        warnings.warn(
+            'psycopg2 driver is not supported anymore using psycopg instead')
+        driver = 'psycopg'
+    if driver == 'psycopg':
         cur = conn.cursor()
         if preamb is not None:
             cur.execute(preamb)
@@ -267,7 +247,7 @@ def __getDType(row, typeCodes, strLength):
 def get(query,
         params=None,
         db="wsdb",
-        driver="psycopg2",
+        driver="psycopg",
         user=None,
         password=None,
         host=None,
@@ -304,7 +284,7 @@ def get(query,
     db : string
         The name of the database
     driver : string, optional
-        The sql driver to be used (psycopg2 or sqlite3)
+        The sql driver to be used (psycopg or sqlite3)
     user : string, optional
         User name for the DB connection
     password : string, optional
@@ -359,6 +339,11 @@ def get(query,
         proc = None
         colNames = []
         if driver == 'psycopg2':
+            warnings.warn(
+                'psycopg2 driver is not supported anymore using psycopg instead'
+            )
+            driver = 'psycopg'
+        if driver == 'psycopg':
             try:
                 while (True):
                     # Iterating over the cursor, retrieving batches of results
@@ -473,7 +458,7 @@ def get(query,
 def execute(query,
             params=None,
             db='wsdb',
-            driver="psycopg2",
+            driver="psycopg",
             user=None,
             password=None,
             host=None,
@@ -493,7 +478,7 @@ def execute(query,
     db : string
         Database name
     driver : string
-        Driver for the DB connection ('psycopg2' or 'sqlite3')
+        Driver for the DB connection ('psycopg' or 'sqlite3')
     user : string, optional
         user name for the DB connection
     password : string, optional
@@ -542,7 +527,7 @@ def __create_schema(tableName, arrays, names, temp=False):
                  (np.uint64, 'bigint'), (np.int16, 'smallint'),
                  (np.uint8, 'smallint'), (np.int8, 'smallint'),
                  (np.float32, 'real'), (np.float64, 'double precision'),
-                 (np.string_, 'varchar'), (np.str_, 'varchar'),
+                 (np.bytes_, 'varchar'), (np.str_, 'varchar'),
                  (np.bool_, 'boolean'), (np.datetime64, 'timestamp')])
     if temp:
         temp = 'temporary'
@@ -568,7 +553,7 @@ def __print_arrays(arrays, f, delimiter=' '):
     format_dict = dict([(np.int32, '%d'), (np.int64, '%d'), (np.int16, '%d'),
                         (np.int8, '%d'), (np.uint8, '%d'),
                         (np.float32, '%.18e'), (np.float64, '%.18e'),
-                        (np.string_, '%s'), (np.str_, '%s'),
+                        (np.bytes_, '%s'), (np.str_, '%s'),
                         (np.datetime64, '%s'), (np.bool_, '%d')])
     fmts = []
     array_mode = False
@@ -626,7 +611,7 @@ def upload(tableName,
            arrays,
            names=None,
            db="wsdb",
-           driver="psycopg2",
+           driver="psycopg",
            user=None,
            password=None,
            host=None,
@@ -653,7 +638,7 @@ def upload(tableName,
     db: string
          Databas name
     driver: string
-         Python database driver "psycopg2",
+         Python database driver "psycopg",
     user: string,
     password: string
     host: string
@@ -743,18 +728,18 @@ table/pandas/dictionary or provide a separate list of arrays and their names')
             cur.execute(query1)
         nsplit = 100000
         N = len(arrays[0])
+        names = ','.join(names)
         for i in range(0, N, nsplit):
-            f = StringIO()
-            __print_arrays([_[i:i + nsplit] for _ in arrays],
-                           f,
-                           delimiter=delimiter)
-            f.seek(0)
             try:
-                thread = psycopg2.extensions.get_wait_callback()
-                psycopg2.extensions.set_wait_callback(None)
-                cur.copy_from(f, tableName, sep=delimiter, columns=names)
+                with cur.copy(
+                        f'''copy {tableName}({names}) from stdin with delimiter '{delimiter}' '''
+                ) as copy:
+                    __print_arrays([_[i:i + nsplit] for _ in arrays],
+                                   copy,
+                                   delimiter=delimiter)
+
             finally:
-                psycopg2.extensions.set_wait_callback(thread)
+                pass
     except BaseException:
         failure_cleanup(conn, connSupplied)
         raise
@@ -772,7 +757,7 @@ def local_join(query,
                arrays,
                names,
                db=None,
-               driver="psycopg2",
+               driver="psycopg",
                user=None,
                password=None,
                host=None,

sqlutilpy-0.23.0/src/sqlutilpy/version.py

@@ -0,0 +1 @@
+__version__ = '0.23.0'

{sqlutilpy-0.21.0 → sqlutilpy-0.23.0/src/sqlutilpy.egg-info}/PKG-INFO

@@ -1,20 +1,21 @@
 Metadata-Version: 2.1
 Name: sqlutilpy
-Version: 0.21.0
+Version: 0.23.0
 Summary: Database query code returning numpy arrays
-Home-page: https://github.com/segasai/sqlutilpy
-Author: Sergey Koposov
-Author-email: skoposov@ed.ac.uk
+Author-email: "Sergey E. Koposov" <skoposov@ed.ac.uk>
 License: BSD
+Project-URL: Homepage, https://github.com/segasai/sqlutilpy
 Keywords: numpy postgresql query sql sqlite array
 Classifier: Development Status :: 3 - Alpha
-Classifier: Topic :: Utilities
 Classifier: License :: OSI Approved :: BSD License
+Classifier: Topic :: Utilities
+Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
 Requires-Dist: numpy
-Requires-Dist: psycopg2-binary
+Requires-Dist: psycopg[binary]
 Requires-Dist: numpydoc
+Requires-Dist: numpy
 
 [![Build Status](https://github.com/segasai/sqlutilpy/workflows/Testing/badge.svg)](https://github.com/segasai/sqlutilpy/actions)
 [![Documentation Status](https://readthedocs.org/projects/sqlutilpy/badge/?version=latest)](http://sqlutilpy.readthedocs.io/en/latest/?badge=latest)
@@ -24,7 +25,8 @@ Requires-Dist: numpydoc
 # sqlutilpy
 Python module to query SQL databases and return numpy arrays, upload
 tables and run join queries involving local arrays and the tables in the DB.
-The module only works with PostgreSQL and SQLite databases.
+This module is optimized to be able to deal efficiently with query results with millions of rows.
+The module only works with PostgreSQL, SQLite and DuckDB databases.
 
 The full documentation is available [here](http://sqlutilpy.readthedocs.io/en/latest/)
 
@@ -36,10 +38,11 @@ To install the package you just need to do pip install.
 ```
 pip install sqlutilpy
 ```
-## Authentification
-Throughout this readme, I'll assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
-has been created with the login/password details for Postgresql. If that is not the case, all of the 
-commands given above will also need user='....' and password='...' options
+## Authentication
+
+Throughout this readme, I will assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
+has been created with your login/password details for Postgresql. If that is not the case, all of the 
+commands given below will also need user='....' and password='...' options
 
 ## Connection information
 
@@ -50,6 +53,8 @@ of the connection.
 
 
 ## Querying the database and retrieving the results
+
+This command will run the query and put the columns into variables ra,dec
 ```python
 import sqlutilpy
 ra,dec = squtilpy.get('select ra,dec from mytable', 
@@ -57,10 +62,11 @@ ra,dec = squtilpy.get('select ra,dec from mytable',
                  db='THE_NAME_OF_MY_DB')
 ```
 
-By default sqlutilpy.get executes the result and returns the tuple of 
-results. But you can return the results as dictionary using asDict option.
+By default sqlutilpy.get executes the query and returns the tuple with 
+results. You can return the results as dictionary using asDict option.
 
 ## Uploading your arrays as column in a table
+
 ```python
 x = np.arange(10)                                                   
 y = x**.5                                                           
@@ -76,20 +82,28 @@ you could upload the arrays in the DB and run a query, but local_join function d
 
 ```python
 myid = np.arange(10)
-y = x**.5
+y = np.random.uniform(size=10)
+
 R=sqlutilpy.local_join('''select * from mytmptable as m, 
-           somebigtable as s where s.id=m.myid order by m.myid''',                                                                       'mytmptable',(x,y),('myid','ycol'))
+           somebigtable as s where s.id=m.myid order by m.myid''',                                              
+           'mytmptable',(myid, y),('myid','ycol'))
 ```
-It executes a query as if you arrays where in a mytmptable. ( behind the scenes
-it uploads the data to the db and runs a query)
 
+It executes a query as if you arrays were in mytmptable. (behind the scenes
+it uploads the data to the db and runs a query)
 
 ## Keeping the connection open. 
 
-Often it's benefitial to preserve an open connection. You can do that if you first 
+Often it is beneficial to preserve an open connection to the database. You can do that if you first 
 obtain the connection using sqlutilpy.getConnection() and then provide it directly
 to sqlutil.get() and friends using conn=conn argument
+```python
+conn = sqlutilpy.getConnection(db='mydb', user='meuser', password='something', host='hostname')
+R= sqlutilpy.get('select 1', conn=conn)
+R1= sqlutilpy.get('select 1', conn=conn)
+```
+
+# How to cite the software
 
+If you use this package, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957
 
-# How to cite it
-If you use this, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957

sqlutilpy-0.23.0/src/sqlutilpy.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+LICENSE.txt
+MANIFEST.in
+README.md
+pyproject.toml
+src/sqlutilpy/__init__.py
+src/sqlutilpy/sqlutil.py
+src/sqlutilpy/version.py
+src/sqlutilpy.egg-info/PKG-INFO
+src/sqlutilpy.egg-info/SOURCES.txt
+src/sqlutilpy.egg-info/dependency_links.txt
+src/sqlutilpy.egg-info/requires.txt
+src/sqlutilpy.egg-info/top_level.txt
+tests/killer.py
+tests/test_pg.py
+tests/test_sqlite.py

sqlutilpy-0.23.0/src/sqlutilpy.egg-info/requires.txt

@@ -0,0 +1,4 @@
+numpy
+psycopg[binary]
+numpydoc
+numpy