sqlutilpy 0.20.0__tar.gz → 0.22.0__tar.gz

{sqlutilpy-0.20.0/py/sqlutilpy.egg-info → sqlutilpy-0.22.0}/PKG-INFO

@@ -1,17 +1,20 @@
 Metadata-Version: 2.1
 Name: sqlutilpy
-Version: 0.20.0
+Version: 0.22.0
 Summary: Database query code returning numpy arrays
-Home-page: https://github.com/segasai/sqlutilpy
-Author: Sergey Koposov
-Author-email: skoposov@ed.ac.uk
+Author-email: "Sergey E. Koposov" <skoposov@ed.ac.uk>
 License: BSD
+Project-URL: Homepage, https://github.com/segasai/sqlutilpy
 Keywords: numpy postgresql query sql sqlite array
 Classifier: Development Status :: 3 - Alpha
-Classifier: Topic :: Utilities
 Classifier: License :: OSI Approved :: BSD License
+Classifier: Topic :: Utilities
+Requires-Python: >=3.6
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
+Requires-Dist: numpy
+Requires-Dist: psycopg2-binary
+Requires-Dist: numpydoc
 
 [![Build Status](https://github.com/segasai/sqlutilpy/workflows/Testing/badge.svg)](https://github.com/segasai/sqlutilpy/actions)
 [![Documentation Status](https://readthedocs.org/projects/sqlutilpy/badge/?version=latest)](http://sqlutilpy.readthedocs.io/en/latest/?badge=latest)
@@ -21,7 +24,8 @@ License-File: LICENSE.txt
 # sqlutilpy
 Python module to query SQL databases and return numpy arrays, upload
 tables and run join queries involving local arrays and the tables in the DB.
-The module only works with PostgreSQL and SQLite databases.
+This module is optimized to be able to deal efficiently with query results with millions of rows.
+The module only works with PostgreSQL, SQLite and DuckDB databases.
 
 The full documentation is available [here](http://sqlutilpy.readthedocs.io/en/latest/)
 
@@ -33,10 +37,11 @@ To install the package you just need to do pip install.
 ```
 pip install sqlutilpy
 ```
-## Authentification
-Throughout this readme, I'll assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
-has been created with the login/password details for Postgresql. If that is not the case, all of the 
-commands given above will also need user='....' and password='...' options
+## Authentication
+
+Throughout this readme, I will assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
+has been created with your login/password details for Postgresql. If that is not the case, all of the 
+commands given below will also need user='....' and password='...' options
 
 ## Connection information
 
@@ -47,6 +52,8 @@ of the connection.
 
 
 ## Querying the database and retrieving the results
+
+This command will run the query and put the columns into variables ra,dec
 ```python
 import sqlutilpy
 ra,dec = squtilpy.get('select ra,dec from mytable', 
@@ -54,10 +61,11 @@ ra,dec = squtilpy.get('select ra,dec from mytable',
                  db='THE_NAME_OF_MY_DB')
 ```
 
-By default sqlutilpy.get executes the result and returns the tuple of 
-results. But you can return the results as dictionary using asDict option.
+By default sqlutilpy.get executes the query and returns the tuple with 
+results. You can return the results as dictionary using asDict option.
 
 ## Uploading your arrays as column in a table
+
 ```python
 x = np.arange(10)                                                   
 y = x**.5                                                           
@@ -73,20 +81,28 @@ you could upload the arrays in the DB and run a query, but local_join function d
 
 ```python
 myid = np.arange(10)
-y = x**.5
+y = np.random.uniform(size=10)
+
 R=sqlutilpy.local_join('''select * from mytmptable as m, 
-           somebigtable as s where s.id=m.myid order by m.myid''',                                                                       'mytmptable',(x,y),('myid','ycol'))
+           somebigtable as s where s.id=m.myid order by m.myid''',                                              
+           'mytmptable',(myid, y),('myid','ycol'))
 ```
-It executes a query as if you arrays where in a mytmptable. ( behind the scenes
-it uploads the data to the db and runs a query)
 
+It executes a query as if you arrays were in mytmptable. (behind the scenes
+it uploads the data to the db and runs a query)
 
 ## Keeping the connection open. 
 
-Often it's benefitial to preserve an open connection. You can do that if you first 
+Often it is beneficial to preserve an open connection to the database. You can do that if you first 
 obtain the connection using sqlutilpy.getConnection() and then provide it directly
 to sqlutil.get() and friends using conn=conn argument
+```python
+conn = sqlutilpy.getConnection(db='mydb', user='meuser', password='something', host='hostname')
+R= sqlutilpy.get('select 1', conn=conn)
+R1= sqlutilpy.get('select 1', conn=conn)
+```
+
+# How to cite the software
 
+If you use this package, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957
 
-# How to cite it
-If you use this, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957

{sqlutilpy-0.20.0 → sqlutilpy-0.22.0}/README.md

@@ -6,7 +6,8 @@
 # sqlutilpy
 Python module to query SQL databases and return numpy arrays, upload
 tables and run join queries involving local arrays and the tables in the DB.
-The module only works with PostgreSQL and SQLite databases.
+This module is optimized to be able to deal efficiently with query results with millions of rows.
+The module only works with PostgreSQL, SQLite and DuckDB databases.
 
 The full documentation is available [here](http://sqlutilpy.readthedocs.io/en/latest/)
 
@@ -18,10 +19,11 @@ To install the package you just need to do pip install.
 ```
 pip install sqlutilpy
 ```
-## Authentification
-Throughout this readme, I'll assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
-has been created with the login/password details for Postgresql. If that is not the case, all of the 
-commands given above will also need user='....' and password='...' options
+## Authentication
+
+Throughout this readme, I will assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
+has been created with your login/password details for Postgresql. If that is not the case, all of the 
+commands given below will also need user='....' and password='...' options
 
 ## Connection information
 
@@ -32,6 +34,8 @@ of the connection.
 
 
 ## Querying the database and retrieving the results
+
+This command will run the query and put the columns into variables ra,dec
 ```python
 import sqlutilpy
 ra,dec = squtilpy.get('select ra,dec from mytable', 
@@ -39,10 +43,11 @@ ra,dec = squtilpy.get('select ra,dec from mytable',
                  db='THE_NAME_OF_MY_DB')
 ```
 
-By default sqlutilpy.get executes the result and returns the tuple of 
-results. But you can return the results as dictionary using asDict option.
+By default sqlutilpy.get executes the query and returns the tuple with 
+results. You can return the results as dictionary using asDict option.
 
 ## Uploading your arrays as column in a table
+
 ```python
 x = np.arange(10)                                                   
 y = x**.5                                                           
@@ -58,20 +63,28 @@ you could upload the arrays in the DB and run a query, but local_join function d
 
 ```python
 myid = np.arange(10)
-y = x**.5
+y = np.random.uniform(size=10)
+
 R=sqlutilpy.local_join('''select * from mytmptable as m, 
-           somebigtable as s where s.id=m.myid order by m.myid''',                                                                       'mytmptable',(x,y),('myid','ycol'))
+           somebigtable as s where s.id=m.myid order by m.myid''',                                              
+           'mytmptable',(myid, y),('myid','ycol'))
 ```
-It executes a query as if you arrays where in a mytmptable. ( behind the scenes
-it uploads the data to the db and runs a query)
 
+It executes a query as if you arrays were in mytmptable. (behind the scenes
+it uploads the data to the db and runs a query)
 
 ## Keeping the connection open. 
 
-Often it's benefitial to preserve an open connection. You can do that if you first 
+Often it is beneficial to preserve an open connection to the database. You can do that if you first 
 obtain the connection using sqlutilpy.getConnection() and then provide it directly
 to sqlutil.get() and friends using conn=conn argument
+```python
+conn = sqlutilpy.getConnection(db='mydb', user='meuser', password='something', host='hostname')
+R= sqlutilpy.get('select 1', conn=conn)
+R1= sqlutilpy.get('select 1', conn=conn)
+```
+
+# How to cite the software
 
+If you use this package, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957
 
-# How to cite it
-If you use this, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957

sqlutilpy-0.22.0/pyproject.toml

@@ -0,0 +1,32 @@
+[project]
+name = "sqlutilpy"
+dynamic = ["version"] 
+description = "Database query code returning numpy arrays"
+authors = [
+    {name = "Sergey E. Koposov", email = "skoposov@ed.ac.uk"},
+]
+dependencies = [
+    "numpy",
+    "psycopg2-binary",
+    "numpydoc",
+]
+requires-python = ">=3.6"
+readme = "README.md"
+license = {text = "BSD"}
+keywords = ["numpy postgresql query sql sqlite array"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "License :: OSI Approved :: BSD License",
+    "Topic :: Utilities",
+]
+
+[build-system]
+requires = ["setuptools", "wheel", "setuptools-git-versioning"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools-git-versioning]
+enabled = true
+
+[project.urls]
+Homepage = "https://github.com/segasai/sqlutilpy"
+

sqlutilpy-0.22.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

{sqlutilpy-0.20.0/py → sqlutilpy-0.22.0/src}/sqlutilpy/sqlutil.py

@@ -141,6 +141,12 @@ def getCursor(conn, driver=None, preamb=None, notNamed=False):
         cur = conn.cursor()
         if preamb is not None:
             cur.execute(preamb)
+    elif driver == 'duckdb':
+        cur = conn.cursor()
+        if preamb is not None:
+            cur.execute(preamb)
+    else:
+        raise SqlUtilException('unrecognized driver')
     return cur
 
 
@@ -224,7 +230,7 @@ def __getDType(row, typeCodes, strLength):
         1022: 'f8',
         1700: 'f8',  # numeric
         1114: '<M8[us]',  # timestamp
-        1082: '<M8[us]',  # date 
+        1082: '<M8[us]',  # date
         25: '|U%d',
         1042: '|U%d',  # character()
         1043: '|U%d'  # varchar
@@ -428,6 +434,13 @@ def get(query,
                 res = numpy.core.records.array(tups)
             else:
                 return [[]] * len(cur.description)
+        elif driver == 'duckdb':
+            tups = cur.fetchall()
+            colNames = [_tmp[0] for _tmp in cur.description]
+            if len(tups) > 0:
+                res = numpy.core.records.array(tups)
+            else:
+                return [[]] * len(cur.description)
 
         res = [res[tmp] for tmp in res.dtype.names]
 
@@ -527,10 +540,10 @@ def execute(query,
 def __create_schema(tableName, arrays, names, temp=False):
     hash = dict([(np.int32, 'integer'), (np.int64, 'bigint'),
                  (np.uint64, 'bigint'), (np.int16, 'smallint'),
-                 (np.uint8, 'bigint'), (np.float32, 'real'),
-                 (np.float64, 'double precision'), (np.string_, 'varchar'),
-                 (np.str_, 'varchar'), (np.bool_, 'boolean'),
-                 (np.datetime64, 'timestamp')])
+                 (np.uint8, 'smallint'), (np.int8, 'smallint'),
+                 (np.float32, 'real'), (np.float64, 'double precision'),
+                 (np.string_, 'varchar'), (np.str_, 'varchar'),
+                 (np.bool_, 'boolean'), (np.datetime64, 'timestamp')])
     if temp:
         temp = 'temporary'
     else:
@@ -538,7 +551,13 @@ def __create_schema(tableName, arrays, names, temp=False):
     outp = 'create %s table %s ' % (temp, tableName)
     outp1 = []
     for arr, name in zip(arrays, names):
-        outp1.append('"' + name + '" ' + hash[arr.dtype.type])
+        curtyp = arr.dtype.type
+        if curtyp == np.object_:
+            # array
+            curotyp = hash[arr[0].dtype.type] + '[]'
+        else:
+            curotyp = hash[arr.dtype.type]
+        outp1.append(f'"{name}" {curotyp}')
     return outp + '(' + ','.join(outp1) + ')'
 
 
@@ -547,13 +566,48 @@ def __print_arrays(arrays, f, delimiter=' '):
     print the input arrays into the open file separated by a delimiter
     """
     format_dict = dict([(np.int32, '%d'), (np.int64, '%d'), (np.int16, '%d'),
-                        (np.uint8, '%d'), (np.float32, '%.18e'),
-                        (np.float64, '%.18e'), (np.string_, '%s'),
-                        (np.str_, '%s'), (np.datetime64, '%s'),
-                        (np.bool_, '%d')])
-    fmt = [format_dict[x.dtype.type] for x in arrays]
+                        (np.int8, '%d'), (np.uint8, '%d'),
+                        (np.float32, '%.18e'), (np.float64, '%.18e'),
+                        (np.string_, '%s'), (np.str_, '%s'),
+                        (np.datetime64, '%s'), (np.bool_, '%d')])
+    fmts = []
+    array_mode = False
+    names = []
+    for i, x in enumerate(arrays):
+        names.append('f%d' % i)
+        if x.dtype.type in format_dict:
+            fmts.append(format_dict[x.dtype.type])
+        else:
+            if x.dtype.type == np.object_:
+                array_mode = True
+                fmts.append(None)
+            else:
+                raise RuntimeError(
+                    'Unsupported column type %s. Please report' %
+                    (x.dtype.type))
     recarr = np.rec.fromarrays(arrays)
-    np.savetxt(f, recarr, fmt=fmt, delimiter=delimiter)
+    if not array_mode:
+        np.savetxt(f, recarr, fmt=fmts, delimiter=delimiter)
+    else:
+        # this is really slow
+        for row in recarr:
+            for i, field in enumerate(names):
+                if i != 0:
+                    f.write(delimiter.encode())
+                if fmts[i] is None:
+                    curstr = np.array2string(
+                        row[field],
+                        max_line_width=np.inf,
+                        threshold=None,
+                        separator=',',
+                        formatter={'all': lambda x: str(x)})
+                    # formatter is needed because otherwise there is
+                    # whitespace padding
+                    curstr = '{' + curstr[1:-1] + '}'
+                    f.write(curstr.encode())
+                else:
+                    f.write(str(row[field]).encode())
+            f.write(b'\n')
 
 
 def failure_cleanup(conn, connSupplied):
@@ -659,6 +713,9 @@ def upload(tableName,
 table/pandas/dictionary or provide a separate list of arrays and their names')
 
     arrays = [np.asarray(_) for _ in arrays]
+    if len(arrays) != len(names):
+        raise RuntimeError('The column names list must have the same '
+                           'length as array list')
     repl_char = {
         ' ': '_',
         '-': '_',

sqlutilpy-0.22.0/src/sqlutilpy/version.py

@@ -0,0 +1 @@
+__version__ = '0.22.0'

{sqlutilpy-0.20.0 → sqlutilpy-0.22.0/src/sqlutilpy.egg-info}/PKG-INFO

@@ -1,17 +1,20 @@
 Metadata-Version: 2.1
 Name: sqlutilpy
-Version: 0.20.0
+Version: 0.22.0
 Summary: Database query code returning numpy arrays
-Home-page: https://github.com/segasai/sqlutilpy
-Author: Sergey Koposov
-Author-email: skoposov@ed.ac.uk
+Author-email: "Sergey E. Koposov" <skoposov@ed.ac.uk>
 License: BSD
+Project-URL: Homepage, https://github.com/segasai/sqlutilpy
 Keywords: numpy postgresql query sql sqlite array
 Classifier: Development Status :: 3 - Alpha
-Classifier: Topic :: Utilities
 Classifier: License :: OSI Approved :: BSD License
+Classifier: Topic :: Utilities
+Requires-Python: >=3.6
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
+Requires-Dist: numpy
+Requires-Dist: psycopg2-binary
+Requires-Dist: numpydoc
 
 [![Build Status](https://github.com/segasai/sqlutilpy/workflows/Testing/badge.svg)](https://github.com/segasai/sqlutilpy/actions)
 [![Documentation Status](https://readthedocs.org/projects/sqlutilpy/badge/?version=latest)](http://sqlutilpy.readthedocs.io/en/latest/?badge=latest)
@@ -21,7 +24,8 @@ License-File: LICENSE.txt
 # sqlutilpy
 Python module to query SQL databases and return numpy arrays, upload
 tables and run join queries involving local arrays and the tables in the DB.
-The module only works with PostgreSQL and SQLite databases.
+This module is optimized to be able to deal efficiently with query results with millions of rows.
+The module only works with PostgreSQL, SQLite and DuckDB databases.
 
 The full documentation is available [here](http://sqlutilpy.readthedocs.io/en/latest/)
 
@@ -33,10 +37,11 @@ To install the package you just need to do pip install.
 ```
 pip install sqlutilpy
 ```
-## Authentification
-Throughout this readme, I'll assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
-has been created with the login/password details for Postgresql. If that is not the case, all of the 
-commands given above will also need user='....' and password='...' options
+## Authentication
+
+Throughout this readme, I will assume that the .pgpass file ( https://www.postgresql.org/docs/11/libpq-pgpass.html ) 
+has been created with your login/password details for Postgresql. If that is not the case, all of the 
+commands given below will also need user='....' and password='...' options
 
 ## Connection information
 
@@ -47,6 +52,8 @@ of the connection.
 
 
 ## Querying the database and retrieving the results
+
+This command will run the query and put the columns into variables ra,dec
 ```python
 import sqlutilpy
 ra,dec = squtilpy.get('select ra,dec from mytable', 
@@ -54,10 +61,11 @@ ra,dec = squtilpy.get('select ra,dec from mytable',
                  db='THE_NAME_OF_MY_DB')
 ```
 
-By default sqlutilpy.get executes the result and returns the tuple of 
-results. But you can return the results as dictionary using asDict option.
+By default sqlutilpy.get executes the query and returns the tuple with 
+results. You can return the results as dictionary using asDict option.
 
 ## Uploading your arrays as column in a table
+
 ```python
 x = np.arange(10)                                                   
 y = x**.5                                                           
@@ -73,20 +81,28 @@ you could upload the arrays in the DB and run a query, but local_join function d
 
 ```python
 myid = np.arange(10)
-y = x**.5
+y = np.random.uniform(size=10)
+
 R=sqlutilpy.local_join('''select * from mytmptable as m, 
-           somebigtable as s where s.id=m.myid order by m.myid''',                                                                       'mytmptable',(x,y),('myid','ycol'))
+           somebigtable as s where s.id=m.myid order by m.myid''',                                              
+           'mytmptable',(myid, y),('myid','ycol'))
 ```
-It executes a query as if you arrays where in a mytmptable. ( behind the scenes
-it uploads the data to the db and runs a query)
 
+It executes a query as if you arrays were in mytmptable. (behind the scenes
+it uploads the data to the db and runs a query)
 
 ## Keeping the connection open. 
 
-Often it's benefitial to preserve an open connection. You can do that if you first 
+Often it is beneficial to preserve an open connection to the database. You can do that if you first 
 obtain the connection using sqlutilpy.getConnection() and then provide it directly
 to sqlutil.get() and friends using conn=conn argument
+```python
+conn = sqlutilpy.getConnection(db='mydb', user='meuser', password='something', host='hostname')
+R= sqlutilpy.get('select 1', conn=conn)
+R1= sqlutilpy.get('select 1', conn=conn)
+```
+
+# How to cite the software
 
+If you use this package, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957
 
-# How to cite it
-If you use this, please cite it through zenodo https://doi.org/10.5281/zenodo.6867957

sqlutilpy-0.22.0/src/sqlutilpy.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+LICENSE.txt
+MANIFEST.in
+README.md
+pyproject.toml
+src/sqlutilpy/__init__.py
+src/sqlutilpy/sqlutil.py
+src/sqlutilpy/version.py
+src/sqlutilpy.egg-info/PKG-INFO
+src/sqlutilpy.egg-info/SOURCES.txt
+src/sqlutilpy.egg-info/dependency_links.txt
+src/sqlutilpy.egg-info/requires.txt
+src/sqlutilpy.egg-info/top_level.txt
+tests/killer.py
+tests/test_pg.py
+tests/test_sqlite.py