instackup 0.2.6__tar.gz → 0.4.0__tar.gz

instackup-0.4.0/.gitignore

@@ -0,0 +1,310 @@
+# Created by https://www.gitignore.io/api/linux,macos,python,pycharm,windows,sublimetext
+# Edit at https://www.gitignore.io/?templates=linux,macos,python,pycharm,windows,sublimetext
+
+### Linux ###
+*~
+
+# temporary files which can be created if a process still has a handle open of a deleted file
+.fuse_hidden*
+
+# KDE directory preferences
+.directory
+
+# Linux trash folder which might appear on any partition or disk
+.Trash-*
+
+# .nfs files are created when an open file is removed but is still being accessed
+.nfs*
+
+### macOS ###
+# General
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# Icon must end with two \r
+Icon
+
+# Thumbnails
+._*
+
+# Files that might appear in the root of a volume
+.DocumentRevisions-V100
+.fseventsd
+.Spotlight-V100
+.TemporaryItems
+.Trashes
+.VolumeIcon.icns
+.com.apple.timemachine.donotpresent
+
+# Directories potentially created on remote AFP share
+.AppleDB
+.AppleDesktop
+Network Trash Folder
+Temporary Items
+.apdisk
+
+### PyCharm ###
+# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm
+# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
+
+# User-specific stuff
+.idea/**/workspace.xml
+.idea/**/tasks.xml
+.idea/**/usage.statistics.xml
+.idea/**/dictionaries
+.idea/**/shelf
+
+# Generated files
+.idea/**/contentModel.xml
+
+# Sensitive or high-churn files
+.idea/**/dataSources/
+.idea/**/dataSources.ids
+.idea/**/dataSources.local.xml
+.idea/**/sqlDataSources.xml
+.idea/**/dynamic.xml
+.idea/**/uiDesigner.xml
+.idea/**/dbnavigator.xml
+
+# Gradle
+.idea/**/gradle.xml
+.idea/**/libraries
+
+# Gradle and Maven with auto-import
+# When using Gradle or Maven with auto-import, you should exclude module files,
+# since they will be recreated, and may cause churn.  Uncomment if using
+# auto-import.
+# .idea/modules.xml
+# .idea/*.iml
+# .idea/modules
+# *.iml
+# *.ipr
+
+# CMake
+cmake-build-*/
+
+# Mongo Explorer plugin
+.idea/**/mongoSettings.xml
+
+# File-based project format
+*.iws
+
+# IntelliJ
+out/
+
+# mpeltonen/sbt-idea plugin
+.idea_modules/
+
+# JIRA plugin
+atlassian-ide-plugin.xml
+
+# Cursive Clojure plugin
+.idea/replstate.xml
+
+# Crashlytics plugin (for Android Studio and IntelliJ)
+com_crashlytics_export_strings.xml
+crashlytics.properties
+crashlytics-build.properties
+fabric.properties
+
+# Editor-based Rest Client
+.idea/httpRequests
+
+# Android studio 3.1+ serialized cache file
+.idea/caches/build_file_checksums.ser
+
+### PyCharm Patch ###
+# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721
+
+# *.iml
+# modules.xml
+# .idea/misc.xml
+# *.ipr
+
+# Sonarlint plugin
+.idea/**/sonarlint/
+
+# SonarQube Plugin
+.idea/**/sonarIssues.xml
+
+# Markdown Navigator plugin
+.idea/**/markdown-navigator.xml
+.idea/**/markdown-navigator/
+
+### Python ###
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# pyenv
+.python-version
+env/
+venv/
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# Mr Developer
+.mr.developer.cfg
+.project
+.pydevproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# Log files
+logs/
+*.log
+
+### SublimeText ###
+# Cache files for Sublime Text
+*.tmlanguage.cache
+*.tmPreferences.cache
+*.stTheme.cache
+
+# Workspace files are user-specific
+*.sublime-workspace
+
+# Project files should be checked into the repository, unless a significant
+# proportion of contributors will probably not be using Sublime Text
+# *.sublime-project
+
+# SFTP configuration file
+sftp-config.json
+
+# Package control specific files
+Package Control.last-run
+Package Control.ca-list
+Package Control.ca-bundle
+Package Control.system-ca-bundle
+Package Control.cache/
+Package Control.ca-certs/
+Package Control.merged-ca-bundle
+Package Control.user-ca-bundle
+oscrypto-ca-bundle.crt
+bh_unicode_properties.cache
+
+# Sublime-github package stores a github token in this file
+# https://packagecontrol.io/packages/sublime-github
+GitHub.sublime-settings
+
+### Windows ###
+# Windows thumbnail cache files
+Thumbs.db
+Thumbs.db:encryptable
+ehthumbs.db
+ehthumbs_vista.db
+
+# Dump file
+*.stackdump
+
+# Folder config file
+[Dd]esktop.ini
+
+# Recycle Bin used on file shares
+$RECYCLE.BIN/
+
+# Windows Installer files
+*.cab
+*.msi
+*.msix
+*.msm
+*.msp
+
+# Windows shortcuts
+*.lnk
+
+# End of https://www.gitignore.io/api/linux,macos,python,pycharm,windows,sublimetext
+
+### instackup ###
+# Generated at build time by hatch-vcs
+src/instackup/_version.py
+.venv/
+.ruff_cache/

instackup-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Daniel Lavedonio de Lima
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

instackup-0.4.0/PKG-INFO

@@ -0,0 +1,211 @@
+Metadata-Version: 2.5
+Name: instackup
+Version: 0.4.0
+Summary: A package to ease interaction with cloud services, DB connections and commonly used functionalities in data analytics.
+Project-URL: Homepage, https://github.com/Lavedonio/instackup
+Project-URL: Documentation, https://github.com/Lavedonio/instackup/tree/main/docs
+Project-URL: Changelog, https://github.com/Lavedonio/instackup/tree/main/version_logs
+Project-URL: Issues, https://github.com/Lavedonio/instackup/issues
+Author-email: Daniel Lavedonio de Lima <daniel.lavedonio@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: bigquery,data-engineering,gcs,google-sheets,mysql,postgresql,redshift,s3
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: pandas>=2.2
+Requires-Dist: pyyaml>=6.0.1
+Provides-Extra: all
+Requires-Dist: boto3>=1.34; extra == 'all'
+Requires-Dist: google-auth>=2.29; extra == 'all'
+Requires-Dist: google-cloud-bigquery-datatransfer>=3.15; extra == 'all'
+Requires-Dist: google-cloud-bigquery[pandas]>=3.25; extra == 'all'
+Requires-Dist: google-cloud-secret-manager>=2.20; extra == 'all'
+Requires-Dist: google-cloud-storage>=2.18; extra == 'all'
+Requires-Dist: gspread>=6.1; extra == 'all'
+Requires-Dist: mysql-connector-python>=9.0; extra == 'all'
+Requires-Dist: pandas-gbq>=0.23; extra == 'all'
+Requires-Dist: psycopg2-binary>=2.9.9; extra == 'all'
+Provides-Extra: aws
+Requires-Dist: boto3>=1.34; extra == 'aws'
+Provides-Extra: gcp
+Requires-Dist: google-auth>=2.29; extra == 'gcp'
+Requires-Dist: google-cloud-bigquery-datatransfer>=3.15; extra == 'gcp'
+Requires-Dist: google-cloud-bigquery[pandas]>=3.25; extra == 'gcp'
+Requires-Dist: google-cloud-secret-manager>=2.20; extra == 'gcp'
+Requires-Dist: google-cloud-storage>=2.18; extra == 'gcp'
+Requires-Dist: pandas-gbq>=0.23; extra == 'gcp'
+Provides-Extra: gsheets
+Requires-Dist: google-auth>=2.29; extra == 'gsheets'
+Requires-Dist: gspread>=6.1; extra == 'gsheets'
+Provides-Extra: mysql
+Requires-Dist: mysql-connector-python>=9.0; extra == 'mysql'
+Provides-Extra: postgresql
+Requires-Dist: psycopg2-binary>=2.9.9; extra == 'postgresql'
+Provides-Extra: redshift
+Requires-Dist: boto3>=1.34; extra == 'redshift'
+Requires-Dist: psycopg2-binary>=2.9.9; extra == 'redshift'
+Description-Content-Type: text/markdown
+
+# Instackup
+
+[![PyPI](https://img.shields.io/pypi/v/instackup)](https://pypi.org/project/instackup/)
+[![Python versions](https://img.shields.io/pypi/pyversions/instackup)](https://pypi.org/project/instackup/)
+[![CI](https://github.com/Lavedonio/instackup/actions/workflows/ci.yml/badge.svg)](https://github.com/Lavedonio/instackup/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/Lavedonio/instackup/blob/main/LICENSE)
+
+This Python library is an open source way to standardize and simplify connections with cloud-based tools, databases and commonly used tools in data manipulation and analysis. It can help BI teams by having a unified source code for local development and testing as well as remote production (automated scheduled run) environments.
+
+# Index
+
+- [Current release](#current-release)
+- [Installation](#installation)
+- [Quick start](#quick-start)
+- [Credentials](#credentials)
+- [Documentation](#documentation)
+- [Development](#development)
+- [Version logs](#version-logs)
+
+# Current release
+**Version 0.4.0 (beta)**: a modernization release with support for Python 3.11–3.15, current versions of every dependency, production-ready credentials (environment variables, secret managers and cloud platform identities), type hints and a full test suite. See [what changed](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.4.0-beta-current_release.md#version-040-beta), including the breaking changes.
+
+# Installation
+Instackup needs [Python 3.11 or newer](https://www.python.org/downloads/). Install it with the extras for the services you use:
+
+    pip install 'instackup[gcp,postgresql]'
+
+| Extra | Modules |
+|---|---|
+| `gcp` | `bigquery_tools`, `gcloudstorage_tools` (and GCP Secret Manager credentials) |
+| `gsheets` | `gsheets_tools` |
+| `aws` | `s3_tools` (and AWS Secrets Manager credentials) |
+| `redshift` | `redshift_tools` |
+| `postgresql` | `sql_tools.PostgreSQLTool` |
+| `mysql` | `sql_tools.MySQLTool` |
+| `all` | Everything above |
+
+`sql_tools.SQLiteTool`, `general_tools` and `heroku_tools` work without extras.
+
+# Quick start
+
+```python
+from instackup import BigQueryTool, PostgreSQLTool, S3Tool
+
+# Databases: credentials come from environment variables, a secret manager or a credentials file
+with PostgreSQLTool() as pg:
+    orders = pg.query("SELECT * FROM orders WHERE created_at >= %s", params=("2026-01-01",))
+
+# Cloud services: with nothing configured, the platform's own identity is used
+bq = BigQueryTool()
+bq.upload(orders, dataset="analytics", table="orders", if_exists="replace")
+
+s3 = S3Tool(uri="s3://my-bucket/exports/")
+s3.upload_file("report.csv")
+```
+
+The library logs through Python's standard `logging` module. To see what it's doing:
+
+```python
+import logging
+
+logging.basicConfig(level=logging.INFO)
+```
+
+# Credentials
+Tools look up credentials by service and connection name (`connection="default"` unless you pass another one), in this order:
+
+1. `INSTACKUP_<SERVICE>__<CONNECTION>__<KEY>` environment variables, e.g. `INSTACKUP_POSTGRESQL__DEFAULT__PASSWORD`;
+2. a credentials document stored in AWS Secrets Manager (`INSTACKUP_AWS_SECRET_ID`) or GCP Secret Manager (`INSTACKUP_GCP_SECRET_NAME`);
+3. a YAML or JSON file set in `INSTACKUP_CREDENTIALS_FILE` (or the legacy `CREDENTIALS_HOME`), whose values can reference environment variables as `${NAME}`.
+
+When Google or AWS credentials aren't configured, the tools use [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials) and the [AWS default credential chain](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html), the recommended setup in production (service accounts, Workload Identity, IAM roles).
+
+A minimal local setup:
+
+```yaml
+# credentials.yml (use credentials/secret_template.yml as a starting point)
+PostgreSQL:
+  default:
+    host: localhost
+    dbname: postgres
+    user: postgres
+    password: ${POSTGRES_PASSWORD}
+```
+
+```bash
+export INSTACKUP_CREDENTIALS_FILE="$HOME/.config/instackup/credentials.yml"
+export POSTGRES_PASSWORD=...
+gcloud auth application-default login  # For Google tools
+aws sso login                          # For AWS tools
+```
+
+Read the [credentials guide](https://github.com/Lavedonio/instackup/blob/main/docs/credentials.md) for every option, production recommendations and how to migrate from `CREDENTIALS_HOME`.
+
+# Documentation
+
+- [Credentials guide](https://github.com/Lavedonio/instackup/blob/main/docs/credentials.md)
+- [Architecture and design decisions](https://github.com/Lavedonio/instackup/blob/main/docs/architecture.md)
+- Modules:
+  - [bigquery_tools](https://github.com/Lavedonio/instackup/blob/main/docs/bigquery_tools.md#bigquery_tools): `BigQueryTool`, to query, load and manage BigQuery data, plus PostgreSQL to BigQuery schema conversion.
+  - [gcloudstorage_tools](https://github.com/Lavedonio/instackup/blob/main/docs/gcloudstorage_tools.md#gcloudstorage_tools): `GCloudStorageTool`, to manage Google Cloud Storage files and folders.
+  - [general_tools](https://github.com/Lavedonio/instackup/blob/main/docs/general_tools.md#general_tools): URI parsing, text normalization and the code location.
+  - [gsheets_tools](https://github.com/Lavedonio/instackup/blob/main/docs/gsheets_tools.md#gsheets_tools): `GSheetsTool`, to read and write Google Sheets as DataFrames.
+  - [heroku_tools](https://github.com/Lavedonio/instackup/blob/main/docs/heroku_tools.md#heroku_tools): `HerokuTool`, a Heroku CLI wrapper.
+  - [redshift_tools](https://github.com/Lavedonio/instackup/blob/main/docs/redshift_tools.md#redshift_tools): `RedShiftTool`, for Redshift queries and UNLOAD to S3.
+  - [s3_tools](https://github.com/Lavedonio/instackup/blob/main/docs/s3_tools.md#s3_tools): `S3Tool`, to manage S3 files and folders.
+  - [sql_tools](https://github.com/Lavedonio/instackup/blob/main/docs/sql_tools.md#sql_tools): `SQLiteTool`, `PostgreSQLTool` and `MySQLTool`.
+
+# Development
+The project uses [uv](https://docs.astral.sh/uv/).
+
+```bash
+uv sync --all-extras          # Create .venv with every extra and the dev tools
+uv run pytest                 # Unit tests
+uv run ruff check             # Lint
+uv run ruff format            # Format
+uv run mypy                   # Type check
+```
+
+Integration tests run against real databases and are skipped unless their URLs are set:
+
+```bash
+docker run -d --rm -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:17-alpine
+TEST_POSTGRESQL_URL=postgresql://postgres:postgres@localhost:5432/postgres uv run pytest -m integration
+```
+
+Every push and pull request runs the [CI workflow](https://github.com/Lavedonio/instackup/blob/main/.github/workflows/ci.yml). Publishing a GitHub release uploads the package to PyPI; see [releasing.md](https://github.com/Lavedonio/instackup/blob/main/docs/releasing.md).
+
+# Version logs
+See what changed in every version.
+
+- Beta releases
+  - [Version 0.4.0](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.4.0-beta-current_release.md#version-040-beta) (current release)
+  - [Version 0.3.0](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.3.0-beta.md#version-030-beta)
+  - [Version 0.2.6](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.2.6-beta.md#version-026-beta)
+  - [Version 0.2.5](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.2.5-beta.md#version-025-beta)
+  - [Version 0.2.4](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.2.4-beta.md#version-024-beta)
+  - [Version 0.2.3](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.2.3-beta.md#version-023-beta)
+  - [Version 0.2.2](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.2.2-beta.md#version-022-beta)
+  - [Version 0.2.1](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.2.1-beta.md#version-021-beta)
+  - [Version 0.2.0](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.2.0-beta.md#version-020-beta)
+  - [Version 0.1.2](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.1.2-beta.md#version-012-beta)
+  - [Version 0.1.1](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.1.1-beta.md#version-011-beta)
+  - [Version 0.1.0](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.1.0-beta.md#version-010-beta)
+- Alpha releases
+  - [Version 0.0.6](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.0.6-alpha.md#version-006-alpha)
+  - [Version 0.0.5](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.0.5-alpha.md#version-005-alpha)
+  - [Version 0.0.4](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.0.4-alpha.md#version-004-alpha)
+  - [Version 0.0.3](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.0.3-alpha.md#version-003-alpha)
+  - [Version 0.0.2](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.0.2-alpha.md#version-002-alpha)
+  - [Version 0.0.1](https://github.com/Lavedonio/instackup/blob/main/version_logs/v0.0.1-alpha.md#version-001-alpha)