bauta 0.1.0__tar.gz

bauta-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 David Ribeiro
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

bauta-0.1.0/PKG-INFO

@@ -0,0 +1,211 @@
+Metadata-Version: 2.4
+Name: bauta
+Version: 0.1.0
+Summary: Safe, realistic copies of production across databases: masking, subsets, synthetic data, and the ETL to move them
+Author: David Ribeiro
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/ribeiro11075/bauta
+Project-URL: Documentation, https://github.com/ribeiro11075/bauta/tree/master/docs
+Project-URL: Issues, https://github.com/ribeiro11075/bauta/issues
+Keywords: data masking,test data,anonymization,pseudonymization,database subsetting,synthetic data,etl,format-preserving encryption,oracle,sql server,postgresql,mysql
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Database
+Classifier: Topic :: Security
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml<7,>=6.0.1
+Requires-Dist: pydantic<3,>=2.6
+Provides-Extra: mysql
+Requires-Dist: mysql-connector-python>=8.4; extra == "mysql"
+Provides-Extra: postgresql
+Requires-Dist: psycopg2>=2.9.6; extra == "postgresql"
+Provides-Extra: oracle
+Requires-Dist: oracledb>=2.5; extra == "oracle"
+Provides-Extra: mssql
+Requires-Dist: pymssql>=2.3.5; extra == "mssql"
+Provides-Extra: mariadb
+Requires-Dist: mysql-connector-python>=8.4; extra == "mariadb"
+Provides-Extra: sqlite
+Provides-Extra: fpe
+Requires-Dist: cryptography>=42; extra == "fpe"
+Provides-Extra: all
+Requires-Dist: bauta[fpe,mariadb,mssql,mysql,oracle,postgresql,sqlite]; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest==9.1.1; extra == "dev"
+Requires-Dist: mypy==2.3.1; extra == "dev"
+Requires-Dist: types-PyYAML==6.0.12.20260815; extra == "dev"
+Requires-Dist: build==1.6.1; extra == "dev"
+Requires-Dist: twine==7.0.0; extra == "dev"
+Dynamic: license-file
+
+# Bauta
+
+**Put a mask on production:** a safe, realistic stand-in for your production data, in whichever database you need it. Bauta masks what it copies, copies only the slice you need with every relationship intact, generates what may not be copied at all, and moves data between databases on a schedule you already run, with nothing to host.
+
+- **Masking:** consistent across tables and runs, one-to-one for keys (NIST FF1 where policy requires it), applied before anything reaches the target, and every column must be covered.
+- **Discovery, subsets and synthetic data:** propose a masking policy from a live schema, copy a referentially complete slice of production, create the copy's tables in whichever database it goes to, and fill tables that can't be copied with generated rows.
+- **Audit:** report what every job does with data and what a reviewer should question, and seal each run's masking manifest so it can be verified later.
+- **Six databases:** Oracle, SQL Server, PostgreSQL, MySQL, MariaDB and SQLite, as source or target in any combination.
+- **Streaming:** memory stays flat however large the table, and PostgreSQL and SQL Server targets load in bulk.
+- **Fast:** ten million rows of six masked columns in under two minutes on one core with the optional native masker, and under eight without. Either way the masks are the same.
+- **Incremental loads:** extract only what changed since the last successful run.
+- **A dependency graph:** jobs run in order, concurrently where they can, each in its own process with an optional timeout.
+- **Operable:** run history, Prometheus metrics, webhook alerts, run state in a file or a table, and passwords from a command for cloud IAM tokens.
+- **No infrastructure:** a `pip install`, some YAML, and a command you run from cron.
+
+
+## Install
+
+Python 3.10 or newer. Choose the drivers you need as extras; each is loaded only when a connection uses it.
+
+```
+pip install "bauta[postgresql,oracle]"
+```
+
+It isn't on PyPI yet. Until the first release, install from a clone:
+
+```
+git clone https://github.com/ribeiro11075/bauta.git
+cd bauta
+pip install -e ".[postgresql,oracle]"
+```
+
+| Extra | Driver | Needs besides pip |
+| --- | --- | --- |
+| `mysql`, `mariadb` | mysql-connector-python | nothing |
+| `postgresql` | psycopg2, built from source | a C compiler and PostgreSQL's client library (below) |
+| `oracle` | oracledb, in thin mode | nothing — no Oracle client |
+| `mssql` | pymssql | nothing |
+| `sqlite` | Python's own `sqlite3` | nothing |
+| `fpe` | cryptography, for the `fpe` masking strategy | nothing; `oracle` already brings it |
+| `all` | every driver above | as for `postgresql` |
+
+**Building psycopg2.** pip compiles it, so it needs a compiler and `pg_config`:
+
+- macOS: `xcode-select --install`, then `brew install libpq` and `export PATH="$(brew --prefix libpq)/bin:$PATH"` (Homebrew doesn't put libpq on the path by itself).
+- Debian or Ubuntu: `apt install build-essential libpq-dev`.
+
+To skip the build, leave `postgresql` (and `all`) out and install the prebuilt driver beside the other extras: `pip install "bauta[mysql,oracle,mssql]" psycopg2-binary`. psycopg2's maintainers recommend the source build for production.
+
+**The native masker (optional).** `bauta-rs`, in `mask-rs/`, masks in Rust: four to five times the throughput, identical masks, nothing to configure. With [Rust](https://rustup.rs) 1.83 or newer, pip builds it in the same command as the rest:
+
+```
+pip install -e ".[all]" ./mask-rs/py
+```
+
+Without Rust, leave `./mask-rs/py` off; everything works, only slower. See [the native masker](docs/masking.md#the-native-masker).
+
+
+## Quickstart
+
+```
+mkdir configuration
+cp example/starter/configuration/*.yaml configuration/
+```
+
+Edit `configuration/database.yaml` and `configuration/jobs.yaml` for your databases, then supply the credentials they reference:
+
+```
+export SOURCE_DB_PASSWORD=...  TARGET_DB_PASSWORD=...  MASKING_KEY=...
+
+bauta validate          # check the configuration, offline
+bauta run --dry-run     # check connections and tables, moving nothing
+bauta run               # run every job once
+```
+
+To see it work without any of that, using throwaway SQLite databases:
+
+```
+python example/walkthrough/demo.py       # the whole workflow: discover, subset, audit, mask, verify, synthesize
+python example/incremental/demo.py       # streaming and incremental loads
+python example/masking/demo.py           # masking, discovery and a subset, from Python
+python example/native-masking/demo.py    # the same job masked in Python and in Rust, compared
+```
+
+
+## The command
+
+```
+bauta run              run data jobs once, masking any with a `masking` section
+bauta validate         check configuration without connecting
+bauta jobs             show the job graph and what's due
+bauta history          show recent job outcomes recorded with --history
+
+bauta discover         propose a masking policy for tables
+bauta subset           generate jobs that copy a referentially complete subset
+bauta schema           create target tables from source ones, in the target's dialect
+bauta synthesize       fill tables with generated rows, for data that can't be copied
+bauta clear            empty the target tables of jobs, children first
+
+bauta audit            report what each job does with data, and what to question
+bauta verify-manifest  check a manifest is unaltered, and who signed it
+```
+
+| Exit code | Meaning |
+| --- | --- |
+| `0` | every job completed |
+| `1` | a job failed, or was skipped because a predecessor failed, or the command failed on a database error |
+| `2` | invalid configuration or usage |
+| `130` | interrupted by a signal: running jobs finished, the rest were skipped |
+
+`run` makes one pass and exits, so it fits under cron or a Kubernetes CronJob. A second `run` sharing the same run state refuses to start while the first is still going. The useful flags:
+
+| Flag | |
+| --- | --- |
+| `--config DIR` | where the YAML lives; default `./configuration` |
+| `--job NAME` | run only this job — without its predecessors, which it warns about |
+| `--force` | ignore `refresh` windows |
+| `--forever` | stay running; for freshness under a minute |
+| `--log-format json` | structured logs for a collector |
+| `--log FILE` | also log to a file, in addition to stderr (`--quiet` silences stderr) |
+| `--memory FILE` | where run state (last runs, watermarks) is kept, overriding `jobs.yaml`'s [`memory`](docs/configuration.md#file-level) |
+| `--memory-database ALIAS` | keep run state in a database table instead |
+| `--history FILE` | append each job's outcome to a JSON-lines history |
+| `--metrics FILE` | write Prometheus metrics for the textfile collector (`--metrics-push URL` for a Pushgateway) |
+| `--notify-url URL` | post to a webhook when a run doesn't succeed; default `$BAUTA_NOTIFY_URL` |
+| `--accept-key-change` | run upsert jobs whose masking key changed since their last run |
+| `--manifest FILE` | write a sealed JSON record of what was masked, and how; signed if `$BAUTA_MANIFEST_KEY` is set |
+
+
+## Documentation
+
+| | |
+| --- | --- |
+| [Configuration](docs/configuration.md) | every field, how credentials are read from the environment, and connection options such as TLS |
+| [Masking](docs/masking.md) | strategies, consistent masks across tables, the key, the manifest, `audit`, `discover`, `subset`, `schema`, `synthesize` and `clear` |
+| [How it works](docs/design.md) | streaming, incremental loads, retries, scheduling, and the masking design |
+| [Operating it](docs/operations.md) | run state, history, metrics and notifications |
+| [Security model](docs/security.md) | what masking protects and what it doesn't, the constructions, keys, and a deployment checklist |
+| [Library](docs/library.md) | embedding it in Python, results, memory backends |
+| [Development](docs/development.md) | running the tests, including against real databases |
+
+
+## Layout
+
+| | |
+| --- | --- |
+| `bauta/` | the package; `runner.py` runs jobs, `masking.py` masks, `databaseDialects.py` holds per-database SQL |
+| `mask-rs/` | the optional native masker, in Rust — see [its README](mask-rs/README.md) |
+| `example/` | runnable demos, each with its `configuration/`, and a starter configuration — see [its README](example/README.md) |
+| `docs/` | the documentation above |
+| `tests/` | the test suite |
+
+
+## License
+
+[MIT](LICENSE)

bauta-0.1.0/README.md

@@ -0,0 +1,156 @@
+# Bauta
+
+**Put a mask on production:** a safe, realistic stand-in for your production data, in whichever database you need it. Bauta masks what it copies, copies only the slice you need with every relationship intact, generates what may not be copied at all, and moves data between databases on a schedule you already run, with nothing to host.
+
+- **Masking:** consistent across tables and runs, one-to-one for keys (NIST FF1 where policy requires it), applied before anything reaches the target, and every column must be covered.
+- **Discovery, subsets and synthetic data:** propose a masking policy from a live schema, copy a referentially complete slice of production, create the copy's tables in whichever database it goes to, and fill tables that can't be copied with generated rows.
+- **Audit:** report what every job does with data and what a reviewer should question, and seal each run's masking manifest so it can be verified later.
+- **Six databases:** Oracle, SQL Server, PostgreSQL, MySQL, MariaDB and SQLite, as source or target in any combination.
+- **Streaming:** memory stays flat however large the table, and PostgreSQL and SQL Server targets load in bulk.
+- **Fast:** ten million rows of six masked columns in under two minutes on one core with the optional native masker, and under eight without. Either way the masks are the same.
+- **Incremental loads:** extract only what changed since the last successful run.
+- **A dependency graph:** jobs run in order, concurrently where they can, each in its own process with an optional timeout.
+- **Operable:** run history, Prometheus metrics, webhook alerts, run state in a file or a table, and passwords from a command for cloud IAM tokens.
+- **No infrastructure:** a `pip install`, some YAML, and a command you run from cron.
+
+
+## Install
+
+Python 3.10 or newer. Choose the drivers you need as extras; each is loaded only when a connection uses it.
+
+```
+pip install "bauta[postgresql,oracle]"
+```
+
+It isn't on PyPI yet. Until the first release, install from a clone:
+
+```
+git clone https://github.com/ribeiro11075/bauta.git
+cd bauta
+pip install -e ".[postgresql,oracle]"
+```
+
+| Extra | Driver | Needs besides pip |
+| --- | --- | --- |
+| `mysql`, `mariadb` | mysql-connector-python | nothing |
+| `postgresql` | psycopg2, built from source | a C compiler and PostgreSQL's client library (below) |
+| `oracle` | oracledb, in thin mode | nothing — no Oracle client |
+| `mssql` | pymssql | nothing |
+| `sqlite` | Python's own `sqlite3` | nothing |
+| `fpe` | cryptography, for the `fpe` masking strategy | nothing; `oracle` already brings it |
+| `all` | every driver above | as for `postgresql` |
+
+**Building psycopg2.** pip compiles it, so it needs a compiler and `pg_config`:
+
+- macOS: `xcode-select --install`, then `brew install libpq` and `export PATH="$(brew --prefix libpq)/bin:$PATH"` (Homebrew doesn't put libpq on the path by itself).
+- Debian or Ubuntu: `apt install build-essential libpq-dev`.
+
+To skip the build, leave `postgresql` (and `all`) out and install the prebuilt driver beside the other extras: `pip install "bauta[mysql,oracle,mssql]" psycopg2-binary`. psycopg2's maintainers recommend the source build for production.
+
+**The native masker (optional).** `bauta-rs`, in `mask-rs/`, masks in Rust: four to five times the throughput, identical masks, nothing to configure. With [Rust](https://rustup.rs) 1.83 or newer, pip builds it in the same command as the rest:
+
+```
+pip install -e ".[all]" ./mask-rs/py
+```
+
+Without Rust, leave `./mask-rs/py` off; everything works, only slower. See [the native masker](docs/masking.md#the-native-masker).
+
+
+## Quickstart
+
+```
+mkdir configuration
+cp example/starter/configuration/*.yaml configuration/
+```
+
+Edit `configuration/database.yaml` and `configuration/jobs.yaml` for your databases, then supply the credentials they reference:
+
+```
+export SOURCE_DB_PASSWORD=...  TARGET_DB_PASSWORD=...  MASKING_KEY=...
+
+bauta validate          # check the configuration, offline
+bauta run --dry-run     # check connections and tables, moving nothing
+bauta run               # run every job once
+```
+
+To see it work without any of that, using throwaway SQLite databases:
+
+```
+python example/walkthrough/demo.py       # the whole workflow: discover, subset, audit, mask, verify, synthesize
+python example/incremental/demo.py       # streaming and incremental loads
+python example/masking/demo.py           # masking, discovery and a subset, from Python
+python example/native-masking/demo.py    # the same job masked in Python and in Rust, compared
+```
+
+
+## The command
+
+```
+bauta run              run data jobs once, masking any with a `masking` section
+bauta validate         check configuration without connecting
+bauta jobs             show the job graph and what's due
+bauta history          show recent job outcomes recorded with --history
+
+bauta discover         propose a masking policy for tables
+bauta subset           generate jobs that copy a referentially complete subset
+bauta schema           create target tables from source ones, in the target's dialect
+bauta synthesize       fill tables with generated rows, for data that can't be copied
+bauta clear            empty the target tables of jobs, children first
+
+bauta audit            report what each job does with data, and what to question
+bauta verify-manifest  check a manifest is unaltered, and who signed it
+```
+
+| Exit code | Meaning |
+| --- | --- |
+| `0` | every job completed |
+| `1` | a job failed, or was skipped because a predecessor failed, or the command failed on a database error |
+| `2` | invalid configuration or usage |
+| `130` | interrupted by a signal: running jobs finished, the rest were skipped |
+
+`run` makes one pass and exits, so it fits under cron or a Kubernetes CronJob. A second `run` sharing the same run state refuses to start while the first is still going. The useful flags:
+
+| Flag | |
+| --- | --- |
+| `--config DIR` | where the YAML lives; default `./configuration` |
+| `--job NAME` | run only this job — without its predecessors, which it warns about |
+| `--force` | ignore `refresh` windows |
+| `--forever` | stay running; for freshness under a minute |
+| `--log-format json` | structured logs for a collector |
+| `--log FILE` | also log to a file, in addition to stderr (`--quiet` silences stderr) |
+| `--memory FILE` | where run state (last runs, watermarks) is kept, overriding `jobs.yaml`'s [`memory`](docs/configuration.md#file-level) |
+| `--memory-database ALIAS` | keep run state in a database table instead |
+| `--history FILE` | append each job's outcome to a JSON-lines history |
+| `--metrics FILE` | write Prometheus metrics for the textfile collector (`--metrics-push URL` for a Pushgateway) |
+| `--notify-url URL` | post to a webhook when a run doesn't succeed; default `$BAUTA_NOTIFY_URL` |
+| `--accept-key-change` | run upsert jobs whose masking key changed since their last run |
+| `--manifest FILE` | write a sealed JSON record of what was masked, and how; signed if `$BAUTA_MANIFEST_KEY` is set |
+
+
+## Documentation
+
+| | |
+| --- | --- |
+| [Configuration](docs/configuration.md) | every field, how credentials are read from the environment, and connection options such as TLS |
+| [Masking](docs/masking.md) | strategies, consistent masks across tables, the key, the manifest, `audit`, `discover`, `subset`, `schema`, `synthesize` and `clear` |
+| [How it works](docs/design.md) | streaming, incremental loads, retries, scheduling, and the masking design |
+| [Operating it](docs/operations.md) | run state, history, metrics and notifications |
+| [Security model](docs/security.md) | what masking protects and what it doesn't, the constructions, keys, and a deployment checklist |
+| [Library](docs/library.md) | embedding it in Python, results, memory backends |
+| [Development](docs/development.md) | running the tests, including against real databases |
+
+
+## Layout
+
+| | |
+| --- | --- |
+| `bauta/` | the package; `runner.py` runs jobs, `masking.py` masks, `databaseDialects.py` holds per-database SQL |
+| `mask-rs/` | the optional native masker, in Rust — see [its README](mask-rs/README.md) |
+| `example/` | runnable demos, each with its `configuration/`, and a starter configuration — see [its README](example/README.md) |
+| `docs/` | the documentation above |
+| `tests/` | the test suite |
+
+
+## License
+
+[MIT](LICENSE)

bauta-0.1.0/bauta/__init__.py

@@ -0,0 +1,93 @@
+from .configuration import (
+    BaseJobConfig,
+    Configuration,
+    ConfigurationError,
+    DatabaseConnectionConfig,
+    DatabaseType,
+    DataJobConfig,
+    DataJobsFile,
+    InsertStrategy,
+    MaskingConfig,
+    expandEnvironmentVariables,
+    )
+from .databaseDialects import ColumnCategory, DatabaseDialect, ForeignKey, MariaDBDialect, MSSQLDialect, MySQLDialect, OracleDialect, PostgreSQLDialect, SQLiteDialect
+from .database import Database
+from .dependencyGraph import DependencyGraph, JobOutcome, JobStatus
+from .discovery import TableProposal, proposeTable
+from .log import Log
+from .audit import auditJobs, renderAudit
+from .masking import LOCALES, STRATEGIES, MaskingError, MaskingPlan, Strategy, buildMaskingManifest, keyFingerprint, resolveStrategy, sealManifest, \
+    verifyManifest
+from .memory import DATABASE_MEMORY_SCHEMA, DatabaseMemory, FileMemory, MemoryBackend, RunInProgressError, exclusiveRun
+from .reporting import DATABASE_HISTORY_SCHEMA, DatabaseHistory, FileHistory, RunHistory, notify, pushMetrics, writeMetricsFile
+from .runner import RunResult, runDataJobs
+from .subset import SubsetError, SubsetPlan, planSubset
+from .synthesize import SynthesisError, planTable, synthesizeTable
+from .transform import Transform, Transformer, TransformError, TransformResolutionError, resolveTransformer
+
+__all__ = [
+    'auditJobs',
+    'BaseJobConfig',
+    'buildMaskingManifest',
+    'ColumnCategory',
+    'Configuration',
+    'ConfigurationError',
+    'DATABASE_HISTORY_SCHEMA',
+    'DATABASE_MEMORY_SCHEMA',
+    'DatabaseHistory',
+    'Database',
+    'DatabaseConnectionConfig',
+    'DatabaseDialect',
+    'DatabaseMemory',
+    'DatabaseType',
+    'exclusiveRun',
+    'expandEnvironmentVariables',
+    'DataJobConfig',
+    'DataJobsFile',
+    'DependencyGraph',
+    'FileHistory',
+    'FileMemory',
+    'ForeignKey',
+    'InsertStrategy',
+    'JobOutcome',
+    'JobStatus',
+    'keyFingerprint',
+    'LOCALES',
+    'Log',
+    'MariaDBDialect',
+    'MaskingConfig',
+    'MaskingError',
+    'MaskingPlan',
+    'MemoryBackend',
+    'MSSQLDialect',
+    'notify',
+    'MySQLDialect',
+    'OracleDialect',
+    'planSubset',
+    'PostgreSQLDialect',
+    'proposeTable',
+    'pushMetrics',
+    'RunHistory',
+    'RunInProgressError',
+    'RunResult',
+    'SQLiteDialect',
+    'STRATEGIES',
+    'Strategy',
+    'SubsetError',
+    'SubsetPlan',
+    'SynthesisError',
+    'synthesizeTable',
+    'planTable',
+    'TableProposal',
+    'Transform',
+    'Transformer',
+    'TransformError',
+    'TransformResolutionError',
+    'renderAudit',
+    'resolveStrategy',
+    'resolveTransformer',
+    'sealManifest',
+    'verifyManifest',
+    'writeMetricsFile',
+    'runDataJobs',
+    ]