dbextractors 1.0.0__tar.gz

dbextractors-1.0.0/ARCHITECTURE.md

@@ -0,0 +1,195 @@
+# ARCHITECTURE.md — dbextractors
+
+## Why it is shaped like this
+
+Measured by a machine comparison of the 15 existing extractor files (25,723 lines):
+
+| | identical lines |
+|---|---|
+| MySQL ↔ MSSQL | 87 % |
+| MySQL ↔ PostgreSQL | 78 % |
+| MySQL ↔ Firebird | 55 % |
+| common core of MySQL+MSSQL+PG | ~69 % of significant lines |
+
+And of 95 functions, **45 are identical everywhere** they appear. Splitting the code
+into *a core plus thin adapters* is therefore not a design idea but a description of
+what that code already looks like.
+
+**The target side is always PostgreSQL** — it is the single largest block of code and
+it is 100 % shared.
+
+## Layout
+
+```
+src/dbextractors/
+  entrypoint.py            run(config, dialect, logger) -> pd.DataFrame
+
+  core/
+    config.py              parsing and validation of TABLE / LOAD_SETTINGS / SOURCE_DB
+    retry.py               with_retry, wait_for_port
+    tunnel.py              SSH tunnel, keys, PDEATHSIG, connection_mode
+    coerce.py              vectorised type conversion and sanitisation
+    hashing.py             row_hash, choice of hashed columns
+    naming.py              column name normalisation (reserved-word underscore prefixes)
+    status.py              the returned DataFrame, logging, per-phase metrics
+    target_pg.py           ← the largest module, 100 % shared
+
+    strategies/
+      base.py              LoadStrategy ABC
+      full.py              shadow table + TRUNCATE and INSERT … SELECT in one transaction
+      incremental.py       window by updated_at / days_back
+      hash_diff.py         hash comparison without a CDC log
+      id_watermark.py      advance by an increasing PK
+      full_by_source.py    partitioning of the target by source (MSSQL only so far)
+      parent_incremental.py window taken from a parent table (Firebird only so far)
+
+  dialects/
+    base.py                SourceDialect ABC
+    mysql.py  mssql.py  postgres.py  firebird.py
+```
+
+The swap in `full.py` is deliberately **not** a `RENAME`. A view holds the OID of the
+table it was built on, not its name, so after renaming the view stays bound to the old
+relation and quietly serves data from the previous run. Loading into a shadow table and
+then doing `TRUNCATE` + `INSERT … SELECT` in a single transaction keeps every dependent
+view pointing at the same relation throughout, and an interrupted run leaves the target
+untouched instead of truncated.
+
+## Data flow
+
+```
+  config (from the calling block)
+        │
+        ▼
+  core.config ── validation, filling in defaults
+        │
+        ▼
+  core.tunnel ── direct | ssh | auto  →  SQLAlchemy engine
+        │
+        ▼
+  dialects.<X> ── column introspection, type map, pagination SQL
+        │
+        ▼
+  strategies.<Y> ── drives the loop: what to read, in what batches, what to write
+        │            (calls the dialect to read, target_pg to write)
+        ▼
+  core.coerce ── vectorised sanitisation of the batch
+        │
+        ▼
+  core.target_pg ── COPY into the shadow/target table, upsert, indexes,
+                    _deleted_in_source, _timestamp
+        │
+        ▼
+  core.status ── the returned DataFrame
+```
+
+The key division of responsibility:
+
+- **The dialect knows nothing about the target.** It can say what columns a table has,
+  how to read it in batches, and how its types map onto PG.
+- **The strategy knows nothing about the SQL dialect.** It asks the dialect abstractly.
+- **`target_pg` knows nothing about the source.** It is handed a DataFrame and column
+  metadata.
+
+Because of that, a new source is a new file in `dialects/` and nothing else.
+
+## `SourceDialect` — the adapter interface
+
+This is the only thing that has to be written for a new kind of database. A design, not
+dogma — adjust it to what the comparison of the existing variants shows, but keep the
+principle that the dialect does not know the target.
+
+```python
+class SourceDialect(ABC):
+    name: str                      # 'mysql' | 'mssql' | 'postgres' | 'firebird'
+    default_port: int
+    type_map: dict[str, str]       # source type -> PG type
+
+    def build_conn_str(self, params, host, port) -> str: ...
+    def probe(self, host, port, timeout) -> bool: ...
+
+    def introspect_columns(self, engine, database, schema, table) -> list[ColumnDef]: ...
+    def estimate_size(self, engine, table, where) -> tuple[float, int]: ...
+
+    def quote_ident(self, name: str) -> str: ...
+    def render_select(self, columns, table, where, order_by) -> str: ...
+
+    def source_ident(self, name: str) -> str: ...
+        # a name that did not come from introspection (configuration, a default in the
+        # code); Firebird upper-cases it, everywhere else this is quote_ident
+
+    def supports(self, feature: str) -> bool: ...
+        # 'keyset' | 'hash_diff' | 'partition_by_source' | 'parent_incremental'
+```
+
+**One deviation from this sketch:** `page_offset` and `page_keyset` were originally part
+of the interface, but they were **deleted as dead code** — every path reads through a
+single cursor via `pd.read_sql(chunksize=)`, so `LIMIT/OFFSET` is never built anywhere in
+the package. The cursor is also faster and has none of the quadratic overhead of a
+growing `OFFSET`. The `keyset` capability in `supports()` stays: `id_watermark` and `full`
+use it, and it is an independent property. See the CHANGELOG.
+
+`source_ident` was added instead: generated SQL mixes names that came from introspection
+(which have the shape the source uses) with names that came from the configuration or
+from a default in the code (which do not). The second group has to go through here, or
+Firebird will not find them.
+
+Notes from the existing code:
+
+- **MSSQL** has `TOP (n)` and `OFFSET … ROWS`, not `LIMIT`. It is the only one that
+  currently has multi-source and partitioning.
+- **Firebird** is the most distant (55 % identical). It has no hash mode at all today,
+  introspection goes through the `RDB$` tables, and incremental loading is resolved
+  through a parent table. **Migrate it last.**
+- **PostgreSQL as a source** additionally has `pagination_mode` and `id_watermark`.
+- **MySQL** is 73 % of the volume — migrate it after the two smaller dialects, not first.
+
+## Strategies
+
+| strategy | when | tables today |
+|---|---|---|
+| `full` | small tables, or no usable key | ~90 |
+| `hash_diff` | no CDC log, but a stable PK | ~530 |
+| `incremental` | there is a reliable modification-time column | ~27 |
+| `id_watermark` | append-only, increasing PK | 4 |
+| `full_by_source` | one target table fed from several sources | 16 |
+| `parent_incremental` | window taken from a parent table (Firebird) | 24 |
+
+`hash_diff` is both the dominant and the most expensive one — today it is
+**O(source rows, not changed rows)**, ~15,000 rows/s. Optimising this single strategy
+touches ~530 tables.
+
+## Distribution
+
+A standalone package, installed with pip and pinned to a version:
+
+```
+# requirements.txt in the consuming repository
+dbextractors==1.0.0
+```
+
+Extras pull in only the drivers a deployment actually needs, for example
+`pip install "dbextractors[target,mysql]"`.
+
+- The version is one line → upgrade and rollback happen per deployment, visibly in git.
+- The package is tested outside Mage, in ordinary pytest.
+- **The image build has to run in CI against a vendored wheel**, not `pip install` at
+  build time from a network source — otherwise an unreachable index takes the deploy
+  down.
+
+## Migration order
+
+By size of impact, smallest first:
+
+| # | group | tables | why here |
+|---|---|---|---|
+| 1 | variant B / PostgreSQL | 21 | the smallest real dialect; fixes a known hash bug on the way |
+| 2 | variant C / MSSQL | 16 | adds partitioning and multi-source |
+| 3 | variant A / PostgreSQL | 58 | proves it scales |
+| 4 | variant A / MySQL | 254 | |
+| 5 | variant C / MySQL | 237 | including folding the `_v2` fork back into options |
+| 6 | variant B / Firebird | 80 | the most distant dialect, last |
+| 7 | retiring the older generation | 3 | `*_loader`, `*_universal_extractor` |
+
+MySQL is 73 % of the volume but is deliberately only fourth. If the golden test turns out
+to have holes, better that it shows on 21 tables.

dbextractors-1.0.0/CHANGELOG.md

@@ -0,0 +1,377 @@
+# Changelog
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+the versioning follows [semver](https://semver.org/).
+
+Every release is tagged, and the tag is what a deployment pins in `requirements.txt`.
+**Every change carries a note on what it breaks** — roughly 670 tables depend on
+this package.
+
+## [1.0.0]
+
+### Added
+
+- **`LOAD_SETTINGS.convert_nchar_to_varchar` — MSSQL `NVARCHAR` text that arrives
+  whole.** MSSQL is read over a `cp1250` connection that pymssql and FreeTDS
+  disagree about: the bytes are produced as latin-1 and decoded as cp1250, so an
+  `NVARCHAR`/`NCHAR`/`NTEXT` value is cut at the first character latin-1 cannot
+  express (`'příliš žluťoučký kůň'` → `'p'`) and one that latin-1 *can* express
+  comes back as a different character (`ø` → `ř`). Documented under *Known
+  limitations* below; the charset cannot simply be changed, because it is exactly
+  what makes the legacy single-byte columns of a CP1250-collated source read
+  correctly, and seven candidate values were measured with none right for both.
+
+  With the key on, the N-typed columns — and only those, chosen by their
+  introspected type — are wrapped in `CONVERT(VARCHAR(MAX), …)` in the generated
+  `SELECT`. The server then converts them through the column's own collation and
+  they arrive whole over the same connection; what CP1250 cannot hold is degraded
+  rather than truncating the value (Cyrillic and emoji become `?`, `ø` is folded
+  onto `o`). Legacy `VARCHAR`/`CHAR`/`TEXT` are never wrapped. Verified against a
+  live MSSQL in `tests/dialects/test_source_db.py`, on the `dbo.unicode_edge`
+  fixture, with every assertion having a counterpart for the key off.
+
+  It lives in `LOAD_SETTINGS` rather than in `SOURCE_DB` beside `charset` because
+  it is a property of one table's load, not of the connection — `SOURCE_DB` is
+  shared by all 16 pipelines against this source, and `SourceDbConfig` never
+  reaches the SELECT in the first place.
+
+  **What breaks: nothing.** The default is `False` and the generated SQL is then
+  byte-identical to before, pinned by a test. Enabling it for a table is a
+  deliberate act with a cost: the contents of those columns change, while
+  `row_hash` does not move with them (the digests were always computed by the
+  server from the raw `NVARCHAR`), so under `load_method: hash` existing rows are
+  not seen as changed — that table needs one `forced_full_load` to repair its
+  history. Not for a table whose primary key is itself N-typed; see
+  [Backward compatibility](docs/legacy-compat.md#mssql-nvarchar-text-arrives-truncated).
+
+- **`LOAD_SETTINGS.incremental_parent_key_column` and
+  `incremental_parent_id_column` — the `parent_incremental` join is configurable.**
+  `ParentIncrementalStrategy` has always read both keys, but neither was part of
+  the contract: `entrypoint._settings_dict` builds `ctx.settings` with
+  `dataclasses.asdict`, so a key that is not a field of `LoadSettingsConfig` could
+  never arrive, the hard-coded `parent_id` / `id` always won, and `config.parse`
+  logged the key as unknown on top. The strategy's own error message said *"Set
+  incremental_parent_key_column."* — advice that could not be followed.
+
+  **What breaks: nothing.** The defaults are the two constants the strategy used
+  anyway, which are what the predecessor hard-codes, so a configuration that does
+  not name them loads exactly as it did. What changed is that naming them now
+  works — for a child table whose foreign key is not called `parent_id`, or a
+  parent not keyed by `id`.
+
+- **`LOAD_SETTINGS.strict_integer_precision` — large integers fail loudly instead
+  of being silently rounded.** An integer column that also holds a `NULL` is
+  promoted to `float64` by `pd.read_sql` before this package sees the frame, and
+  `float64` has a 53-bit mantissa: measured end to end on all four dialects, a
+  source value of `9007199254740993` lands in the target as `9007199254740992`,
+  and `1234567890123456789` as `1234567890123456768`. The row count is right and
+  the run is green. The trigger is the `NULL`, not the magnitude.
+
+  The rounding itself is **not** fixed and must not be — the predecessor loses the
+  same bits in the same place, so repairing the read would change what lands in
+  the target relative to it, and because the hash is rendered out of the frame it
+  would recompute `row_hash` for every table with a nullable numeric column and
+  force a full reload of all of them. Pinned in
+  `tests/coerce/test_int64_precision.py`.
+
+  With the new key on, a `float64` column bound for an integer column in the
+  target that holds `|value| > 2**53` raises `coerce.IntegerPrecisionError`,
+  naming the column, an offending value and the key that produced the failure.
+  The check is vectorised (one numpy comparison per column) and sits in
+  `target_pg.prepare_export_df`, at the last point where such a column is still a
+  float. The boundary is inclusive — exactly 2**53 passes.
+
+  **What breaks: nothing.** The default is `False` and every pipeline that does
+  not name the key writes exactly what it wrote before, rounding included. Enabled
+  by default it would fail an unknown number of tables, so switching it on is a
+  per-pipeline decision.
+
+- **A test stack for all four sources** (`docker/compose.yml`, `make db-up`).
+  PostgreSQL as the target plus MySQL, MSSQL and Firebird as sources, each
+  seeded with a table built around the types that behave neither like numbers
+  nor like text: MySQL's zero date and its `TIME` that legitimately exceeds 24
+  hours, MSSQL's three date/time families and `MONEY`, Firebird's negative
+  `NUMERIC` scale and space-padded `CHAR`.
+
+  Until now three of the four sources had **no live test at all** — only string
+  assertions over generated SQL, which cannot catch a type that maps correctly
+  and whose value still does not survive the trip. New markers `needs_mysql`,
+  `needs_mssql` and `needs_firebird`; without a DSN those tests skip, and CI
+  requires them to actually run.
+
+  Also `.env.example`, which the `.gitignore` had whitelisted for a long time
+  without the file ever existing.
+
+
+
+- **`SOURCE_DB.ssh_host_key_checking` — verification of the SSH host key.**
+  Until now the tunnel was opened with `StrictHostKeyChecking=no` and
+  `UserKnownHostsFile=/dev/null`, that is **without verifying the remote end**.
+  Inside a private network that is defensible, but it is the one place in the
+  package where the connection can be spoofed — and the only one where the
+  behaviour could not even be turned off::
+
+      SOURCE_DB:
+        ssh_host_key_checking: off | accept-new | strict
+
+  `off` is the inherited behaviour, `accept-new` remembers the key on the first
+  connection and refuses it once it changes, `strict` requires the host to be in
+  `known_hosts` beforehand. An invalid value fails in `config.validate`, not later
+  at the tunnel.
+
+  **What breaks:** nothing. The default is `off`, i.e. today's behaviour — the
+  configuration contract is frozen, and changing the default would take down runs
+  against hosts whose key was never recorded anywhere. For new deployments
+  `accept-new` is the recommended value.
+
+- **`src/dbextractors/core/secrets.py` — one credential redaction for the whole package.**
+  The password is interpolated into a string in seven places (four dialect
+  `build_conn_str` implementations and three paths to the target), and in the
+  implementations this package replaces only one of them was guarded — a single
+  helper on the path to the target, protecting a single message. Everywhere else
+  it was enough for the string to reach the text of an exception; typically
+  through `create_engine`, which prints an unusable URL in full, password
+  included, into the log.
+
+  `secrets.redact()` handles a SQLAlchemy URL and a libpq DSN alike, and through
+  `extra` also literal values that no pattern knows about. It **masks** the
+  password (`password=***`) rather than discarding the whole token, so a redacted
+  line still shows the shape of what failed.
+
+  `entrypoint` wraps `create_engine` failures through the redaction and re-raises
+  with `from None`. That last part matters more than it looks: chaining with
+  `from err` would leave the original exception — carrying the full URL, password
+  and all — attached as `__cause__`, and any handler that prints a whole traceback
+  would put it in the log anyway. `target_conn.describe_dsn` delegates to the same
+  redaction, so there is one implementation rather than two that can drift.
+
+- **Every remaining path a credential could take out of the process now goes
+  through that redaction.** The audit that closed the security pass triggered each
+  candidate on purpose — a wrong password, an unreachable host, an invalid DSN, a
+  session statement the server rejects, a tunnel that cannot bind — and read what
+  actually came out. Three findings were real leaks and are fixed:
+
+  - **A private key pasted into `SOURCE_DB.ssh_pkey` reached the log verbatim.**
+    That key is contracted to be a *path*, but when it holds the key itself
+    `os.stat` fails with `ENAMETOOLONG` and `core/tunnel.py` printed the whole
+    value — at `ERROR` level, not gated on `DEBUG`. `ssh` then echoes its own `-i`
+    argument back on stderr, which the package splices into the `RuntimeError`
+    raised when the tunnel does not come up, and under `DEBUG` the argv is logged
+    as well. `secrets.redact()` now masks any PEM private-key block, in every
+    spelling `ssh` accepts and including one whose `-----END-----` never arrived.
+    Of everything this package masks it is the only secret a changed database
+    password does not retire.
+
+  - **`psycopg2` quotes back the token it choked on in a malformed DSN.** A target
+    password containing a space makes the DSN invalid *because of* the password,
+    and the fragment quoted back is a piece of it: `invalid dsn: missing "=" after
+    "…"`. `secrets.dsn_secrets()` extracts the password as written, whole and in
+    pieces, for `redact(extra=…)`; `entrypoint._target_connection` and
+    `golden.session.connect` both wrap the connect and re-raise redacted, with
+    `from None`.
+
+  - **`run()`'s per-database handler emitted an unredacted traceback** through
+    `logging.exception`, and put `str(err)` into the Mage log, into the `error`
+    column of the returned DataFrame and into `SourceExtractionError` — four exits
+    for one string. The traceback is now rendered in the handler so that it *can*
+    be redacted, and the redaction happens once, with the configured password in
+    `extra`.
+
+  Redaction was also added where nothing leaks today but the text is quoted
+  wholesale and comes from a third party: `core/reading.py`, `core/retry.py`, the
+  MSSQL and Firebird retry logs, `entrypoint._attach_session_sql`,
+  `status.error_status`, `hash_diff`'s fallback reason, `golden/compare.py`'s
+  `report.error` (which is written to the `--json` file on disk) and the
+  malformed-format-string branch of the logger adapter, the one place in the
+  package that `repr()`s log arguments it never inspected.
+
+  All four drivers were checked live: none of psycopg2, mysql-connector, pymssql
+  or fdb puts the connection string into a failed connection's message today.
+  That is four third-party packages' current behaviour rather than a guarantee, so
+  it is pinned by tests — a driver upgrade that changes it fails a test instead of
+  filling a production log.
+
+  **What breaks:** two exception *types* change on paths that previously raised a
+  raw driver error. A failure to connect to the target raises
+  `TargetConnectionError` instead of `psycopg2.OperationalError` /
+  `ProgrammingError` (both are caught by `run`'s `except Exception`, so a pipeline
+  sees no difference), and `golden.session.connect` raises `SessionError`, which
+  `dbx-golden` already catches — the CLI now prints one redacted line and exits 2
+  (`ERROR`, its documented code) instead of dumping a traceback and exiting 1. The
+  `🛑 [db] extraction failed` record is emitted with `logging.error` and a rendered
+  traceback rather than `logging.exception`, so the record no longer carries
+  `exc_info`; the text is the same, chained `__cause__` frames included.
+
+- **The characterisation oracle runs without the legacy extractor sources.**
+  The characterisation tests used to need the predecessor's code; without it three
+  test modules were not collected at all and another 154 tests failed.
+  `tests/reference_oracle.py` now has two modes:
+
+  - `live` — the legacy sources are available and answers are extracted from their
+    AST as before. With `DBX_ORACLE_RECORD=1` every call is recorded along the way.
+  - `replay` — the legacy sources are absent and answers come from frozen fixtures
+    in `tests/fixtures/oracle/` (108 files, 1 359 calls, 904 kB).
+
+  The mode is detected from whether that source tree is present; `DBX_ORACLE_MODE`
+  forces it. Fixtures are recorded with `DBX_ORACLE_RECORD=1` where the
+  predecessor sources exist, and CI runs the replay as a separate step so they cannot go
+  stale.
+
+  The format is JSON, not pickle: fixtures can be read by eye, a diff shows what
+  changed, and loading one does not execute foreign code.
+
+  **What is not replayed:** functions with a side effect outside their arguments
+  (`ensure_private_key_permissions` changes file permissions) and functions that
+  take a function as an argument (`with_retry`). They are listed in
+  `reference_oracle.LIVE_ONLY` and their tests are skipped in `replay` mode —
+  78 tests. It is an acknowledged gap, not a silent loss of coverage.
+
+  **What breaks:** nothing. Where the legacy sources are available the behaviour is
+  unchanged and the same 1171 tests pass.
+
+- `scripts/golden_batch.py` carries its own `expand_env`, which it used to import
+  from a migration helper built on top of the legacy extractor sources. That
+  dependency pointed the wrong way: it made `golden_batch` — and with it
+  `tests/golden/test_perturb.py` — impossible to import without the predecessor's
+  code.
+
+### Fixed
+
+- **Three pieces of documentation that described something the code does not do.**
+  None of them changes behaviour; all three were the kind of plausible, readable
+  falsehood that costs an hour of looking in the wrong place.
+
+  - `IncrementalStrategy` claimed to report a `stale_by_days` metric and to warn
+    when it is exceeded. The name appears nowhere else in the package. The claim
+    is removed rather than implemented, and the docstring now says why: detecting
+    the hole an outage leaves needs the interval between two successful runs, and
+    nothing records when a table last ran — every proxy the target can be asked
+    for (`MAX(_timestamp)`, `MAX(updated_at)`) cannot tell a quiet table from one
+    whose changes were missed, so it would warn on healthy tables for ever. The
+    hazard itself is still documented; only the false promise is gone. The metric
+    set is now pinned by a test, because this file had drifted twice.
+  - `docs/mage-loader-block.md` listed six of the ten columns of the returned
+    frame, omitting `load_method` and `error`. The table is corrected; "four extra
+    columns" was right and stays.
+  - `pagination_mode`, `keyset_pagination`, `conflict_columns` and `num_parallel`
+    are parsed and read by no strategy. They **stay** in the contract — a pipeline
+    that sets one has to keep starting — but each field in `core/config.py` now
+    says it is accepted and inert, and `docs/legacy-compat.md` gained a section
+    listing them with the reason. A key that silently does nothing is a trap for
+    whoever sets it expecting an effect.
+
+- **MySQL: a source without an explicit `charset` could not connect.**
+  Configuration parsing deliberately does not fill a charset in — which
+  encoding a database expects is the dialect's knowledge — so `SOURCE_DB`
+  without one reached the dialect as an explicit `None`. `params.get("charset",
+  "utf8mb4")` then returned that `None` rather than the default, the URL became
+  `?charset=None`, and the driver refused it with *"Character set 'None'
+  unsupported"*.
+
+  Every MySQL pipeline that did not spell the charset out was affected. It went
+  unnoticed because no test had ever built a connection against a live server;
+  the very first one did. Firebird already used the `or` form and was never
+  affected.
+
+  **What breaks:** nothing — this only replaces a failure with a working
+  default of `utf8mb4`, which is what the predecessors used.
+
+First public release. The package itself is older than this version number —
+everything before it happened in a private repository and is summarised under
+`[0.1.0]` below. 1.0.0 is where the configuration contract becomes a public
+promise rather than an internal one.
+
+
+### Known limitations
+
+- **MSSQL: `NVARCHAR`, `NCHAR` and `NTEXT` values are truncated at the first
+  character latin-1 cannot express.** `'příliš žluťoučký kůň'` arrives as
+  `'p'` — silently, with no error and no replacement character. Pure ASCII is
+  unaffected, which is why most columns look healthy; a character that *is* in
+  latin-1 arrives as a different character instead of being cut.
+
+  The cause is the connection charset `cp1250`
+  (`MSSQLDialect.connect_args`): with pymssql 2.3.13 the driver converts the
+  server's UCS-2 as latin-1 while the client decodes the result as cp1250.
+
+  **This is inherited, not new.** Both predecessor extractors open the
+  connection with the same charset, so the affected target columns have held
+  the truncated text since they were created. It remains the **default** on
+  purpose: no charset value reads both the N-types and the legacy single-byte
+  columns correctly, and `UTF-8` merely trades the truncation for mojibake in
+  every legacy `VARCHAR` of a CP1250-collated source. Changing it silently would
+  change the target's contents relative to the old side.
+
+  A table can now opt out with `LOAD_SETTINGS.convert_nchar_to_varchar` (see
+  *Added* above), which converts on the server instead of touching the charset.
+  It stays off by default, so this remains the behaviour of every table that does
+  not ask.
+
+  **What breaks:** nothing — this is a description of behaviour that has not
+  changed. The measurements behind the table, and what the opt-in costs, are in
+  [Backward compatibility](docs/legacy-compat.md#mssql-nvarchar-text-arrives-truncated).
+
+## [0.1.0] — pre-release history
+
+Everything below this line happened before publication. The per-version notes from
+that period were internal migration notes rather than a library changelog, so they
+are summarised here instead of being released as public history.
+
+The package replaces a set of hand-copied extractor blocks that had drifted apart
+across several deployments while serving roughly 670 tables. Three constraints
+drove the rewrite and still shape the code:
+
+- **The configuration contract is frozen.** Hundreds of pipeline definitions keep
+  working untouched, including older shapes that predate the sectioned
+  configuration — see [Backward compatibility](docs/legacy-compat.md).
+- **Target column names must not change.** The predecessor wrote through Mage's
+  PostgreSQL exporter, which prefixes an underscore to any name whose uppercase
+  form is one of 825 reserved words; a dbt layer is built on those names. The list
+  is replicated in the package and compared against the live `mage_ai` inside the
+  production image, so a Mage upgrade that touches it fails CI instead of quietly
+  renaming columns.
+- **`_deleted_in_source`, `_timestamp` and `row_hash` are maintained by every
+  strategy**, not only by the ones that happened to support them before.
+
+Capabilities arrived roughly in this order:
+
+1. **Configuration, retry, SSH tunnel and the conversion layer.** Configuration is
+   parsed into a typed dataclass and validated on input; the tunnel is a context
+   manager with `PR_SET_PDEATHSIG` and an explicit `connection_mode`. The
+   conversion layer is vectorised and 2.5x–3.1x faster than its predecessor;
+   throughput is per row *times* column, so table width dominates — 67 000 rows/s
+   at 8 columns, but only 8 600 at 48.
+2. **Target column naming**, replicated from the Mage exporter and verified against
+   a real target: 2 713 columns, 0 differences.
+3. **The write path** (`src/dbextractors/core/target_pg.py`) on bare psycopg2. `COPY … FROM STDIN`
+   is the only write path and reaches 260 000 rows/s; a full load goes into a
+   shadow table swapped in a single transaction, so views survive and an
+   interrupted run leaves the target untouched; deduplication runs through a
+   temporary table instead of a Python `set` holding every key.
+4. **Vectorised `row_hash`** — 8.2x to 9.0x faster than the per-row version and
+   bit-for-bit identical to it.
+5. **The six load strategies**: `full`, `incremental`, `hash_diff` (the dominant
+   one, serving ~530 of the ~670 tables, computing its diff in SQL rather than in
+   RAM), `id_watermark`, `parent_incremental` and `full_by_source`.
+6. **The golden test** (`dbextractors.golden`, CLI `dbx-golden`), which compares
+   two target tables on five levels — row counts, column names and order, types,
+   per-column checksums and per-row `row_hash` — and returns a verdict. No function
+   counted as finished until a golden test for it existed and passed.
+7. **Dialects**: MySQL and PostgreSQL first, then MSSQL and Firebird, each verified
+   against a live source. The largest verified comparison is 25 334 772 rows with
+   no unexplained differences.
+8. **Fixes that only running against live sources could find**: the target is
+   resolved from `io_config.yaml` rather than from environment variables, a
+   connection lost mid-read is retried and resumed from the last key that got
+   through, a source that has gained a column no longer fails the load, and
+   `_timestamp` is accepted as either `text` or `timestamp`.
+9. **Partitioning of the target table** through `LOAD_SETTINGS.partition_by`
+   (`list`, `range_day`, `range_month`, `range_year`), with partitions created from
+   the values the data actually contains.
+
+Two behaviours changed on purpose relative to the predecessor. An unreachable
+source **fails** the run instead of finishing green with zero rows, because a green
+run with zero rows is indistinguishable from "nothing changed in the source" and
+lets a table freeze unnoticed. A missing `primary_column` **fails** as well,
+instead of silently degrading to a full load — the most expensive possible answer
+to a typo in YAML.

dbextractors-1.0.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,17 @@
+# Code of Conduct
+
+This project adopts the [Contributor Covenant, version
+2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
+
+The short version: be decent to people. Harassment, personal attacks and
+demeaning comments are not welcome here, in issues, pull requests or anywhere
+else the project happens. Disagreement about technical decisions is normal and
+expected — this repository is full of choices that look wrong until you know
+the production incident behind them, so ask before assuming, and explain rather
+than dismiss when you are the one who knows.
+
+Reports go to **robert@bisuperhero.cz** and are handled confidentially. The
+maintainer is responsible for enforcement and may remove comments, close
+threads, or ban an account.
+
+The full text, including the enforcement guidelines, is at the link above.