quickhouse 0.3.3__tar.gz → 0.3.4__tar.gz

{quickhouse-0.3.3 → quickhouse-0.3.4}/Cargo.lock

@@ -2634,7 +2634,7 @@ dependencies = [
 
 [[package]]
 name = "quickhouse-core"
-version = "0.3.3"
+version = "0.3.4"
 dependencies = [
  "anyhow",
  "arrow",
@@ -2670,7 +2670,7 @@ dependencies = [
 
 [[package]]
 name = "quickhouse-py"
-version = "0.3.3"
+version = "0.3.4"
 dependencies = [
  "pyo3",
  "quickhouse-core",

{quickhouse-0.3.3 → quickhouse-0.3.4}/Cargo.toml

@@ -3,7 +3,7 @@ resolver = "2"
 members = ["crates/quickhouse-core", "crates/quickhouse-py"]
 
 [workspace.package]
-version = "0.3.3"
+version = "0.3.4"
 edition = "2021"
 license = "MIT"
 authors = ["quickhouse contributors"]

{quickhouse-0.3.3 → quickhouse-0.3.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: quickhouse
-Version: 0.3.3
+Version: 0.3.4
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
@@ -72,6 +72,16 @@ print(result)   # rows_read, rows_written, bytes_written, duration_secs, new_wat
   legacy quirks like MySQL zero-dates or out-of-range timestamps are coerced to
   `NULL` with a warning instead of aborting the run.
 
+- **It's gentle on a small production database.** Set `read_max_rows_per_sec`
+  and quickhouse paces the read to that aggregate rate across all partitions;
+  because `COPY`/streaming results only produce as fast as the client consumes,
+  the source scan itself backs off — you're throttling the database's work, not
+  just your own. Combine it with `parallelism=1` (one connection), incremental
+  mode (only new rows), and a `statement_timeout_secs`, point it at a read
+  replica, and a bulk export stops competing with your app. The Postgres
+  connection also shows up as `application_name = 'quickhouse'` in
+  `pg_stat_activity`, so a DBA can see and kill it.
+
 - **There's nothing to stand up.** `pip install quickhouse` and you're done —
   no JVM, no Spark cluster, no separate service. It's an ordinary Python
   dependency that runs wherever your jobs already run: cron, Airflow, Dagster,
@@ -218,6 +228,7 @@ errors are surfaced verbatim rather than wrapped in something generic.
 | `parallelism` | Concurrent read streams |
 | `batch_rows` / `batch_bytes` | Per-batch size knobs (rows, or estimated bytes) |
 | `max_memory_bytes` | Hard ceiling on total in-flight memory; decoding blocks when hit (default 512 MiB, `0` = unbounded) |
+| `read_max_rows_per_sec` | Cap the aggregate source read rate to be gentle on a small DB; `None` = unlimited (default). Postgres/MySQL only |
 | `type_overrides` | Force a destination column type, e.g. `{"qty": "Decimal(18, 3)"}` |
 | `rename`, `include`, `exclude` | Column renames and allow/deny lists |
 | `on_progress` | Progress callback |
@@ -240,6 +251,13 @@ knowing:
   front, rather than silently falling back to `Float64`.
 - **`TIME`** columns transfer as canonical text into a `String` column
   (ClickHouse has no time-of-day type).
+- **MySQL `DATETIME`/`TIMESTAMP`** map to a UTC-aware timestamp (BigQuery
+  `TIMESTAMP`, ClickHouse `DateTime64(6, 'UTC')`) — the wall-clock value is read
+  as UTC, matching how a `TIMESTAMP` column expects it. To land a column as a
+  naive BigQuery `DATETIME` instead, opt out per-column with
+  `type_overrides={"col": "DATETIME"}` — that flips the actual encoding, not
+  just the declared type. (PostgreSQL keeps the distinction natively:
+  `timestamptz` → UTC-aware, `timestamp` → naive.)
 - **Out-of-range dates** (and MySQL zero-dates like `0000-00-00`) coerce to
   `NULL` with a warning rather than failing the transfer.
 - **Nullable** source columns stay nullable in the destination.

{quickhouse-0.3.3 → quickhouse-0.3.4}/README.md

@@ -40,6 +40,16 @@ print(result)   # rows_read, rows_written, bytes_written, duration_secs, new_wat
   legacy quirks like MySQL zero-dates or out-of-range timestamps are coerced to
   `NULL` with a warning instead of aborting the run.
 
+- **It's gentle on a small production database.** Set `read_max_rows_per_sec`
+  and quickhouse paces the read to that aggregate rate across all partitions;
+  because `COPY`/streaming results only produce as fast as the client consumes,
+  the source scan itself backs off — you're throttling the database's work, not
+  just your own. Combine it with `parallelism=1` (one connection), incremental
+  mode (only new rows), and a `statement_timeout_secs`, point it at a read
+  replica, and a bulk export stops competing with your app. The Postgres
+  connection also shows up as `application_name = 'quickhouse'` in
+  `pg_stat_activity`, so a DBA can see and kill it.
+
 - **There's nothing to stand up.** `pip install quickhouse` and you're done —
   no JVM, no Spark cluster, no separate service. It's an ordinary Python
   dependency that runs wherever your jobs already run: cron, Airflow, Dagster,
@@ -186,6 +196,7 @@ errors are surfaced verbatim rather than wrapped in something generic.
 | `parallelism` | Concurrent read streams |
 | `batch_rows` / `batch_bytes` | Per-batch size knobs (rows, or estimated bytes) |
 | `max_memory_bytes` | Hard ceiling on total in-flight memory; decoding blocks when hit (default 512 MiB, `0` = unbounded) |
+| `read_max_rows_per_sec` | Cap the aggregate source read rate to be gentle on a small DB; `None` = unlimited (default). Postgres/MySQL only |
 | `type_overrides` | Force a destination column type, e.g. `{"qty": "Decimal(18, 3)"}` |
 | `rename`, `include`, `exclude` | Column renames and allow/deny lists |
 | `on_progress` | Progress callback |
@@ -208,6 +219,13 @@ knowing:
   front, rather than silently falling back to `Float64`.
 - **`TIME`** columns transfer as canonical text into a `String` column
   (ClickHouse has no time-of-day type).
+- **MySQL `DATETIME`/`TIMESTAMP`** map to a UTC-aware timestamp (BigQuery
+  `TIMESTAMP`, ClickHouse `DateTime64(6, 'UTC')`) — the wall-clock value is read
+  as UTC, matching how a `TIMESTAMP` column expects it. To land a column as a
+  naive BigQuery `DATETIME` instead, opt out per-column with
+  `type_overrides={"col": "DATETIME"}` — that flips the actual encoding, not
+  just the declared type. (PostgreSQL keeps the distinction natively:
+  `timestamptz` → UTC-aware, `timestamp` → naive.)
 - **Out-of-range dates** (and MySQL zero-dates like `0000-00-00`) coerce to
   `NULL` with a warning rather than failing the transfer.
 - **Nullable** source columns stay nullable in the destination.

{quickhouse-0.3.3 → quickhouse-0.3.4}/crates/quickhouse-core/src/config.rs

@@ -256,6 +256,17 @@ pub struct TransferConfig {
     /// Column used to split the table into parallel range partitions.
     /// Defaults to the first `key` column, else the sync falls back to a single stream.
     pub partition_column: Option<String>,
+    /// Optional cap on how many source rows are pulled **per second**, summed
+    /// across all parallel partitions (a global limiter, not per-connection).
+    /// Deliberately paces the read so a small/production database isn't
+    /// hammered by a bulk export: after each batch is read, the reader sleeps
+    /// long enough to hold the aggregate rate at this ceiling, which — because
+    /// `COPY TO STDOUT` streams only as fast as the client consumes — makes the
+    /// server-side scan itself back off (TCP backpressure), not just the
+    /// client. `None` (default) reads as fast as possible. Applies to the
+    /// PostgreSQL and MySQL sources; ignored for a BigQuery source (its read
+    /// path is a managed, separately-metered API).
+    pub read_max_rows_per_sec: Option<u64>,
 
     // ---- transforms ----
     /// Per-column destination type overrides (column name -> the
@@ -330,6 +341,11 @@ impl TransferConfig {
                 "max_memory_bytes must be 0 (unbounded) or >= 65536",
             ));
         }
+        if self.read_max_rows_per_sec == Some(0) {
+            return Err(EtlError::config(
+                "read_max_rows_per_sec must be None (unlimited) or >= 1",
+            ));
+        }
         Ok(())
     }
 }
@@ -367,6 +383,7 @@ mod tests {
             batch_bytes: 0,
             max_memory_bytes: 0,
             partition_column: None,
+            read_max_rows_per_sec: None,
             type_overrides: HashMap::new(),
             rename: HashMap::new(),
             include: vec![],
@@ -423,4 +440,21 @@ mod tests {
         c.lookback_seconds = 60;
         assert!(c.validate().is_ok());
     }
+
+    #[test]
+    fn validate_rejects_zero_read_rate() {
+        let mut c = cfg(SyncMode::Full, None);
+        c.read_max_rows_per_sec = Some(0);
+        let err = c.validate().unwrap_err().to_string();
+        assert!(err.contains("read_max_rows_per_sec"), "got: {err}");
+    }
+
+    #[test]
+    fn validate_accepts_none_or_positive_read_rate() {
+        let mut c = cfg(SyncMode::Full, None);
+        c.read_max_rows_per_sec = None;
+        assert!(c.validate().is_ok());
+        c.read_max_rows_per_sec = Some(10_000);
+        assert!(c.validate().is_ok());
+    }
 }

{quickhouse-0.3.3 → quickhouse-0.3.4}/crates/quickhouse-core/src/ddl.rs

@@ -147,6 +147,7 @@ mod tests {
             batch_bytes: 0,
             max_memory_bytes: 0,
             partition_column: None,
+            read_max_rows_per_sec: None,
             type_overrides: HashMap::new(),
             rename: HashMap::new(),
             include: vec![],

{quickhouse-0.3.3 → quickhouse-0.3.4}/crates/quickhouse-core/src/sink/bigquery.rs

@@ -945,6 +945,7 @@ mod tests {
             batch_bytes: 0,
             max_memory_bytes: 0,
             partition_column: None,
+            read_max_rows_per_sec: None,
             type_overrides: HashMap::new(),
             rename: HashMap::new(),
             include: vec![],

{quickhouse-0.3.3 → quickhouse-0.3.4}/crates/quickhouse-core/src/source/postgres.rs

@@ -92,14 +92,22 @@ impl PgSource {
                 tracing::warn!("postgres connection error: {e}");
             }
         });
+        // Session setup, run as one round-trip. `application_name` makes this
+        // export visible (and killable) in `pg_stat_activity` — exactly what a
+        // DBA wants when watching load from a bulk read. When a statement
+        // timeout is configured we also cap `idle_in_transaction_session_timeout`
+        // at the same value: defense-in-depth so a stalled connection can never
+        // sit inside an open transaction holding an MVCC snapshot (which would
+        // block vacuum and bloat the source). The bulk COPY itself runs in
+        // autocommit, so statement_timeout is the primary guard; this covers the
+        // edge/future transactional paths.
+        let mut setup = String::from("SET application_name = 'quickhouse'");
         if self.statement_timeout_secs > 0 {
-            client
-                .batch_execute(&format!(
-                    "SET statement_timeout = {}",
-                    self.statement_timeout_secs * 1000
-                ))
-                .await?;
+            let ms = self.statement_timeout_secs * 1000;
+            setup.push_str(&format!("; SET statement_timeout = {ms}"));
+            setup.push_str(&format!("; SET idle_in_transaction_session_timeout = {ms}"));
         }
+        client.batch_execute(&setup).await?;
         Ok(client)
     }
 
@@ -132,7 +140,7 @@ impl PgSource {
                 column: c.name().to_string(),
                 type_name: c.type_().name().to_string(),
             })?;
-            let nullable = !not_null.contains(&c.name().to_string());
+            let nullable = !not_null.contains(c.name());
             cols.push(ColumnType {
                 name: c.name().to_string(),
                 type_id: pg_oid,

{quickhouse-0.3.3 → quickhouse-0.3.4}/crates/quickhouse-core/src/sync.rs

@@ -5,8 +5,8 @@
 //! source-agnostic.
 
 use std::sync::atomic::{AtomicU64, Ordering};
-use std::sync::Arc;
-use std::time::Instant;
+use std::sync::{Arc, Mutex};
+use std::time::{Duration, Instant};
 
 use arrow_array::RecordBatch;
 use arrow_schema::SchemaRef;
@@ -54,6 +54,60 @@ struct Counters {
     bytes_written: AtomicU64,
 }
 
+/// Global source-read rate limiter, shared across every parallel partition so
+/// the ceiling is an *aggregate* rows/sec (not per-connection). Uses a
+/// virtual-scheduling (GCRA-style) clock: each `acquire(n)` reserves `n / rate`
+/// seconds of read time by pushing a shared `next_available` instant forward,
+/// and the caller sleeps until its reserved slot. Because reads are gated
+/// *after* each batch, and `COPY TO STDOUT` (and MySQL's streaming result) only
+/// produce as fast as the client consumes, pausing here applies TCP
+/// backpressure that slows the server-side scan itself — the whole point of the
+/// knob. No burst credit accumulates while idle (`next_available` is clamped
+/// forward to `now`, never left in the past), which is the safe choice for
+/// "be gentle to the source".
+struct ReadThrottle {
+    rate_per_sec: f64,
+    next_available: Mutex<Instant>,
+}
+
+impl ReadThrottle {
+    fn new(rows_per_sec: u64) -> Self {
+        Self {
+            rate_per_sec: rows_per_sec as f64,
+            next_available: Mutex::new(Instant::now()),
+        }
+    }
+
+    /// Reserve capacity for `rows` just-read rows and return how long the
+    /// caller must wait before pulling more. Pure of any `.await` (so the
+    /// `std::sync::Mutex` is never held across a suspension point, and the math
+    /// is unit-testable without sleeping); `acquire` wraps it with the sleep.
+    fn reserve(&self, rows: u64) -> Duration {
+        // A zero rate can't happen via the public API (`validate` rejects
+        // `Some(0)` before any throttle is built), but guard anyway so a Rust
+        // caller constructing `TransferConfig` directly gets a safe no-op
+        // instead of an `inf` `Duration` panic (`rows / 0.0`) in the read loop.
+        // `rate_per_sec` is a `u64 as f64`, so it's always finite and >= 0.
+        if rows == 0 || self.rate_per_sec <= 0.0 {
+            return Duration::ZERO;
+        }
+        let mut next = self.next_available.lock().unwrap();
+        let now = Instant::now();
+        // Idle time grants no credit: never start earlier than `now`.
+        let start = (*next).max(now);
+        let cost = Duration::from_secs_f64(rows as f64 / self.rate_per_sec);
+        *next = start + cost;
+        start.saturating_duration_since(now)
+    }
+
+    async fn acquire(&self, rows: u64) {
+        let wait = self.reserve(rows);
+        if !wait.is_zero() {
+            tokio::time::sleep(wait).await;
+        }
+    }
+}
+
 /// Shared handle for uploading finished batches. Cloned cheaply into each
 /// partition task (all fields are `Arc`/`Copy`), so every partition spawns
 /// sends against the *same* memory budget and counters.
@@ -69,6 +123,9 @@ struct SendCtx {
     /// `None` otherwise (including always for BigQuery). See
     /// `ArchiveRunInfo::writer_for`.
     archive: Option<Arc<ArchiveRunInfo>>,
+    /// `Some` when `read_max_rows_per_sec` is set: a single limiter shared by
+    /// all partition tasks, so the cap is an aggregate across the whole read.
+    throttle: Option<Arc<ReadThrottle>>,
 }
 
 impl SendCtx {
@@ -354,6 +411,11 @@ async fn run_transfer_impl(
 
         // --- Fan out partitions with bounded concurrency. ---
         let counters = Arc::new(Counters::default());
+        // One limiter shared across every partition, so `read_max_rows_per_sec`
+        // caps the *aggregate* read rate regardless of `parallelism`.
+        let throttle = cfg
+            .read_max_rows_per_sec
+            .map(|r| Arc::new(ReadThrottle::new(r)));
         let cfg = Arc::new(cfg);
         let extra_filter = Arc::new(extra_filter);
         let target_table = Arc::new(target_table);
@@ -365,6 +427,7 @@ async fn run_transfer_impl(
             progress: progress.clone(),
             started,
             archive: archive_info.clone(),
+            throttle,
         };
 
         let mut results = futures::stream::iter(partitions.into_iter().map(|part| {
@@ -549,6 +612,10 @@ async fn run_transfer_bigquery(
             progress: progress.clone(),
             started,
             archive: archive_info,
+            // read_max_rows_per_sec is a Postgres/MySQL knob; the BigQuery
+            // Storage Read API is a separately-metered managed service, so this
+            // path never throttles.
+            throttle: None,
         };
 
         let mut iter = source
@@ -767,10 +834,16 @@ async fn transfer_partition_postgres(
         let chunk = chunk?;
         let batches = decoder.feed(&chunk)?;
         for batch in batches {
+            let rows = batch.num_rows() as u64;
             if let Some(w) = archive_writer.as_mut() {
                 w.write(&batch).await?;
             }
             ctx.spawn_upload(&mut sends, schema.clone(), batch).await;
+            // Pace the read: pausing before pulling the next chunk applies TCP
+            // backpressure to the COPY stream, slowing the server-side scan.
+            if let Some(t) = &ctx.throttle {
+                t.acquire(rows).await;
+            }
         }
         reap(&mut sends, false).await?; // surface any upload error promptly
     }
@@ -853,11 +926,17 @@ async fn transfer_partition_mysql(
     while let Some(row) = stream.next().await {
         let row = row.map_err(|e| EtlError::other(format!("mysql row error: {e}")))?;
         if let Some(batch) = batcher.append_row(row)? {
+            let rows = batch.num_rows() as u64;
             if let Some(w) = archive_writer.as_mut() {
                 w.write(&batch).await?;
             }
             ctx.spawn_upload(&mut sends, schema.clone(), batch).await;
             reap(&mut sends, false).await?; // surface any upload error promptly
+            // Pace the read: pausing before fetching more rows applies
+            // backpressure to the streaming result set, slowing the scan.
+            if let Some(t) = &ctx.throttle {
+                t.acquire(rows).await;
+            }
         }
     }
     if let Some(batch) = batcher.finish()? {
@@ -1374,6 +1453,59 @@ mod tests {
         assert_ne!(staging_name("orders", "1"), staging_name("orders", "2"));
     }
 
+    #[test]
+    fn throttle_reserve_zero_is_immediate() {
+        let t = ReadThrottle::new(1000);
+        assert!(t.reserve(0).is_zero());
+    }
+
+    #[test]
+    fn throttle_zero_rate_is_safe_noop_not_panic() {
+        // `validate` prevents this via the public API, but a direct Rust
+        // caller must not trip an `inf` Duration panic (rows / 0.0).
+        let t = ReadThrottle::new(0);
+        assert!(t.reserve(1_000_000).is_zero());
+    }
+
+    #[test]
+    fn throttle_first_reserve_does_not_wait() {
+        // The limiter starts unfilled: the first read is never delayed
+        // (`next_available` begins at construction time, already in the past by
+        // the time we reserve, so `start` clamps to `now` and the wait is 0).
+        let t = ReadThrottle::new(1000);
+        assert!(t.reserve(500).is_zero());
+    }
+
+    #[test]
+    fn throttle_paces_subsequent_reads_by_rate() {
+        // At 1000 rows/s, reserving 1000 rows pushes the schedule ~1s forward,
+        // so the very next reservation must wait close to a full second. Bounds
+        // are wide (>0.5s, <1s) so only genuine pacing — not scheduling jitter
+        // in the microseconds between two calls — can satisfy them.
+        let t = ReadThrottle::new(1000);
+        assert!(t.reserve(1000).is_zero());
+        let wait = t.reserve(1000);
+        assert!(
+            wait > Duration::from_millis(500) && wait < Duration::from_secs(1),
+            "expected ~1s pacing wait, got {wait:?}"
+        );
+    }
+
+    #[test]
+    fn throttle_is_shared_across_handles() {
+        // Two clones of the same Arc share one schedule, so the cap is an
+        // aggregate across all partitions rather than per-connection: after one
+        // handle consumes a full second of budget, the other must wait.
+        let t = Arc::new(ReadThrottle::new(1000));
+        let t2 = t.clone();
+        assert!(t.reserve(1000).is_zero());
+        let wait = t2.reserve(1000);
+        assert!(
+            wait > Duration::from_millis(500),
+            "a second handle should be paced by the first's reservation, got {wait:?}"
+        );
+    }
+
     fn col(name: &str) -> ColumnType {
         col_typed(name, DataType::Int64)
     }

{quickhouse-0.3.3 → quickhouse-0.3.4}/crates/quickhouse-core/src/transform.rs

@@ -13,15 +13,60 @@
 //! `decimal.rs`.
 
 use std::collections::HashSet;
+use std::sync::Arc;
 
 use arrow_array::types::{validate_decimal_precision_and_scale, Decimal128Type};
-use arrow_schema::DataType;
+use arrow_schema::{DataType, TimeUnit};
 
 use crate::config::TransferConfig;
 use crate::decimal::parse_decimal_override;
 use crate::error::{EtlError, Result};
 use crate::types::{may_coerce_to_null, ColumnType};
 
+/// Classify a destination datetime type name into the Arrow timezone it
+/// implies: `Some(Some(tz))` = tz-aware (BigQuery `TIMESTAMP` / ClickHouse
+/// `DateTime64(P, 'tz')`), `Some(None)` = naive (BigQuery `DATETIME` /
+/// ClickHouse `DateTime64(P)`), and `None` = not a datetime type at all (so
+/// the source Arrow type is left untouched).
+///
+/// This is what makes a `type_overrides` entry flip a datetime column's
+/// *encoding*, not just its DDL string. MySQL datetimes now default to
+/// tz-aware UTC (see `types::map_mysql_type`), so `type_overrides={col:
+/// "DATETIME"}` is the per-column opt-out back to naive — and because the Arrow
+/// tz flag drives the BigQuery Storage Write proto encoding (Int64 micros vs
+/// civil string) and the destination column type, flipping it here fixes the
+/// whole chain, which a DDL-only relabel could not (it would leave a
+/// TIMESTAMP-encoded payload aimed at a DATETIME column, or vice-versa).
+///
+/// Recognizes BigQuery names (`TIMESTAMP`/`DATETIME`) and ClickHouse names
+/// (`DateTime`/`DateTime64(...)`), case-insensitively.
+fn datetime_override_tz(dest_type: &str) -> Option<Option<Arc<str>>> {
+    let trimmed = dest_type.trim();
+    let upper = trimmed.to_ascii_uppercase();
+    // BigQuery scalar names are exact.
+    if upper == "TIMESTAMP" {
+        return Some(Some(Arc::from("UTC")));
+    }
+    if upper == "DATETIME" {
+        return Some(None);
+    }
+    // ClickHouse `DateTime` / `DateTime64(P[, 'tz']])`: tz-aware iff a quoted
+    // timezone argument is present.
+    if upper.starts_with("DATETIME") {
+        return Some(extract_quoted_tz(trimmed));
+    }
+    None
+}
+
+/// Extract a single-quoted timezone argument (e.g. `UTC` from
+/// `DateTime64(6, 'UTC')`), or `None` when the type carries no timezone.
+fn extract_quoted_tz(s: &str) -> Option<Arc<str>> {
+    let start = s.find('\'')?;
+    let rest = &s[start + 1..];
+    let end = rest.find('\'')?;
+    Some(Arc::from(&rest[..end]))
+}
+
 #[derive(Debug)]
 pub struct SelectPlan {
     /// Source column names to read, in order (drives the COPY SELECT list).
@@ -133,6 +178,20 @@ pub fn plan(source: &[ColumnType], cfg: &TransferConfig) -> Result<SelectPlan> {
                 }
                 None => c.arrow.clone(),
             }
+        } else if matches!(c.arrow, DataType::Timestamp(_, _)) {
+            // A datetime column's tz-awareness follows its (possibly
+            // overridden) destination type — this is the per-column opt-out
+            // that lets `type_overrides={col: "DATETIME"}` render a
+            // (UTC-by-default) MySQL datetime as a naive BigQuery DATETIME, and
+            // the reverse. With no override, `ch_inner` is the source's own
+            // default mapping string, so this reproduces the source tz exactly
+            // (no behavior change). A non-datetime override (e.g. "String")
+            // returns `None` and leaves the Arrow timestamp untouched, matching
+            // the pre-existing DDL-only-relabel behavior for such overrides.
+            match datetime_override_tz(&ch_inner) {
+                Some(tz) => DataType::Timestamp(TimeUnit::Microsecond, tz),
+                None => c.arrow.clone(),
+            }
         } else {
             c.arrow.clone()
         };
@@ -219,6 +278,7 @@ mod tests {
             batch_bytes: 0,
             max_memory_bytes: 0,
             partition_column: None,
+            read_max_rows_per_sec: None,
             type_overrides: HashMap::new(),
             rename: HashMap::new(),
             include: vec![],
@@ -427,4 +487,80 @@ mod tests {
         assert!(err.contains("amount"), "must name the column: {err}");
         assert!(err.contains("Decimal256"), "must mention the Decimal256 follow-up: {err}");
     }
+
+    fn ts(tz: Option<&str>) -> DataType {
+        DataType::Timestamp(TimeUnit::Microsecond, tz.map(Arc::from))
+    }
+
+    /// The classifier that drives the per-column datetime opt-out: BigQuery and
+    /// ClickHouse type names, tz-aware vs naive, and non-datetime strings.
+    #[test]
+    fn datetime_override_tz_classifies_type_names() {
+        // tz-aware
+        assert_eq!(datetime_override_tz("TIMESTAMP"), Some(Some(Arc::from("UTC"))));
+        assert_eq!(datetime_override_tz("timestamp"), Some(Some(Arc::from("UTC"))));
+        assert_eq!(datetime_override_tz("DateTime64(6, 'UTC')"), Some(Some(Arc::from("UTC"))));
+        assert_eq!(datetime_override_tz("DateTime('Asia/Jakarta')"), Some(Some(Arc::from("Asia/Jakarta"))));
+        // naive
+        assert_eq!(datetime_override_tz("DATETIME"), Some(None));
+        assert_eq!(datetime_override_tz("DateTime64(6)"), Some(None));
+        assert_eq!(datetime_override_tz("DateTime"), Some(None));
+        // not a datetime type at all -> leave the Arrow type alone
+        assert_eq!(datetime_override_tz("String"), None);
+        assert_eq!(datetime_override_tz("Int64"), None);
+        assert_eq!(datetime_override_tz("Decimal(18, 2)"), None);
+    }
+
+    /// A `type_overrides={col: "DATETIME"}` on a (UTC-by-default) MySQL-style
+    /// timestamp column flips its Arrow tz to naive — the per-column opt-out
+    /// that makes it land as a BigQuery DATETIME. This drives the Storage Write
+    /// proto encoding, not just the DDL string, so it's the only thing that
+    /// actually works (a DDL-only relabel would leave a mismatched payload).
+    #[test]
+    fn datetime_override_flips_utc_timestamp_to_naive() {
+        let src = vec![typed_col("event_at", ts(Some("UTC")), true)];
+        let mut cfg = cfg();
+        cfg.type_overrides = HashMap::from([("event_at".to_string(), "DATETIME".to_string())]);
+        let p = plan(&src, &cfg).unwrap();
+        assert_eq!(p.dest_columns[0].arrow, ts(None), "override must flip tz-aware -> naive");
+        assert_eq!(p.dest_columns[0].clickhouse_inner, "DATETIME");
+    }
+
+    /// The reverse: an override to `TIMESTAMP` promotes a naive timestamp
+    /// column to tz-aware UTC (BigQuery TIMESTAMP).
+    #[test]
+    fn datetime_override_promotes_naive_timestamp_to_utc() {
+        let src = vec![typed_col("event_at", ts(None), true)];
+        let mut cfg = cfg();
+        cfg.type_overrides = HashMap::from([("event_at".to_string(), "TIMESTAMP".to_string())]);
+        let p = plan(&src, &cfg).unwrap();
+        assert_eq!(p.dest_columns[0].arrow, ts(Some("UTC")), "override must flip naive -> tz-aware UTC");
+    }
+
+    /// With no override, a timestamp column keeps its source tz exactly — the
+    /// classifier reproduces the default from the source's own `clickhouse_inner`
+    /// mapping string, so this is a no-op for existing behavior.
+    #[test]
+    fn timestamp_without_override_keeps_source_tz() {
+        let mut utc_col = typed_col("a", ts(Some("UTC")), true);
+        utc_col.clickhouse_inner = "DateTime64(6, 'UTC')".into();
+        let mut naive_col = typed_col("b", ts(None), true);
+        naive_col.clickhouse_inner = "DateTime64(6)".into();
+        let p = plan(&[utc_col, naive_col], &cfg()).unwrap();
+        assert_eq!(p.dest_columns[0].arrow, ts(Some("UTC")), "UTC source stays UTC");
+        assert_eq!(p.dest_columns[1].arrow, ts(None), "naive source stays naive");
+    }
+
+    /// A non-datetime override (e.g. storing a timestamp as text) leaves the
+    /// Arrow timestamp untouched — preserving the pre-existing DDL-only-relabel
+    /// behavior for such overrides (ClickHouse coerces at insert time).
+    #[test]
+    fn non_datetime_override_leaves_timestamp_arrow_untouched() {
+        let src = vec![typed_col("event_at", ts(Some("UTC")), true)];
+        let mut cfg = cfg();
+        cfg.type_overrides = HashMap::from([("event_at".to_string(), "String".to_string())]);
+        let p = plan(&src, &cfg).unwrap();
+        assert_eq!(p.dest_columns[0].arrow, ts(Some("UTC")), "arrow type unchanged by a non-datetime override");
+        assert_eq!(p.dest_columns[0].clickhouse_inner, "String", "DDL string still honored");
+    }
 }

{quickhouse-0.3.3 → quickhouse-0.3.4}/crates/quickhouse-core/src/types.rs

@@ -335,12 +335,29 @@ pub mod mysql {
             MyType::MYSQL_TYPE_DATE | MyType::MYSQL_TYPE_NEWDATE => {
                 (DataType::Date32, "Date32".to_string())
             }
+            // MySQL DATETIME/TIMESTAMP default to a tz-aware UTC mapping
+            // (Arrow `Some("UTC")` -> BigQuery `TIMESTAMP` / ClickHouse
+            // `DateTime64(6, 'UTC')`), NOT the tz-naive arm. Rationale: the
+            // decoder already interprets the wall-clock value as UTC
+            // (`dt.and_utc().timestamp_micros()`, decode_mysql.rs) and MySQL
+            // has no wire type that would ever reach a tz-aware arm otherwise,
+            // so a MySQL datetime could previously only ever land as BigQuery
+            // DATETIME — making it impossible to sync into an existing BigQuery
+            // `TIMESTAMP` column (fails the staging->dest MERGE with "Value of
+            // type DATETIME cannot be assigned to <col>, which has type
+            // TIMESTAMP"). This matches the legacy pandas->to_gbq semantics
+            // (naive wall-clock stored into BQ TIMESTAMP as UTC — same instant)
+            // and the Postgres `TIMESTAMPTZ` mapping. A column that genuinely
+            // wants naive (BigQuery DATETIME / `DateTime64(6)`) opts out
+            // per-column via `type_overrides={col: "DATETIME"}` — see
+            // `transform::datetime_override_tz`, which flips the Arrow tz flag
+            // (and thus the Storage Write proto encoding), not just the DDL.
             MyType::MYSQL_TYPE_DATETIME
             | MyType::MYSQL_TYPE_DATETIME2
             | MyType::MYSQL_TYPE_TIMESTAMP
             | MyType::MYSQL_TYPE_TIMESTAMP2 => (
-                DataType::Timestamp(arrow_schema::TimeUnit::Microsecond, None),
-                "DateTime64(6)".to_string(),
+                DataType::Timestamp(arrow_schema::TimeUnit::Microsecond, Some("UTC".into())),
+                "DateTime64(6, 'UTC')".to_string(),
             ),
             // TIME -> text into a ClickHouse String column. MySQL TIME can be
             // negative and exceed 24h (range +/-838:59:59), which no time-of-day
@@ -389,6 +406,30 @@ mod tests {
         assert!(map_oid(oid::TIMESTAMPTZ).unwrap().1.contains("UTC"));
     }
 
+    /// MySQL DATETIME/TIMESTAMP default to a tz-aware UTC mapping so they can
+    /// land in a BigQuery `TIMESTAMP` column (see the map arm's comment). The
+    /// decoder computes the same UTC epoch micros regardless of this flag, so
+    /// this only changes the destination type, not the stored instant.
+    #[test]
+    fn mysql_datetime_defaults_to_utc_aware() {
+        use super::mysql::map_mysql_type;
+        use mysql_async::consts::ColumnType as MyType;
+        for ty in [
+            MyType::MYSQL_TYPE_DATETIME,
+            MyType::MYSQL_TYPE_DATETIME2,
+            MyType::MYSQL_TYPE_TIMESTAMP,
+            MyType::MYSQL_TYPE_TIMESTAMP2,
+        ] {
+            let (arrow, ch) = map_mysql_type(ty, false, false).unwrap();
+            assert_eq!(
+                arrow,
+                DataType::Timestamp(arrow_schema::TimeUnit::Microsecond, Some("UTC".into())),
+                "{ty:?} must map to a tz-aware (UTC) Arrow timestamp"
+            );
+            assert_eq!(ch, "DateTime64(6, 'UTC')", "{ty:?} ClickHouse type");
+        }
+    }
+
     #[test]
     fn nullable_wrapping() {
         let c = ColumnType {

{quickhouse-0.3.3 → quickhouse-0.3.4}/crates/quickhouse-py/src/lib.rs

@@ -479,6 +479,7 @@ fn parse_parquet_compression(c: &str) -> PyResult<core::ParquetCompression> {
     batch_bytes=4_194_304,
     max_memory_bytes=536_870_912,
     partition_column=None,
+    read_max_rows_per_sec=None,
     type_overrides=None,
     rename=None,
     include=None,
@@ -507,6 +508,7 @@ fn sync(
     batch_bytes: usize,
     max_memory_bytes: usize,
     partition_column: Option<String>,
+    read_max_rows_per_sec: Option<u64>,
     type_overrides: Option<HashMap<String, String>>,
     rename: Option<HashMap<String, String>>,
     include: Option<Vec<String>>,
@@ -534,6 +536,7 @@ fn sync(
         batch_bytes,
         max_memory_bytes,
         partition_column,
+        read_max_rows_per_sec,
         type_overrides: type_overrides.unwrap_or_default(),
         rename: rename.unwrap_or_default(),
         include: include.unwrap_or_default(),

{quickhouse-0.3.3 → quickhouse-0.3.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "quickhouse"
-version = "0.3.3"
+version = "0.3.4"
 description = "Fast PostgreSQL/MySQL/BigQuery -> ClickHouse ETL with a Rust engine (parallel, bounded-memory, Arrow-based)."
 readme = "README.md"
 requires-python = ">=3.9"

{quickhouse-0.3.3 → quickhouse-0.3.4}/python/quickhouse/_quickhouse.pyi

@@ -232,6 +232,7 @@ def sync(
     batch_bytes: int = 4_194_304,
     max_memory_bytes: int = 536_870_912,
     partition_column: Optional[str] = None,
+    read_max_rows_per_sec: Optional[int] = None,
     type_overrides: Optional[Mapping[str, str]] = None,
     rename: Optional[Mapping[str, str]] = None,
     include: Optional[Sequence[str]] = None,
@@ -291,6 +292,38 @@ def sync(
       with concurrent uploads and blocks (backpressure) when this ceiling is
       reached, so peak RSS stays bounded regardless of ``parallelism`` or row
       width. Default 512 MiB; ``0`` disables the ceiling (unbounded).
+
+    Being gentle to a small source database:
+
+    - ``read_max_rows_per_sec`` caps how many source rows are pulled per
+      second, summed across *all* parallel partitions (a global limiter, not
+      per-connection). After each batch is read the reader pauses to hold the
+      aggregate rate at this ceiling; because ``COPY``/streaming results only
+      produce as fast as the client consumes, that pause pushes back on the
+      server-side scan itself, so the source does proportionally less work —
+      not just quickhouse. ``None`` (default) reads as fast as possible.
+      Applies to PostgreSQL and MySQL sources; ignored for a BigQuery source
+      (its read path is a separately-metered managed API). For the lightest
+      possible footprint on a small instance, combine a modest
+      ``read_max_rows_per_sec`` with ``parallelism=1`` (one connection, one
+      scan), ``mode="incremental"`` (reads only new rows, not the whole
+      table), and a ``statement_timeout_secs`` on the ``Postgres``/``MySQL``
+      connection. The Postgres connection also reports itself as
+      ``application_name = 'quickhouse'`` so the export is visible (and
+      killable) in ``pg_stat_activity``.
+
+    Datetime/timezone handling:
+
+    - MySQL ``DATETIME``/``TIMESTAMP`` map to a UTC-aware timestamp — BigQuery
+      ``TIMESTAMP``, ClickHouse ``DateTime64(6, 'UTC')`` — reading the
+      wall-clock value as UTC (the same instant the legacy pandas/``to_gbq``
+      path stored, and what an existing BigQuery ``TIMESTAMP`` column expects).
+      To land a column as a naive BigQuery ``DATETIME`` (or ClickHouse
+      ``DateTime64(6)``) instead, opt out per-column with
+      ``type_overrides={"col": "DATETIME"}``; this flips the actual wire
+      encoding, not just the declared destination type, so it works on the
+      Storage Write path too. PostgreSQL keeps the distinction from the source
+      type: ``timestamptz`` → UTC-aware, ``timestamp`` → naive.
     """
     ...