siftq 0.1.0__tar.gz

siftq-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ken
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

siftq-0.1.0/PKG-INFO

@@ -0,0 +1,94 @@
+Metadata-Version: 2.4
+Name: siftq
+Version: 0.1.0
+Summary: Big-file data prep that never runs out of memory — no SQL required. Powered by DuckDB.
+Author: Ken
+License: MIT
+Keywords: duckdb,data,csv,parquet,etl,cli,big-data,dataframe
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: duckdb>=0.10
+Provides-Extra: mem
+Requires-Dist: psutil>=5.9; extra == "mem"
+Dynamic: license-file
+
+# sift
+
+**Big-file data prep that never runs out of memory — no SQL required.**
+
+`sift` is a tiny command-line tool for cleaning and reshaping data files
+(CSV, Parquet, JSON) that are *too big for pandas*. It's a friendly front-end
+over [DuckDB](https://duckdb.org): DuckDB does the heavy lifting (streaming,
+disk-spill, all your CPU cores) and `sift` makes it a one-liner — and
+auto-configures memory so your job doesn't crash.
+
+```bash
+pip install siftq            # pip name; the command is `dq`
+```
+
+## Why
+
+- **It doesn't OOM.** Memory is capped to a fraction of *free* RAM and DuckDB
+  spills to disk instead of dying. Point it at a 10 GB file on a laptop; it's fine.
+- **No SQL, no pandas.** Simple verbs, or a readable recipe file.
+- **One streaming pass.** A whole recipe compiles to a single query — no
+  intermediate files, so it's fast and light.
+- **Any format.** Reads/writes CSV, Parquet, JSON — auto-detected by extension.
+
+## One-liners
+
+```bash
+dq profile data.parquet                       # schema + row count, instantly
+dq keep    data.parquet --cols id,lat,lon -o small.csv
+dq drop    data.csv     --cols ip,ua       -o clean.parquet
+dq filter  data.parquet --where "carrier = 'DU'" -o du.csv
+dq dedup   data.csv     --on device_id     -o unique.parquet
+dq sample  data.parquet --n 50000          -o sample.csv
+dq clip    pings.parquet --region uae       -o uae.parquet
+dq convert data.parquet -o data.csv          # just change format
+```
+
+## Recipes
+
+Chain steps in a readable `.dq` file — they run as one pass:
+
+```yaml
+# du_clean.dq
+input:  data/raw/Feb2.parquet
+keep:   [device_id, lat, lon, carrier]
+clip:   uae                 # drop global GPS noise
+filter: carrier = 'DU'
+dedup:  device_id
+sample: 50000
+output: outputs/du_clean.csv
+```
+
+```bash
+dq run du_clean.dq
+```
+
+Real run: **54.6M rows / 2.5 GB parquet → 50k-row clean CSV in ~1 second.**
+
+## Install for max memory-safety
+
+```bash
+pip install "siftq[mem]"       # adds psutil for smarter RAM sizing
+```
+
+## Roadmap
+
+Core is intentionally small and grows release by release: `join`, `rename`,
+`split` (by column into many files), `stats`, timestamp shifting, and an
+optional geo/mobility extension. Ideas welcome.
+
+---
+
+> **Naming:** the project is **sift**; it installs from PyPI as **`siftq`**
+> (`sift` was taken) and gives you the **`dq`** command.
+
+MIT licensed.

siftq-0.1.0/README.md

@@ -0,0 +1,75 @@
+# sift
+
+**Big-file data prep that never runs out of memory — no SQL required.**
+
+`sift` is a tiny command-line tool for cleaning and reshaping data files
+(CSV, Parquet, JSON) that are *too big for pandas*. It's a friendly front-end
+over [DuckDB](https://duckdb.org): DuckDB does the heavy lifting (streaming,
+disk-spill, all your CPU cores) and `sift` makes it a one-liner — and
+auto-configures memory so your job doesn't crash.
+
+```bash
+pip install siftq            # pip name; the command is `dq`
+```
+
+## Why
+
+- **It doesn't OOM.** Memory is capped to a fraction of *free* RAM and DuckDB
+  spills to disk instead of dying. Point it at a 10 GB file on a laptop; it's fine.
+- **No SQL, no pandas.** Simple verbs, or a readable recipe file.
+- **One streaming pass.** A whole recipe compiles to a single query — no
+  intermediate files, so it's fast and light.
+- **Any format.** Reads/writes CSV, Parquet, JSON — auto-detected by extension.
+
+## One-liners
+
+```bash
+dq profile data.parquet                       # schema + row count, instantly
+dq keep    data.parquet --cols id,lat,lon -o small.csv
+dq drop    data.csv     --cols ip,ua       -o clean.parquet
+dq filter  data.parquet --where "carrier = 'DU'" -o du.csv
+dq dedup   data.csv     --on device_id     -o unique.parquet
+dq sample  data.parquet --n 50000          -o sample.csv
+dq clip    pings.parquet --region uae       -o uae.parquet
+dq convert data.parquet -o data.csv          # just change format
+```
+
+## Recipes
+
+Chain steps in a readable `.dq` file — they run as one pass:
+
+```yaml
+# du_clean.dq
+input:  data/raw/Feb2.parquet
+keep:   [device_id, lat, lon, carrier]
+clip:   uae                 # drop global GPS noise
+filter: carrier = 'DU'
+dedup:  device_id
+sample: 50000
+output: outputs/du_clean.csv
+```
+
+```bash
+dq run du_clean.dq
+```
+
+Real run: **54.6M rows / 2.5 GB parquet → 50k-row clean CSV in ~1 second.**
+
+## Install for max memory-safety
+
+```bash
+pip install "siftq[mem]"       # adds psutil for smarter RAM sizing
+```
+
+## Roadmap
+
+Core is intentionally small and grows release by release: `join`, `rename`,
+`split` (by column into many files), `stats`, timestamp shifting, and an
+optional geo/mobility extension. Ideas welcome.
+
+---
+
+> **Naming:** the project is **sift**; it installs from PyPI as **`siftq`**
+> (`sift` was taken) and gives you the **`dq`** command.
+
+MIT licensed.

siftq-0.1.0/pyproject.toml

@@ -0,0 +1,29 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "siftq"                           # pip name (brand: "sift"; command: dq)
+version = "0.1.0"
+description = "Big-file data prep that never runs out of memory — no SQL required. Powered by DuckDB."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "Ken" }]
+keywords = ["duckdb", "data", "csv", "parquet", "etl", "cli", "big-data", "dataframe"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "License :: OSI Approved :: MIT License",
+  "Topic :: Scientific/Engineering",
+  "Topic :: Utilities",
+]
+dependencies = ["duckdb>=0.10"]
+
+[project.optional-dependencies]
+mem = ["psutil>=5.9"]                     # better auto memory-sizing (optional)
+
+[project.scripts]
+dq = "sift.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

siftq-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

siftq-0.1.0/src/sift/__init__.py

@@ -0,0 +1,11 @@
+"""sift — big-file data prep that never runs out of memory.
+
+One `pip install`, then simple commands. DuckDB does the heavy lifting under
+the hood (streaming, disk-spill, all cores); sift just makes it a one-liner
+and auto-configures memory so it doesn't OOM.
+"""
+from .engine import connect
+from .ops import run_spec, profile, build_query
+
+__version__ = "0.1.0"
+__all__ = ["connect", "run_spec", "profile", "build_query", "__version__"]

siftq-0.1.0/src/sift/__main__.py

@@ -0,0 +1,4 @@
+from .cli import main
+
+if __name__ == "__main__":
+    main()

siftq-0.1.0/src/sift/cli.py

@@ -0,0 +1,102 @@
+"""`dq` command-line front-end. Every subcommand builds a spec and hands it to
+the same streaming engine, so a one-liner and a recipe run identically.
+"""
+from __future__ import annotations
+
+import argparse
+
+from . import __version__
+from .ops import profile, run_spec
+from .recipe import parse as parse_recipe
+
+
+def _io(sp):
+    sp.add_argument("input")
+    sp.add_argument("-o", "--output", required=True)
+
+
+def build_parser():
+    p = argparse.ArgumentParser(
+        prog="dq",
+        description="sift - big-file data prep that never runs out of memory (DuckDB-powered).",
+    )
+    p.add_argument("--version", action="version", version=f"sift {__version__}")
+    sub = p.add_subparsers(dest="cmd", required=True)
+
+    sp = sub.add_parser("profile", help="schema + row count, fast (no full load)")
+    sp.add_argument("input")
+
+    sp = sub.add_parser("drop", help="remove columns")
+    _io(sp)
+    sp.add_argument("--cols", required=True, help="comma-separated")
+
+    sp = sub.add_parser("keep", help="keep only these columns")
+    _io(sp)
+    sp.add_argument("--cols", required=True, help="comma-separated")
+
+    sp = sub.add_parser("filter", help="keep rows matching a SQL condition")
+    _io(sp)
+    sp.add_argument("--where", required=True)
+
+    sp = sub.add_parser("dedup", help="drop duplicate rows (default: whole-row)")
+    _io(sp)
+    sp.add_argument("--on", default="all", help="key column(s), comma-separated")
+
+    sp = sub.add_parser("sample", help="random N rows")
+    _io(sp)
+    sp.add_argument("--n", type=int, required=True)
+
+    sp = sub.add_parser("head", help="first N rows")
+    _io(sp)
+    sp.add_argument("--n", type=int, required=True)
+
+    sp = sub.add_parser("clip", help="clip lat/lon to a region or bbox")
+    _io(sp)
+    sp.add_argument("--region", help="uae | ksa | world")
+    sp.add_argument("--bbox", help="min_lon,min_lat,max_lon,max_lat")
+
+    sp = sub.add_parser("convert", help="change format (by output extension)")
+    _io(sp)
+
+    sp = sub.add_parser("run", help="run a .dq recipe file")
+    sp.add_argument("recipe")
+
+    return p
+
+
+def main(argv=None):
+    a = build_parser().parse_args(argv)
+
+    if a.cmd == "profile":
+        profile(a.input)
+        return
+
+    if a.cmd == "run":
+        with open(a.recipe, "r", encoding="utf-8") as f:
+            spec = parse_recipe(f.read())
+        print(f"-> running recipe {a.recipe}")
+        run_spec(spec)
+        return
+
+    spec = {"input": a.input, "output": a.output}
+    if a.cmd == "drop":
+        spec["drop"] = a.cols
+    elif a.cmd == "keep":
+        spec["keep"] = a.cols
+    elif a.cmd == "filter":
+        spec["filter"] = a.where
+    elif a.cmd == "dedup":
+        spec["dedup"] = a.on
+    elif a.cmd == "sample":
+        spec["sample"] = a.n
+    elif a.cmd == "head":
+        spec["head"] = a.n
+    elif a.cmd == "clip":
+        if a.region:
+            spec["clip"] = a.region
+        if a.bbox:
+            spec["bbox"] = a.bbox
+    # convert = no transforms, just re-COPY to the new format
+
+    print(f"-> {a.cmd}")
+    run_spec(spec)

siftq-0.1.0/src/sift/engine.py

@@ -0,0 +1,45 @@
+"""DuckDB connection, auto-tuned so big files don't OOM.
+
+The whole anti-OOM story lives here:
+  - memory_limit is set to a fraction of *available* RAM (not total), so a
+    big job can't grab everything and get OS-killed while other apps run.
+  - temp_directory is set, so DuckDB spills to disk instead of dying when a
+    job genuinely needs more than RAM.
+  - preserve_insertion_order is off — a real memory saver on large writes.
+"""
+from __future__ import annotations
+
+import os
+import tempfile
+
+import duckdb
+
+
+def sql_path(p: str) -> str:
+    """Absolute, forward-slashed, single-quote-safe path for embedding in SQL."""
+    return os.path.abspath(p).replace("\\", "/").replace("'", "''")
+
+
+def _available_gb():
+    try:
+        import psutil  # optional extra
+
+        return psutil.virtual_memory().available / 1e9
+    except Exception:
+        return None
+
+
+def connect(mem_fraction: float = 0.6, threads=None, temp_dir=None):
+    con = duckdb.connect()
+    con.execute(f"SET threads={threads or os.cpu_count() or 4}")
+
+    td = temp_dir or os.path.join(tempfile.gettempdir(), "sift_spill")
+    os.makedirs(td, exist_ok=True)
+    con.execute(f"SET temp_directory='{sql_path(td)}'")
+
+    con.execute("SET preserve_insertion_order=false")
+
+    avail = _available_gb()
+    if avail:
+        con.execute(f"SET memory_limit='{max(1, int(avail * mem_fraction))}GB'")
+    return con

siftq-0.1.0/src/sift/ops.py

@@ -0,0 +1,158 @@
+"""The operations. Each maps a simple intent to a DuckDB SQL fragment, and a
+whole recipe compiles to ONE streaming query (not N passes writing N temp
+files) — which is what makes it both memory-safe and fast.
+"""
+from __future__ import annotations
+
+import os
+import time
+
+from .engine import connect, sql_path
+
+LAT_NAMES = {"lat", "latitude"}
+LON_NAMES = {"lon", "lng", "long", "longitude"}
+
+# name -> (min_lon, min_lat, max_lon, max_lat)
+BBOX = {
+    "uae": (51.0, 22.0, 56.6, 26.6),
+    "ksa": (34.0, 16.0, 56.0, 33.0),
+    "world": (-180.0, -90.0, 180.0, 90.0),
+}
+
+
+def _ident(c: str) -> str:
+    return '"' + str(c).replace('"', '""') + '"'
+
+
+def _source(path: str) -> str:
+    # DuckDB auto-detects csv / parquet / json by extension.
+    return "'" + sql_path(path) + "'"
+
+
+def _aslist(v):
+    if isinstance(v, (list, tuple)):
+        return [str(x).strip() for x in v if str(x).strip()]
+    return [s.strip() for s in str(v).split(",") if s.strip()]
+
+
+def columns(con, source: str):
+    return [r[0] for r in con.execute(f"DESCRIBE SELECT * FROM {source}").fetchall()]
+
+
+def _find(cols, names):
+    for c in cols:
+        if c.lower() in names:
+            return c
+    return None
+
+
+def _clip_sql(spec, cols):
+    box = None
+    if spec.get("clip"):
+        key = str(spec["clip"]).lower()
+        if key not in BBOX:
+            raise ValueError(
+                f"unknown region '{key}'. known: {', '.join(BBOX)} "
+                f"(or use bbox: min_lon,min_lat,max_lon,max_lat)"
+            )
+        box = BBOX[key]
+    if spec.get("bbox"):
+        v = spec["bbox"]
+        v = [float(x) for x in v.split(",")] if isinstance(v, str) else [float(x) for x in v]
+        box = tuple(v)
+    if not box:
+        return None
+    lat, lon = _find(cols, LAT_NAMES), _find(cols, LON_NAMES)
+    if not lat or not lon:
+        raise ValueError("clip needs latitude/longitude columns (lat/latitude, lon/longitude)")
+    mnlon, mnlat, mxlon, mxlat = box
+    return f"{_ident(lon)} BETWEEN {mnlon} AND {mxlon} AND {_ident(lat)} BETWEEN {mnlat} AND {mxlat}"
+
+
+def build_query(con, spec) -> str:
+    src = _source(spec["input"])
+    cols = columns(con, src)
+
+    if spec.get("keep"):
+        sel = ", ".join(_ident(c) for c in _aslist(spec["keep"]))
+    elif spec.get("drop"):
+        drop = {c.lower() for c in _aslist(spec["drop"])}
+        sel = ", ".join(_ident(c) for c in cols if c.lower() not in drop)
+        if not sel:
+            raise ValueError("drop removed every column")
+    else:
+        sel = "*"
+
+    where = []
+    if spec.get("filter"):
+        where.append(f"({spec['filter']})")
+    clip = _clip_sql(spec, cols)
+    if clip:
+        where.append(clip)
+
+    q = f"SELECT {sel} FROM {src}"
+    if where:
+        q += " WHERE " + " AND ".join(where)
+
+    if spec.get("dedup"):
+        keys = _aslist(spec["dedup"])
+        if [k.lower() for k in keys] == ["all"]:
+            q = f"SELECT DISTINCT * FROM ({q}) _q"
+        else:
+            q = f"SELECT DISTINCT ON ({', '.join(_ident(k) for k in keys)}) * FROM ({q}) _q"
+
+    if spec.get("sample"):
+        q = f"SELECT * FROM ({q}) _q USING SAMPLE {int(spec['sample'])} ROWS"
+
+    if spec.get("head"):
+        q = f"SELECT * FROM ({q}) _q LIMIT {int(spec['head'])}"
+
+    return q
+
+
+def _copy_opts(path: str) -> str:
+    ext = os.path.splitext(path)[1].lower()
+    if ext == ".tsv":
+        return "(HEADER, DELIMITER '\t')"
+    if ext in (".parquet", ".pq"):
+        return "(FORMAT PARQUET)"
+    if ext in (".json", ".ndjson"):
+        return "(FORMAT JSON)"
+    return "(HEADER, DELIMITER ',')"  # csv default
+
+
+def run_spec(spec, con=None, quiet=False) -> int:
+    own = con is None
+    con = con or connect()
+    try:
+        t0 = time.time()
+        q = build_query(con, spec)
+        out = spec["output"]
+        os.makedirs(os.path.dirname(os.path.abspath(out)), exist_ok=True)
+        dst = "'" + sql_path(out) + "'"
+        n = con.execute(f"COPY ({q}) TO {dst} {_copy_opts(out)}").fetchone()[0]
+        if not quiet:
+            print(f"  done: {n:,} rows -> {out}  ({time.time() - t0:,.1f}s)")
+        return n
+    finally:
+        if own:
+            con.close()
+
+
+def profile(path, con=None) -> int:
+    own = con is None
+    con = con or connect()
+    try:
+        src = _source(path)
+        schema = con.execute(f"DESCRIBE SELECT * FROM {src}").fetchall()
+        n = con.execute(f"SELECT count(*) FROM {src}").fetchone()[0]
+        size = os.path.getsize(path) if os.path.exists(path) else 0
+        print(f"\n  {os.path.basename(path)}")
+        print(f"  {n:,} rows  |  {len(schema)} columns  |  {size / 1e9:,.2f} GB on disk\n")
+        for row in schema:
+            print(f"    {row[0]:<22} {row[1]}")
+        print()
+        return n
+    finally:
+        if own:
+            con.close()

siftq-0.1.0/src/sift/recipe.py

@@ -0,0 +1,47 @@
+"""Tiny recipe parser — a readable .dq file, no YAML dependency.
+
+    input:  data/raw/Feb2.parquet
+    keep:   [device_id, lat, lon, carrier]
+    clip:   uae
+    filter: carrier = 'DU'
+    sample: 50000
+    output: outputs/du_clean.csv
+
+Only rule: one `key: value` per line, `[a, b, c]` for lists, `#` comments.
+"""
+from __future__ import annotations
+
+_REQUIRED = ("input", "output")
+
+
+def _strip_comment(line: str) -> str:
+    """Drop a trailing #comment, but not a # that sits inside quotes."""
+    inq = None
+    for i, ch in enumerate(line):
+        if inq:
+            if ch == inq:
+                inq = None
+        elif ch in "\"'":
+            inq = ch
+        elif ch == "#":
+            return line[:i]
+    return line
+
+
+def parse(text: str) -> dict:
+    spec: dict = {}
+    for raw in text.splitlines():
+        s = _strip_comment(raw).strip()
+        if not s or ":" not in s:
+            continue
+        key, val = s.split(":", 1)          # split on FIRST colon (keeps C:\ paths intact)
+        key, val = key.strip().lower(), val.strip()
+        if val.startswith("[") and val.endswith("]"):
+            val = [x.strip() for x in val[1:-1].split(",") if x.strip()]
+        elif (val[:1], val[-1:]) in (('"', '"'), ("'", "'")):
+            val = val[1:-1]
+        spec[key] = val
+    missing = [k for k in _REQUIRED if k not in spec]
+    if missing:
+        raise ValueError(f"recipe is missing: {', '.join(missing)}")
+    return spec

siftq-0.1.0/src/siftq.egg-info/PKG-INFO

@@ -0,0 +1,94 @@
+Metadata-Version: 2.4
+Name: siftq
+Version: 0.1.0
+Summary: Big-file data prep that never runs out of memory — no SQL required. Powered by DuckDB.
+Author: Ken
+License: MIT
+Keywords: duckdb,data,csv,parquet,etl,cli,big-data,dataframe
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: duckdb>=0.10
+Provides-Extra: mem
+Requires-Dist: psutil>=5.9; extra == "mem"
+Dynamic: license-file
+
+# sift
+
+**Big-file data prep that never runs out of memory — no SQL required.**
+
+`sift` is a tiny command-line tool for cleaning and reshaping data files
+(CSV, Parquet, JSON) that are *too big for pandas*. It's a friendly front-end
+over [DuckDB](https://duckdb.org): DuckDB does the heavy lifting (streaming,
+disk-spill, all your CPU cores) and `sift` makes it a one-liner — and
+auto-configures memory so your job doesn't crash.
+
+```bash
+pip install siftq            # pip name; the command is `dq`
+```
+
+## Why
+
+- **It doesn't OOM.** Memory is capped to a fraction of *free* RAM and DuckDB
+  spills to disk instead of dying. Point it at a 10 GB file on a laptop; it's fine.
+- **No SQL, no pandas.** Simple verbs, or a readable recipe file.
+- **One streaming pass.** A whole recipe compiles to a single query — no
+  intermediate files, so it's fast and light.
+- **Any format.** Reads/writes CSV, Parquet, JSON — auto-detected by extension.
+
+## One-liners
+
+```bash
+dq profile data.parquet                       # schema + row count, instantly
+dq keep    data.parquet --cols id,lat,lon -o small.csv
+dq drop    data.csv     --cols ip,ua       -o clean.parquet
+dq filter  data.parquet --where "carrier = 'DU'" -o du.csv
+dq dedup   data.csv     --on device_id     -o unique.parquet
+dq sample  data.parquet --n 50000          -o sample.csv
+dq clip    pings.parquet --region uae       -o uae.parquet
+dq convert data.parquet -o data.csv          # just change format
+```
+
+## Recipes
+
+Chain steps in a readable `.dq` file — they run as one pass:
+
+```yaml
+# du_clean.dq
+input:  data/raw/Feb2.parquet
+keep:   [device_id, lat, lon, carrier]
+clip:   uae                 # drop global GPS noise
+filter: carrier = 'DU'
+dedup:  device_id
+sample: 50000
+output: outputs/du_clean.csv
+```
+
+```bash
+dq run du_clean.dq
+```
+
+Real run: **54.6M rows / 2.5 GB parquet → 50k-row clean CSV in ~1 second.**
+
+## Install for max memory-safety
+
+```bash
+pip install "siftq[mem]"       # adds psutil for smarter RAM sizing
+```
+
+## Roadmap
+
+Core is intentionally small and grows release by release: `join`, `rename`,
+`split` (by column into many files), `stats`, timestamp shifting, and an
+optional geo/mobility extension. Ideas welcome.
+
+---
+
+> **Naming:** the project is **sift**; it installs from PyPI as **`siftq`**
+> (`sift` was taken) and gives you the **`dq`** command.
+
+MIT licensed.

siftq-0.1.0/src/siftq.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+LICENSE
+README.md
+pyproject.toml
+src/sift/__init__.py
+src/sift/__main__.py
+src/sift/cli.py
+src/sift/engine.py
+src/sift/ops.py
+src/sift/recipe.py
+src/siftq.egg-info/PKG-INFO
+src/siftq.egg-info/SOURCES.txt
+src/siftq.egg-info/dependency_links.txt
+src/siftq.egg-info/entry_points.txt
+src/siftq.egg-info/requires.txt
+src/siftq.egg-info/top_level.txt

siftq-0.1.0/src/siftq.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

siftq-0.1.0/src/siftq.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+dq = sift.cli:main

siftq-0.1.0/src/siftq.egg-info/requires.txt

@@ -0,0 +1,4 @@
+duckdb>=0.10
+
+[mem]
+psutil>=5.9

siftq-0.1.0/src/siftq.egg-info/top_level.txt

@@ -0,0 +1 @@
+sift