dftracer-utils 0.0.14.post2.dev0__tar.gz → 0.0.14.post8.dev0__tar.gz

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/CHANGELOG.md

@@ -9,10 +9,145 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
 
 ### Changed
 
+- Push `unique()` and `drop_duplicates()` into the trace scan, so a distinct row query dedups each worker's rows before they become frame rows.
+  The result keeps the first occurrence in unit order, a key column the scan cannot key exactly falls back to the host cursor, and the host `unique` cursor now keys rows by a 64-bit hash with no per-row string allocation.
+  On the 17 GB genesis sweep, `filter(gtype == "counter").select("run", "metric").unique()` takes 33.5 s instead of 592 s, uses 74 cores instead of 10 and peaks at 11.0 GB instead of 13.8 GB.
+- Find the gzip members of a trace with 16 reads in flight instead of one, and keep the `sketch` strings of genesis records out of the index build.
+  The member scan of the 20.7 GB genesis sweep took about 45 s on one core, and its sketches filled the string table to about 36 GB.
+  The index of the regrouped 17 GB sweep now builds in 313 to 338 s at a 5 to 8 GB peak instead of 385 s at 42 GB ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Write the `run` and `func` records of `dftracer_genesis_gen_dist` after every `counter` record, in gzip members of their own, so a query on `func` or `run` records reads a few chunks instead of most of the file.
+  On the 20.7 GB sweep, a `gtype == "func" and name == "main"` query takes 1.3 s instead of 44 s, and the amg scaling read takes 1 s instead of 13 s ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Keep the bloom index that a first `dftracer_view` query builds for each file within that file's share of `--memory-budget`.
+  Past it, finished chunks go to sorted runs next to the index and are ingested with the rest of the file, as `dftracer_index` already did, and the index is the same ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Write the bloom index of a file as sorted runs that are built at the same time and ingested, also when nothing spilled, instead of one write batch into the database or one last run on one thread.
+  The first `dftracer_view` count over one genesis part of 160 million counter records takes 172 s instead of 190 s, and over 12 parts 222 s instead of 242 s.
+  `--no-spill` keeps the write batch ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Keep the min and max of each auto-indexed args key of an index chunk as numbers while the chunk is built, and format a value as text only when it moves a bound, instead of formatting every value and parsing both bounds again for each comparison.
+  The stored bounds are the same text.
+  The first `dftracer_view` count over one genesis part of 160 million counter records takes 397 s instead of 760 s on one node, at the same 12.4 GB peak ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Build the index of a gzip trace on all the cores it is given, in bounded memory, and give the same index.
+  The members are inflated in runs of about 8 MB, which up to 16 workers take in order into reused buffers, with at most two runs past the workers ahead of the parse.
+  The plaintext is parsed ahead of the index folds in 4 MB pieces, and every fold steps on its own task in file order.
+  The bloom index of different chunks is built at the same time, each chunk from its records in file order.
+  A first `dftracer_view` query over several files divides the cores between the files it indexes at once.
+  Before, one file was inflated and its index built on about one core, and the workers ahead buffered up to 160 MB each without bound.
+  Over the 12 parts (20.7 GB) of a genesis sweep on one 48-core node, the first `dftracer_view --group-by gtype --agg count` takes 222 s at a 36.9 GB peak instead of 863 s with 52 GB resident, and one part of 160 million counter records takes 172 s instead of 397 s ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Make `dftracer_genesis_gen_dist` sort and walk a run group in parallel.
+  The calls of a group too large for its sort buffer are read back from the spill file, mapped and sorted by 16 tasks at once.
+  The processes of a group are walked in up to 64 parts at once, process p in part p modulo the parts, and the partial records are merged in part order (exact sketch merges, counts, minima and maxima).
+  A group of 10.3 billion calls from 64 processes takes 54 min on one 96-core node with a 56.6 GB peak, where the single-threaded walk alone ran at 130 K calls/s, about 22 h.
+  Two small groups take 11 s instead of 1 min 44 s.
+  The split depends on the group alone, so the output is the same for any thread count.
+  The `sum` and `avg` of a group with several processes per set differ from the single-threaded walk in the last digits (at most 4e-12 relative on amg), and every other value is the same ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Make `dftracer_genesis_gen_dist` count the decoded gzip member that each open trace reader holds against the reader quarter of a run's share, sized from the file's last gzip trailer, instead of 8 MB per reader.
+  A run of many files with large members now opens as many at once as its share holds, so the tool stays near `--memory-budget`.
+  Four runs of 40 files with 20 MB members peak at 518 MB under a 512 MB budget ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Decode a gzip member into a buffer sized from the file's last gzip trailer, and grow both decode buffers without zero-filling them or copying bytes that are decoded again.
+  Each line inside a member is now yielded in place instead of copied.
+  This read path serves `dftracer_genesis_gen_dist` and every unindexed trace read.
+  One 932 MB member reads in 0.92 s instead of 1.29 s with a third less peak memory and page faults, and a file of 16 MB members peaks at 113 MB instead of 147 MB ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
 - Keep on PyPI only the newest `<tag>.postN.devX` prerelease of the previous release when a release is published, and delete the older ones, so the project stays under its PyPI storage limit.
 
+### Added
+
+- Take the pandas names `col(...).isin`, `startswith`, `endswith`, `between`, `isnull`, `notnull`, `cumsum`, `cummax`, `cummin`, `cumprod`, `ffill`, `bfill` and `astype` in column expressions.
+  `col(...).str` takes every pandas string method that the engine has, such as `startswith`, `lower`, `strip`, `len` and `split` ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Round to any number of places with `Series.round(decimals)` and `DataFrame.round(decimals)`, with halves to even as numpy and pandas do.
+  A negative `decimals` rounds to the left of the decimal point, and an integer column stays integer ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Pivot on several keys and with any group-by reduction, such as `median`, in `DataFrame.pivot_table(values, index=[...], columns, aggfunc)` ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Take the keys of `TraceViewer.group_by` as one list as well as separate arguments ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Add `col(...).duplicated(keep=...)`, which marks the copies after the first (or before the last) as pandas does ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Add `str.isascii`, `str.encode` and `str.decode` for UTF-8 on a Series and on `col(...)`.
+  `decode` raises on bytes that are not UTF-8, or gives null with `errors="null"` ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Add `Series.str.extractall`, which gives one row per regex match with the columns `row`, `match` and one per capture group ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Add `col(...).sketch_quantile(q)`, a group-by aggregate that merges the stored DDSketches of each group, such as `dur.sketch`, and reads the quantile `q`.
+  It runs eager and lazy, and it gives what duql `quantile(merge(dur.sketch), q)` gives ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Add `str.normalize` (NFC, NFKC, NFD or NFKD, with utf8proc), `str.translate` (a table of codepoints to strings, integers or `None`) and `str.wrap` (`textwrap` of CPython 3.13.10, 3.14.1 and later, with the width in codepoints) on a Series and on `col(...)`.
+  A row that is not valid UTF-8 gives null ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Add the record schema setting `index.never_index`, a list of paths that the index build puts in the path catalog without evidence or interned values.
+  Queries still read these paths.
+  The built-in `genesis` schema lists `dur.sketch` and `v.sketch`, so the index build of a genesis file no longer interns the unique sketch of every function and counter record.
+  A schema refuses a listed path that has a role or `always_index` or that a stored row set reads ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Add `--shard i/N` to `dftracer_genesis_gen_dist`, so that a process takes part `i` of `N` of the run groups.
+  The groups are split largest first so the parts hold about the same trace bytes, and the parts concatenate into one file.
+  A build with `DFTRACER_UTILS_ENABLE_MPI=ON` splits the groups between its ranks by default, and the ranks copy their parts into `-o` at their offsets at the same time ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Print the progress of `dftracer_genesis_gen_dist` to stderr.
+  At the start it prints the run groups, their trace bytes, the shares and the member size.
+  Then at most every 10 s it prints one line with the elapsed time, the groups done, how many groups wait for memory, read, sort, walk or write, the trace bytes read, the calls sorted and walked with their rates, the peak RSS against the budget and an estimate of the time left.
+  At most every 60 s, each group that has run for a minute or more gets a line of its own with its step, its progress in that step and its time there ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Print a `DataFrame` or a `Series` the way pandas does, with the row index, right-aligned values and a common number of decimals for each float column.
+  Past 60 rows it prints the first 5 and the last 5 rows with a `...` row and a `[N rows x M columns]` footer, and it reads only the rows that it shows ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Show a `DataFrame` as an HTML table in Jupyter, with the same rows and cells as its printed form ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+
 ### Fixed
 
+- Fix the first query or build on a new index failing with `lock hold by current process` when a read-write open of the index ran while an older read-write handle of it was still closing.
+  It happened on an index that a rebuild had reset while a reader still held it, such as the first `TraceViewer` read of 29 files with relative paths ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Match `_` and `%` literally in a `TraceViewer.filter` of `col(...).str.starts_with`, `ends_with` and `contains`.
+  The filter escaped them without naming the escape character, so `starts_with("MPI_")` matched no record ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Keep the columns of an empty right side in a lazy join under a memory budget.
+  A right plan that sent no rows failed with `join: no right column named <key>` ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Fix duql `quantile(merge(x), q)` keeping the accuracy of the first sketch it read.
+  A sketch of another accuracy than the default is now always skipped, so the result no longer depends on the order of the rows ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Reject a stored DDSketch with a bad accuracy or an out-of-range bin offset instead of reading it, which could loop forever in a merge ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Fix `DataFrame.duplicated` marking every copy.
+  It now marks only the copies after the first, as pandas does ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Keep the rows with a null key in `col(...).<agg>().over(key)`, eager and lazy.
+  The result came back shorter than its frame, `cum_count().over(key)` failed, a lazy transform put values on the wrong rows after a null key and a lazy aggregate of a computed value was refused ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Return an empty frame from `GroupBy.median` and `GroupBy.quantile` of an empty frame instead of failing in the join of the quantile plan ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Leave the `run` records out of the first aggregate over a genesis file that has no index yet.
+  The first-touch pass counted them, so `--group-by gtype` showed a `run` group that later queries did not ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Fix a correlated duql sub-query on a trace with no index yet returning an empty frame when the outer filter could not be answered while the first-touch index was built, such as `where ph == "X" and dur > (from data | where name == ^.name | agg { m = mean(dur) })`.
+  The side query reserved the file in the index root while the scan took that root for a built index and read zero bytes.
+  The first build of a root now finishes before a scan reads the metadata of its files, and a root that does not hold the file is built ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Build the index that `dftracer_view` and `dftracer_index` need before their query or build on all their threads, not on the one thread that drives the command.
+  The first `dftracer_view --query` on a genesis part of 197 MB takes 5 s instead of 31 s on a 48-core corona node, and a single file of 20.7 GB indexes in 584 s with 14 cores busy instead of 1 ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Print the chunks that a `dftracer_view` aggregation or `--query` read and skipped, which showed as `scanned=0 skipped=0`.
+  A query for one run of a 20.7 GB genesis file reads 34 of 73,340 chunks ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Sort the result of `DataFrame.groupby` by the key columns, nulls last, as pandas does by default.
+  `size`, `sum`, `agg` and the other aggregates came back in hash order.
+  `sort=False` keeps that order, and `group_by` is not changed ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Index every gzip member of a trace that has zero bytes between two members, as the padded striped writer left before this release.
+  The inflate worker stopped at the zeros, so the rest of its part of the file was missing from the index and from every count served from it.
+  `dftracer_view` counted 9,547 of the 9,615 runs and 702,657,406 of the 710,692,194 counters of the 12 parts of an 8-node genesis sweep, and it now counts all of them.
+  The zeros now belong to the compressed size of the member before them ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Fail an index build when a visitor fails, for example when a line does not parse, instead of returning the part of the index built so far as if it were complete.
+  The error now ends the inflate workers and reaches the caller ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Fix `dftracer_view` dying with SIGILL (illegal instruction) on the first aggregate of a file when built with GCC 12.1 on x86_64.
+  GCC compiled the call that builds the file's index on the way, awaited inside an `if` condition, into a trap.
+  The awaited result is now named before it is tested, here and on the export path.
+  GCC 12.3 on aarch64 and Clang were not affected ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Fill every stripe of the padded striped writer with whole gzip members.
+  When the space left after the data of a stripe was a little more than one full padding member, the last 1 to 24 bytes stayed zero, too few for another member, and `gzip`, `zcat` and zlib stopped reading the file there.
+  Files written to striped Lustre, such as the output of `dftracer_genesis_gen_dist`, now read to their end with any gzip reader.
+  Four of the 12 parts of an 8-node genesis sweep had such gaps, so `zcat` saw 146 of its 9,615 runs ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Make `dftracer_genesis_gen_dist` let a run group borrow memory of the budget that no group holds, 256 MB at a time, before it runs the group again with a larger share, and give the walk of a group the readers' and calls' parts of its share once its calls are sorted.
+  On a full 8-node sweep each shard ran 74 to 154 groups again, and the last restart of the largest group threw away 25 minutes of reading and sorting, so 3 of 8 shards missed a 2 h limit ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Make `dftracer_genesis_gen_dist` run a group again with four times its share, up to the whole budget, when its counter statistics or path records do not fit, instead of skipping it.
+  These grow with call paths times counters, not with trace bytes, so a 10 MB trace could need more than its 777 MB share, and a full run on 8 nodes skipped about 1,150 runs this way.
+  The groups now share one memory pool, largest group first, and each group starts with its part of the pool by trace bytes, so one large group reads many files at once instead of one at a time.
+  The tool now exits with 0 when it skips runs, and with 1 only when the output cannot be written ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Keep `dftracer_genesis_gen_dist` under `--memory-budget` on runs of thousands of trace files.
+  Every file of every group opened at once to read the size of its decoded gzip member, so most opens failed with too many open files, and each such file was counted as an 8 MB reader while it decoded up to 120 MB.
+  On 8 nodes this took the tool to 120 GB under an 80 GB budget, and one run of 6 groups to 20 GB under an 8 GB budget.
+  At most 256 files are now opened for their size at once, and a file whose size cannot be read takes all reader slots of its group.
+  The same run peaks at 4.1 GB and reads 10 times faster ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Keep `dftracer_genesis_gen_dist` under `--memory-budget` while it writes.
+  Each run held all its output lines in one string and each worker held a whole 32 MB member, and neither was counted.
+  The lines of a run are now made 1 MiB at a time from its records as they are written, and the group keeps its share until then.
+  The member size is lowered until one member per worker fits the part of the budget for the writer.
+  Under a 512 MB budget, 96 runs of 4000 call paths with 24 counters peak at 302 MB instead of 903 MB, in 29 s instead of 28 s, with the same lines ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Fix a hang at the end of every unordered `GzipLineWriter` that writes to a striped Lustre directory (stripe of 1 MB or more), such as `dftracer_genesis_gen_dist -o` on Lustre.
+  The padded striped writer started its packer in the scope of the writer.
+  The writer joins that scope before it closes the channel of the packer, and the packer ends only when that channel closes, so the join never returned and the output lost its last stripe.
+  The packer now runs in a scope of its own ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Never spill to a RAM-backed directory.
+  With `DFTRACER_UTILS_SPILL_DIR` unset and no writable node-local disk, the spill went to the system temp directory even when it was a tmpfs, so on a diskless compute node every spilled byte stayed in memory and `dftracer_genesis_gen_dist` was killed by the kernel at about 94 GB of 251 GB.
+  The fallback is now the temp directory or `~/.cache/dftracer-utils/spill`, whichever is writable and not RAM-backed (a network file system qualifies).
+  With neither, the first spill fails with an error that names the variable ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Make `dftracer_genesis_gen_dist` read runs that keep their time slices directly in the run directory (`nodes_<N>/ppn_<M>/<stem>-<k>_chunk<c>.pfw.gz`, without `compacted/`), which it skipped without a message.
+  It also reads `raw/<set>/*.pfw.gz` for a set whose `compacted/<set>.pfw.gz` is missing instead of skipping the set ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
+- Build the static zlib-ng archive as position-independent code, so a shared library links it when zlib-ng made no shared target (`BUILD_SHARED_LIBS` off) ([#112](https://github.com/llnl-asr/dftracer-utils/pull/112)).
 - Grow an elastic runtime for a burst of tasks sent before its parked worker starts to run. The burst stayed on one worker for up to one `elastic_keepalive` (250 ms by default) before the pool grew.
 
 ## [0.0.14] - 2026-10-10

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dftracer-utils
-Version: 0.0.14.post2.dev0
+Version: 0.0.14.post8.dev0
 Summary: Python binding for dftracer-utils
 Author-Email: Ray Andrew Sinurat <raydreww@gmail.com>, "Hariharan Devarajan (Hari)" <hariharandev1@llnl.gov>
 License: MIT

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/cmake/modules/CompilerWarnings.cmake

@@ -119,14 +119,18 @@ function(target_set_warnings TARGET_NAME)
         target_compile_options(${TARGET_NAME} PRIVATE -Wno-restrict)
       endif()
 
-      # GCC 12+ has false-positive -Wnull-dereference, -Warray-bounds,
-      # -Wstringop-overflow, and -Wstringop-overread in libstdc++ (streambuf,
-      # exception_ptr, char_traits, vector copy/move). Disable for affected
-      # versions.
+      # GCC reports -Wnull-dereference from inlined libstdc++ code with no
+      # pointer of ours involved: GCC 11 in the destructors and moves of
+      # duql::compile_program, even for an empty Program, and GCC 12+ in
+      # streambuf, exception_ptr, char_traits and vector copy/move. Clang keeps
+      # the check.
+      target_compile_options(${TARGET_NAME} PRIVATE -Wno-null-dereference)
+      # GCC 12+ also has false-positive -Warray-bounds, -Wstringop-overflow and
+      # -Wstringop-overread in the same libstdc++ code.
       if(CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL "12")
         target_compile_options(
-          ${TARGET_NAME} PRIVATE -Wno-null-dereference -Wno-array-bounds
-                                 -Wno-stringop-overflow -Wno-stringop-overread)
+          ${TARGET_NAME} PRIVATE -Wno-array-bounds -Wno-stringop-overflow
+                                 -Wno-stringop-overread)
       endif()
     endif()
 

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/cmake/modules/Dependencies.cmake

@@ -976,10 +976,13 @@ function(_try_zlib_ng OUT_VAR)
     else()
       set(_zng_static zlib-ng-static)
     endif()
+    # PIC: link_zlib falls back to this archive for a shared library when
+    # zlib-ng made no shared target (BUILD_SHARED_LIBS off).
     set_target_properties(
       ${_zng_static}
-      PROPERTIES OUTPUT_NAME dftracer_zlib ARCHIVE_OUTPUT_DIRECTORY
-                                           ${CMAKE_BINARY_DIR}/lib)
+      PROPERTIES OUTPUT_NAME dftracer_zlib
+                 ARCHIVE_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib
+                 POSITION_INDEPENDENT_CODE ON)
     target_include_directories(
       ${_zng_static} PUBLIC $<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>)
     if(NOT TARGET dftracer_zlib_static)
@@ -1904,6 +1907,43 @@ function(need_pcre2)
   endif()
 endfunction()
 
+# ==============================================================================
+# utf8proc (MIT); the Unicode tables behind Series.str.normalize
+# ==============================================================================
+
+function(need_utf8proc)
+  if(NOT TARGET utf8proc_static)
+    if(NOT utf8proc_ADDED)
+      # The pinned release always builds from source, never a system copy.
+      set(CPM_USE_LOCAL_PACKAGES OFF)
+      cpmaddpackage(
+        NAME
+        utf8proc
+        GITHUB_REPOSITORY
+        JuliaStrings/utf8proc
+        VERSION
+        2.9.0
+        GIT_TAG
+        v2.9.0
+        DOWNLOAD_ONLY
+        YES)
+    endif()
+    if(NOT utf8proc_SOURCE_DIR)
+      message(FATAL_ERROR "need_utf8proc: utf8proc sources were not fetched")
+    endif()
+    # utf8proc.c includes utf8proc_data.c itself. The static archive links into
+    # our shared libraries, so it must be PIC.
+    add_library(utf8proc_static STATIC ${utf8proc_SOURCE_DIR}/utf8proc.c)
+    target_include_directories(utf8proc_static SYSTEM
+                               PUBLIC ${utf8proc_SOURCE_DIR})
+    target_compile_definitions(utf8proc_static PUBLIC UTF8PROC_STATIC)
+    target_compile_options(utf8proc_static PRIVATE -w)
+    set_target_properties(utf8proc_static PROPERTIES POSITION_INDEPENDENT_CODE
+                                                     ON)
+    dftracer_utils_ok("Built utf8proc ${utf8proc_VERSION} with CPM")
+  endif()
+endfunction()
+
 # ==============================================================================
 # Vectorscan (static, external build); Ragel and Boost headers are build-only
 # ==============================================================================

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/cmake/modules/PrecompiledHeader.cmake

@@ -103,6 +103,7 @@ function(detect_common_headers)
         # third-party headers that might not be available everywhere
         if(NOT HEADER MATCHES "/"
            AND NOT HEADER MATCHES "^Python\\.h$"
+           AND NOT HEADER MATCHES "^mpi\\.h$"
            AND NOT HEADER MATCHES "^hs\\.h$"
            AND NOT HEADER MATCHES "^zstd\\.h$"
            AND NOT HEADER MATCHES "^argparse/")

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/core/common/base64.h

@@ -14,6 +14,10 @@ std::string base64_encode(const void* data, std::size_t len);
 /// Inverse of base64_encode; nullopt when `sv` is not padded base64.
 std::optional<std::string> base64_decode(std::string_view sv);
 
+/// base64_decode into `out`, reusing its capacity; false (and `out` partial)
+/// when `sv` is not padded base64.
+bool base64_decode_into(std::string_view sv, std::string& out);
+
 }  // namespace dftracer::utils
 
 #endif  // DFTRACER_UTILS_CORE_COMMON_BASE64_H

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/core/common/external_sort.h

@@ -8,6 +8,7 @@
 #include <cstddef>
 #include <cstdint>
 #include <memory>
+#include <mutex>
 #include <queue>
 #include <type_traits>
 #include <utility>
@@ -44,6 +45,31 @@ class ExternalSorter {
         return {};
     }
 
+    /// Sorts `run` and writes it as a run of its own, then clears it, for a
+    /// caller that fills buffers on several threads. Calls of add_run() may
+    /// overlap each other, not add() or drain().
+    Result<void> add_run(std::vector<Rec>& run) {
+        std::sort(run.begin(), run.end(), less_);
+        SpillFile* file = nullptr;
+        {
+            std::lock_guard<std::mutex> lock(mu_);
+            if (!file_) {
+                auto created = SpillFile::create();
+                if (!created) return unexpected(created.error());
+                file_ = std::move(*created);
+            }
+            file = file_.get();
+        }
+        auto at = file->append(run.data(), run.size() * sizeof(Rec));
+        if (!at) return unexpected(at.error());
+        {
+            std::lock_guard<std::mutex> lock(mu_);
+            runs_.push_back({*at, run.size()});
+        }
+        run.clear();
+        return {};
+    }
+
     std::size_t runs() const noexcept { return runs_.size(); }
 
     /// Calls `f(const Rec&)` for every record in order. One-shot.
@@ -145,6 +171,7 @@ class ExternalSorter {
     std::vector<Rec> buf_;
     std::vector<Run> runs_;
     std::unique_ptr<SpillFile> file_;
+    std::mutex mu_;
 };
 
 }  // namespace dftracer::utils

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/core/common/memory_pool.h

@@ -18,7 +18,9 @@ class MemoryPool {
     MemoryPool(const MemoryPool&) = delete;
     MemoryPool& operator=(const MemoryPool&) = delete;
 
-    std::uint64_t capacity() const noexcept { return capacity_; }
+    std::uint64_t capacity() const noexcept {
+        return capacity_.load(std::memory_order_relaxed);
+    }
     std::uint64_t used() const noexcept {
         return used_.load(std::memory_order_relaxed);
     }
@@ -26,7 +28,8 @@ class MemoryPool {
     bool try_reserve(std::uint64_t n) noexcept {
         std::uint64_t cur = used_.load(std::memory_order_relaxed);
         do {
-            if (n > capacity_ - cur) return false;
+            const std::uint64_t cap = capacity_.load(std::memory_order_relaxed);
+            if (cur > cap || n > cap - cur) return false;
         } while (!used_.compare_exchange_weak(cur, cur + n,
                                               std::memory_order_relaxed));
         return true;
@@ -36,8 +39,14 @@ class MemoryPool {
         used_.fetch_sub(n, std::memory_order_relaxed);
     }
 
+    /// Raises the capacity by `n`, for memory handed over from elsewhere; a
+    /// pool never shrinks.
+    void add_capacity(std::uint64_t n) noexcept {
+        capacity_.fetch_add(n, std::memory_order_relaxed);
+    }
+
    private:
-    const std::uint64_t capacity_;
+    std::atomic<std::uint64_t> capacity_;
     std::atomic<std::uint64_t> used_{0};
 };
 
@@ -73,6 +82,17 @@ class Reservation {
         bytes_ = 0;
     }
 
+    /// Moves up to `n` of `o`'s bytes into this reservation without touching
+    /// the pool; both must hold bytes of the same pool. Returns the bytes
+    /// moved.
+    std::uint64_t take_from(Reservation& o, std::uint64_t n) noexcept {
+        const std::uint64_t moved = n < o.bytes_ ? n : o.bytes_;
+        if (!pool_) pool_ = o.pool_;
+        o.bytes_ -= moved;
+        bytes_ += moved;
+        return moved;
+    }
+
    private:
     MemoryPool* pool_ = nullptr;
     std::uint64_t bytes_ = 0;

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/core/common/spill_dir.h

@@ -12,9 +12,11 @@ namespace dftracer::utils {
 /// The directory every spill file goes to: DFTRACER_UTILS_SPILL_DIR when set;
 /// else the node-local disk mount with the most free space that the user can
 /// write (local_spill_root(), never RAM-backed or network); else the system
-/// temp directory. Created if missing. A set variable that cannot be used is an
-/// IO error naming the directory and the variable, with no fallback to another
-/// directory.
+/// temp directory, then ~/.cache/dftracer-utils/spill, whichever is first
+/// writable and not RAM-backed (a network file system qualifies). Created if
+/// missing. A set variable that cannot be used is an IO error naming the
+/// directory and the variable, with no fallback to another directory; when no
+/// directory off RAM exists, an IO error names the variable.
 Result<std::string> spill_dir();
 
 /// A directory under spill_dir() that one owner fills with its spill files,

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/core/coro/async_semaphore.h

@@ -64,6 +64,16 @@ class CoroSemaphore {
     /// Acquire `n` permits (co_await). `n` above capacity is clamped.
     AcquireOperation acquire(std::uint64_t n) noexcept { return {*this, n}; }
 
+    /// Takes `n` permits only if they are free now, ahead of any waiters;
+    /// false, without waiting, when they are not. For a holder that needs a
+    /// little more to finish rather than a new unit of work.
+    bool try_acquire(std::uint64_t n) noexcept {
+        std::lock_guard<std::mutex> lock(mtx_);
+        if (available_ < n) return false;
+        available_ -= n;
+        return true;
+    }
+
     /// Return `n` permits and wake any waiters that now fit, in FIFO order.
     void release(std::uint64_t n) {
         Waiter wake_local[8];

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/dataframe/abi.h

@@ -623,6 +623,46 @@ DFTU_EXPORT dftu_series* dftu_series_str_partition(const dftu_series* v,
                                                    const char* sep,
                                                    int32_t sep_len,
                                                    int32_t from_right);
+
+/** The Unicode normalization forms of dftu_series_str_normalize (Python's
+ * unicodedata.normalize). */
+typedef enum {
+    DFTU_NORM_NFC = 0,
+    DFTU_NORM_NFKC = 1,
+    DFTU_NORM_NFD = 2,
+    DFTU_NORM_NFKD = 3
+} dftu_norm_form;
+/** Each row of a String column in the normalization `form` (a
+ * dftu_norm_form). A row that is not valid UTF-8 is null, as a null row stays.
+ * NULL for another input type or a bad `form`. */
+DFTU_EXPORT dftu_series* dftu_series_str_normalize(const dftu_series* v,
+                                                   int32_t form);
+/** Each codepoint of every row of a String column replaced as the table says
+ * (Python's str.translate): `from` is a String column whose rows are each one
+ * codepoint, all distinct and none null; `to` is a String column of the same
+ * length, where a null row deletes that codepoint and any other row (possibly
+ * empty, possibly several codepoints) replaces it. A codepoint not in the
+ * table is kept. A row that is not valid UTF-8 is null, as a null row stays.
+ * NULL for another input type or an invalid table. */
+DFTU_EXPORT dftu_series* dftu_series_str_translate(const dftu_series* v,
+                                                   const dftu_series* from,
+                                                   const dftu_series* to);
+/** The flag bits of dftu_series_str_wrap, the keyword arguments of Python's
+ * textwrap.TextWrapper. */
+typedef enum {
+    DFTU_WRAP_EXPAND_TABS = 1,
+    DFTU_WRAP_REPLACE_WHITESPACE = 2,
+    DFTU_WRAP_DROP_WHITESPACE = 4,
+    DFTU_WRAP_BREAK_LONG_WORDS = 8,
+    DFTU_WRAP_BREAK_ON_HYPHENS = 16
+} dftu_wrap_flag;
+/** Each row of a String column as "\n".join(textwrap.wrap(row, width)) of
+ * CPython 3.13.10, 3.14.1 and later, with the options in `flags` (an OR of
+ * dftu_wrap_flag); `width` counts codepoints.
+ * A row that is not valid UTF-8 is null, as a null row stays. NULL for another
+ * input type or a `width` below 1. */
+DFTU_EXPORT dftu_series* dftu_series_str_wrap(const dftu_series* v,
+                                              int64_t width, int32_t flags);
 /** The calendar parts of dftu_series_dt_part (the pandas .dt accessor):
  * DAY_OF_WEEK is Monday = 0, DAY_OF_YEAR and ISO_WEEK are 1-based, IS_LEAP_YEAR
  * is 0 / 1, EPOCH_DAYS the day number since 1970-01-01. */
@@ -936,7 +976,12 @@ typedef enum {
 
 /** A string method with text, integer or second-column operands, evaluated
  * by the Series function the eager Series.str method of the same name calls
- * (dftu_expr_str_fn). 0-8 are the dftu_series_str_is classes in order. */
+ * (dftu_expr_str_fn). 0-8 are the dftu_series_str_is classes in order.
+ * NORMALIZE takes the form in `i0`, WRAP the width in `i0` and the flag bits
+ * in `i1`. TRANSLATE takes its table in the two texts: `text` holds the keys,
+ * one codepoint each, and `text2` holds one entry per key, a codepoint L and
+ * then the value's codepoints (L is 1 + their count, or 0 to delete the key).
+ */
 typedef enum {
     DFTU_STR_FN_ISALNUM = 0,
     DFTU_STR_FN_ISALPHA = 1,
@@ -966,7 +1011,10 @@ typedef enum {
     DFTU_STR_FN_JOIN = 25,
     DFTU_STR_FN_GET = 26,
     DFTU_STR_FN_CAT = 27,
-    DFTU_STR_FN_REGEX_REPLACE = 28
+    DFTU_STR_FN_REGEX_REPLACE = 28,
+    DFTU_STR_FN_NORMALIZE = 29,
+    DFTU_STR_FN_TRANSLATE = 30,
+    DFTU_STR_FN_WRAP = 31
 } dftu_str_fn;
 
 /** Rolling-window reduction (`op` is a dftu_rolling_op:

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/dataframe/agg.h

@@ -92,7 +92,13 @@ enum class AggOp {
     RegrR2 = 34,
     /// Product of the present values of `value_col` (Float64; 1 for a group
     /// with none, as pandas).
-    Prod = 35
+    Prod = 35,
+    /// `param` quantile (clamped to the merged min and max) of the merge of the
+    /// base64 serialized DDSketch cells of a String `value_col`, Float64. A
+    /// null or undecodable cell, an empty sketch and a sketch of another
+    /// accuracy than the group's first are skipped; a group with no valid
+    /// sketch is null.
+    SketchQuantile = 36
 };
 
 /// Ops taking a second input through `by_col`: the extremum ArgMax/ArgMin
@@ -116,7 +122,7 @@ inline bool agg_uses_raw_value(AggOp op) {
            op == AggOp::SetUnion || op == AggOp::Distinct ||
            op == AggOp::ListSorted || op == AggOp::TopK ||
            op == AggOp::BottomK || op == AggOp::ApproxTopK ||
-           op == AggOp::Sample;
+           op == AggOp::Sample || op == AggOp::SketchQuantile;
 }
 
 /// The DataType agg_finalize actually writes for `op` over a value column of

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/dataframe/agg_op_codes.h

@@ -48,7 +48,8 @@ typedef enum {
     DFTU_AGG_REGR_SLOPE = 32,
     DFTU_AGG_REGR_INTERCEPT = 33,
     DFTU_AGG_REGR_R2 = 34,
-    DFTU_AGG_PROD = 35
+    DFTU_AGG_PROD = 35,
+    DFTU_AGG_SKETCH_QUANTILE = 36
 } dftu_agg_op;
 
 #ifdef __cplusplus

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/dataframe/agg_ops.def

@@ -53,3 +53,4 @@ DFTU_AGG_OP(RegrSlope, DFTU_AGG_REGR_SLOPE, "regr_slope")
 DFTU_AGG_OP(RegrIntercept, DFTU_AGG_REGR_INTERCEPT, "regr_intercept")
 DFTU_AGG_OP(RegrR2, DFTU_AGG_REGR_R2, "regr_r2")
 DFTU_AGG_OP(Prod, DFTU_AGG_PROD, "prod")
+DFTU_AGG_OP(SketchQuantile, DFTU_AGG_SKETCH_QUANTILE, "sketch_quantile")

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/dataframe/dataframe.h

@@ -68,7 +68,9 @@ enum class Agg {
     RegrSlope,   ///< least-squares slope of `column` on `by`, Float64
     RegrIntercept,  ///< least-squares intercept of `column` on `by`, Float64
     RegrR2,  ///< coefficient of determination of `column` on `by`, Float64
-    Prod     ///< product of `column`'s present values, Float64
+    Prod,    ///< product of `column`'s present values, Float64
+    SketchQuantile  ///< `param` quantile of the merged base64 DDSketch cells
+                    ///< of a String `column`, Float64
 };
 
 /// Canonical lowercase name of `agg` (the string the C ABI accepts).
@@ -91,7 +93,8 @@ struct GroupAgg {
     Agg op = Agg::Count;
     std::string column;
     std::string out;
-    double param = 0.0;  ///< Pct: quantile q in [0, 1]; occupancy: occ_cell_us;
+    double param = 0.0;  ///< Pct, SketchQuantile: quantile q in [0, 1];
+                         ///< occupancy: occ_cell_us;
                          ///< Distinct/TopK/BottomK/ApproxTopK/Sample: k;
                          ///< SetUnion: nonzero asks for the typed list
     std::string by{};    ///< the second input column for the ops

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/dataframe/exported_series_ops.def

@@ -243,4 +243,10 @@ DFTU_SERIES_OP(dftu.series.compare_series, dftu_series_compare_series, SERIES, S
 DFTU_SERIES_OP(dftu.series.str_regex_replace, dftu_series_str_regex_replace, SERIES, SERIES, STR, STR)
 DFTU_SERIES_OP(dftu.series.dt_format, dftu_series_dt_format, SERIES, SERIES, STR, I32)
 
+/* Unicode normalization by form code, per-codepoint translation by a
+ * (from, to) pair of String columns, and textwrap by width and flag bits. */
+DFTU_SERIES_OP(dftu.series.str_normalize, dftu_series_str_normalize, SERIES, SERIES, I32, NONE)
+DFTU_SERIES_OP(dftu.series.str_translate, dftu_series_str_translate, SERIES, SERIES, SERIES, SERIES)
+DFTU_SERIES_OP(dftu.series.str_wrap, dftu_series_str_wrap, SERIES, SERIES, I64, I32)
+
 #undef DFTU_SERIES_OP

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/dataframe/lazyframe.h

@@ -280,6 +280,14 @@ class Source {
         const AggregateSpec&) const {
         return std::nullopt;
     }
+    /// Distinct rows by `subset` (empty = every column), keeping the first
+    /// occurrence in source order. The output keeps every column and its
+    /// schema. A source that accepts it must produce the first-occurrence
+    /// order itself; the planner removes the UniqueOp.
+    virtual std::optional<SourceApplication> apply_distinct(
+        const std::vector<std::string>&) const {
+        return std::nullopt;
+    }
     virtual std::optional<SourceApplication> apply_sort(const SortSpec&) const {
         return std::nullopt;
     }

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/dataframe/series.h

@@ -606,6 +606,13 @@ class Series {
     Series str_cat(const Series& other) const;
     Series str_findall(std::string_view pattern) const;
     Series str_partition(std::string_view sep, bool from_right = false) const;
+    /// Unicode normalization (dftu_norm_form), the codepoint table of
+    /// `from` -> `to` (see dftu_series_str_translate) and textwrap by
+    /// `width` codepoints with dftu_wrap_flag bits; a row that is not valid
+    /// UTF-8 is null.
+    Series str_normalize(std::int32_t form) const;
+    Series str_translate(const Series& from, const Series& to) const;
+    Series str_wrap(std::int64_t width, std::int32_t flags) const;
     /// A calendar part (dftu_dt_part) as Int64, `unit` (dftu_time_unit) for
     /// an Int64 input; each instant rounded to a bucket of `every` units by
     /// dftu_dt_round_mode.

{dftracer_utils-0.0.14.post2.dev0 → dftracer_utils-0.0.14.post8.dev0}/include/dftracer/utils/dataframe/sketch.h

@@ -26,6 +26,7 @@
 
 #include <algorithm>
 #include <array>
+#include <cmath>
 #include <cstddef>
 #include <cstdint>
 #include <limits>
@@ -120,6 +121,22 @@ class BasicDDSketch {
 
 using DDSketch = BasicDDSketch<128>;
 
+/// The sketch that stored sketches merge into: wide enough that a sketch
+/// written with any bin count merges without a collapse.
+using MergeSketch = BasicDDSketch<2048>;
+
+/// Merges `src` into `dst` only when `src` has the default relative accuracy.
+/// Sketches of another accuracy do not merge without loss, so they are
+/// skipped; a fixed accuracy keeps the result independent of merge order.
+template <int BINS>
+void merge_default_accuracy(BasicDDSketch<BINS>& dst,
+                            const BasicDDSketch<BINS>& src) {
+    static const double DEFAULT_LOG_GAMMA = BasicDDSketch<BINS>{}.log_gamma();
+    if (src.empty() || std::abs(src.log_gamma() - DEFAULT_LOG_GAMMA) > 1e-12)
+        return;
+    dst.merge(src);
+}
+
 /// Exact-zero sentinel produced by `sketch_bucket_keys` and consumed by
 /// `DDSketch::add_key`; a zero value has no logarithmic bucket.
 constexpr std::int32_t SKETCH_ZERO_KEY =