contrek 1.4.0 → 1.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8d3c8751ab1e2182443227811912d1b0ad864dde99b62aae50c51b1265314bf3
-  data.tar.gz: d3a1ad93edd90e4ebb41a4e1371cd0ed5a2c134fbc093f898a920b8e81e8c249
+  metadata.gz: 330f21ed351ac9ec5b507c77c49c0595264d174f5403d00449cd6abf25be9e98
+  data.tar.gz: 33813776ce0b886f1807d08e6e3245fb4a2276cb6da7ae0fdc571886101160ae
 SHA512:
-  metadata.gz: e423133c5e9dfe2f490c84e2c47bdd585d9771949ccf8dbd1afd21b254dea876c7bc18f5cd1e71e327d50cec3de2a7abe5bcb39029f578ab043b43c9a785b607
-  data.tar.gz: 3b343824bbe74a380e6e3f084d628c00a7b3efeac373cf65a80b50de66827d5c62b44a4e33ed248b3144336a35c7712e96681b8f754e1dbcee280680b3caf9ab
+  metadata.gz: d9ee2d8f093dad25a2c6913247a3c719fbf63d6c818ca3b451b27c8ec8984eeaa1760688e4bb5baf799afd3ea6b712c1db3940eb46fea859a6b3336815ad5e1f
+  data.tar.gz: b542aec71f43fa199e300d745838ac5ff25e3c3b34e988ea6e1ca58051ac9f93723d2caa5d0ba2a4102fd0c30daf7b1ad3ced65f4ee3d077d1fff85a824c2960

data/CHANGELOG.md

@@ -149,7 +149,13 @@ All notable changes to this project will be documented in this file.
 - Fixed issue found on treemap determination.
 
 ## [1.3.9] - 2026-08-09
-- Removed the curses development dependency along with the old TerminalTracker; actually Bitmaps::Rendering.to_window renders visually better results.
+- Removed the curses development dependency along with the old TerminalTracker; actually Contrek::Bitmaps::PngBitmap.to_window renders visually better results.
 
 ## [1.4.0] - 2026-08-13
 - Added deterministic merge mode usefull when you need a byte-for-byte reproducible outputs across different runs (by default, the merge stage is non-deterministic to maximize performance and thread utilization).
+
+## [1.4.1] - 2026-08-23
+- The number_of_threads effective value is now returned inside metadata output struct too.
+
+## [1.4.2] - 2026-09-06
+- Contrek takes care of the streaming process, providing an API that greatly simplifies the implementation of streaming raster processing.

data/Gemfile.lock

@@ -1,9 +1,10 @@
 PATH
   remote: .
   specs:
-    contrek (1.4.0)
+    contrek (1.4.2)
       chunky_png (~> 1.4)
       concurrent-ruby (~> 1.3.5)
+      ffi (~> 1.17)
       rice (= 4.5.0)
 
 GEM
@@ -14,6 +15,8 @@ GEM
     chunky_png (1.4.0)
     concurrent-ruby (1.3.5)
     diff-lcs (1.6.1)
+    ffi (1.17.4)
+    ffi (1.17.4-x86_64-linux-gnu)
     json (2.11.3)
     language_server-protocol (3.17.0.4)
     lint_roller (1.1.0)

data/PERFORMANCE.md

@@ -1,4 +1,4 @@
-# ⚡ Contrek Performance Tuning
+# Contrek Performance Tuning
 
 This document describes optional dependencies and configuration tips to get the best performance out of Contrek on large images.
 
@@ -41,7 +41,7 @@ sudo make install
 sudo ldconfig
 ```
 
-> ⚠️ The `-DZLIB_COMPAT=ON` flag is mandatory. Without it, zlib-ng uses a different ABI and CMake's `find_package(ZLIB)` won't detect it.
+> The `-DZLIB_COMPAT=ON` flag is mandatory. Without it, zlib-ng uses a different ABI and CMake's `find_package(ZLIB)` won't detect it.
 
 **macOS:**
 ```bash
@@ -136,9 +136,45 @@ Consider this depends your system. Probably is better not to saturate all cores
 
 ---
 
-## 4. Combining All Optimizations
+## 4. Transparent Huge Pages (THP) — Kernel Memory Allocation
 
-Install zlib-ng and tcmalloc, then configure:
+When processing high-resolution images, Contrek performs dense dynamic memory allocations. By default, Linux manages memory in standard 4 KiB pages, which creates heavy kernel overhead from frequent page faults and Translation Lookaside Buffer (TLB) misses under multithreaded loads.
+
+Enabling **Transparent Huge Pages (`always`)** forces the Linux kernel to allocate memory in **2 MiB blocks**, reducing page table lookup entries by a factor of 512 and dramatically speeding up memory access.
+
+### Temporary Activation
+
+To enable it immediately for your current Linux session:
+
+```bash
+echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
+```
+
+### Permanent Activation
+To ensure this setting persists across system reboots:
+
+Open /etc/default/grub in an editor:
+
+```bash
+sudo nano /etc/default/grub
+```
+Add transparent_hugepage=always to GRUB_CMDLINE_LINUX_DEFAULT:
+
+```Plaintext
+GRUB_CMDLINE_LINUX_DEFAULT="quiet splash transparent_hugepage=always"
+```
+
+Update GRUB:
+
+```bash
+sudo update-grub
+```
+
+---
+
+## 5. Combining All Optimizations
+
+Install zlib-ng and tcmalloc, enable THP, then configure:
 
 ```ruby
 # Ruby

data/README.md

@@ -8,6 +8,8 @@ The engine is based on programmable pixel matchers, allowing the caller to decid
 
 Although the tracing engine is written in C++, Contrek is also distributed as a Ruby gem exposing almost the complete native API through an idiomatic Ruby interface.
 
+A **Python** binding is available 👉 **[here](https://github.com/runout77/contrek-python)**
+
 ## About Contrek
 
 **Contrek** (**CON**tour **TREK**king) at its simplest, the library scans a bitmap and returns the contours of all regions matching a user-defined criterion. Each region is represented by one outer polygon and, when necessary, one or more inner polygons describing holes.
@@ -25,58 +27,227 @@ The image below shows a simple example. Every non-white pixel is considered part
 
 Most contour tracing libraries process an image sequentially. That approach is perfectly adequate for many workloads, but it becomes less practical once images become very large or when memory usage starts to matter.
 
-Contrek follows a different strategy.
+Contrek supports monothread extraction but also different strategies.
+
+### Mode 1: Single-threaded processing
 
-Instead of processing the whole image at once, it can split it into independent vertical stripes. Each stripe is traced separately and the resulting polygons are merged afterwards. Most of the implementation complexity lies in this merge phase, whose purpose is to reconstruct polygons crossing stripe boundaries without breaking their topology.
+<table>
+<tr>
+<td width="40%" valign="top">
+<img src="docs/images/modes/mode1.jpg" width="100%" alt="Mode 1">
+</td>
+<td width="60%" valign="top">
+The entire image is processed using a single core.
 
-The same design makes it possible to use the library in two different ways:
+**Profile:** low speed; low memory efficiency.
+<br><br>
+<center><img src="docs/images/modes/mode1_panel.jpg" width="80%" alt="Mode 1"></center>
+</td></tr></table>
 
-- **Parallel processing**, where several stripes are traced simultaneously on different CPU cores.
-- **Streaming processing**, where only a small portion of the image is kept in memory at any given time.
+Trace polygons from a PNG file (`CPPPngBitMap`).
+
+```ruby
+  png_bitmap = CPPPngBitMap.new("spec/files/images/labyrinth3.png")
+  color = Contrek::Bitmaps::RgbCppColor.new(r: 255, g: 255, b: 255, a: 255)
+  rgb_matcher = CPPRGBNotMatcher.new(color.raw)
+  polygonfinder = CPPPolygonFinder.new(
+    png_bitmap,
+    rgb_matcher,
+    nil,
+    { versus: :a,
+      compress: {
+        visvalingam: true,
+        visvalingam_tolerance: 1.5
+      }
+    }
+  )
+  result = polygonfinder.process_info
+  puts result.metadata[:number_of_threads] # => 0
+```
 
-The tracing algorithm itself is identical in both cases; only the execution strategy changes.
+`versus` accepts `:a`,`:o` for "anticlockwise"/"clockwise".
 
-## Parallel execution
 
-When multiple CPU cores are available, independent stripes can be processed concurrently. The actual speedup depends on the image content and on how much work is required during the merge stage, but large datasets generally benefit from additional cores.
+### Mode 2: Parallel processing
 
-This approach allows Contrek to make effective use of modern multicore processors without changing the resulting geometry.
+<table>
+<tr>
+<td width="40%" valign="top">
+<img src="docs/images/modes/mode2.jpg" width="100%" alt="Mode 1">
+</td>
+<td width="60%" valign="top">
+The entire image is loaded first, then split into tiles and processed across multiple CPU cores. Partial results are progressively and dynamically merged: there is no predefined merge order, and adjacent pairs are processed as soon as they become available.
+This mode prioritizes performance, using parallelism both for tile processing and for merging partial results.
 
-## Streaming large datasets
+**Profile:** maximum speed, with processing time decreasing as more cores become available; low memory efficiency.
+<br><br>
+<center><img src="docs/images/modes/mode2_panel.jpg" width="80%" alt="Mode 1"></center>
+</td></tr></table>
 
-Loading an entire gigapixel image into memory is often unnecessary.
+Use 8 threads and 8 tiles
 
-Contrek can instead process the image incrementally by reading one stripe at a time. Only the current working buffer needs to be allocated, making memory consumption predictable regardless of the final image height.
+```ruby
+  png_bitmap = CPPPngBitMap.new("spec/files/images/sample_10240x10240.png")
+  color = Contrek::Bitmaps::RgbCppColor.new(r: 255, g: 255, b: 255, a: 255)
+  rgb_matcher = CPPRGBNotMatcher.new(color.raw)
+  polygonfinder = Contrek::Cpp::CPPConcurrentFinder.new(
+    number_of_threads: 8,
+    bitmap: png_bitmap,
+    matcher: rgb_matcher,
+    options: {number_of_tiles: 8, versus: :o, compress: {uniq: true}}
+  )
+  result = polygonfinder.process_info
+  puts result.metadata[:benchmarks].inspect
+  # => {"compress"=>5.08, "init"=>230.79, "inner"=>6.54, "outer"=>84.11, "total"=>235.87}
+  puts result.metadata[:number_of_threads] # => 8
+```
+
+### Mode 3: Input streaming
 
-This execution mode is particularly useful when processing images that are too large to fit comfortably into RAM.
+<table>
+<tr>
+<td width="40%" valign="top">
+<img src="docs/images/modes/mode3.jpg" width="100%" alt="Mode 1">
+</td>
+<td width="60%" valign="top">
+The image does not need to be loaded entirely into memory. Instead, it can be read progressively using a fixed-size buffer. For example, with a PNG source, this can be done using libspng's progressive decoding.
+Adjacent tiles share an overlapping scanline to preserve geometry continuity across tile boundaries. Once all tiles have been added, the merge is performed (optionally using multiple threads) to reconstruct the complete geometry.
+This mode provides a trade-off between performance and memory usage: the entire raster does not need to be kept in RAM, while the vector state required to build the final result is retained.
+
+**Profile:** medium speed; medium-high memory efficiency.
+<br><br>
+<center><img src="docs/images/modes/mode3_panel.jpg" width="80%" alt="Mode 1"></center>
+</td></tr></table>
+
+Trace polygons from two in-memory pattern strings (`CPPBitMap`, useful for synthetic tiles or tests).
+Up is 6 rows height, down is 5 rows. Total after merging: 10 rows, because one row is the shared scanline
 
-## Merging polygons
+```ruby
+  up =    " 00000000000000               " \
+          " 00000000000000               " \
+          " 00          00               " \
+          " 00          00               " \
+          " 00          00               " \
+          " 00          00               "
+
+  down =  " 00          00               " \
+          " 00          00               " \
+          " 00          00               " \
+          " 00000000000000               " \
+          " 00000000000000               "
+  matcher = CPPValueNotMatcher.new(" ")
+  result_up = CPPPolygonFinder.new(CPPBitMap.new(up, 30),
+    matcher,
+    nil,
+    {versus: :a, bounds: true}).process_info
+  result_down = CPPPolygonFinder.new(CPPBitMap.new(down, 30),
+    matcher,
+    nil,
+    {versus: :a, bounds: true}).process_info
+
+  step_finder = Contrek::Cpp::CPPConcurrentVerticalMerger.new(options: {compress: {linear: true}})
+  step_finder.add_tile(result_up)
+  step_finder.add_tile(result_down)
+  result = step_finder.process_info
+
+  expect(result.metadata[:groups]).to eq(1)
+  expect(result.metadata[:width]).to eq(30)
+  expect(result.metadata[:height]).to eq(10)
+```
+See an other example of **[input streaming graphical result](docs/images/stripes/merging_polygons.md)**.
+
+The same operation using Contrek's built-in streaming API and progressive PNG decoding via libspng:
 
-Whenever a polygon crosses the boundary between two adjacent stripes, the library reconstructs it during the merge phase.
+```ruby
+  # PNG image to stream from
+  source = CPPPngSource.new("./spec/files/images/labyrinth2.png")
+  # streamer with a 20-row window
+  streamer = CPPRasterStreamer.new(source, stripe_height: 20)
+  white = Contrek::Bitmaps::RgbCppColor.new(r: 255, g: 255, b: 255, a: 255)
+  matcher = CPPRGBNotMatcher.new(white.raw)
+  # reusable bitmap where libspng decodes rows directly
+  buffer_bitmap = CPPRawBitMap.new(source.width, streamer.stripe_height)
+  finder = Contrek::Cpp::CPPConcurrentVerticalMerger.new(options: {compress: {uniq: true, linear: true}})
+  
+  # streaming loop, 20 rows each
+  streamer.each(buffer_bitmap) do |bitmap, buffer_rows, buffer_size|
+    tile = CPPPolygonFinder.new(
+      bitmap,
+      matcher,
+      nil,
+      {processing_height: buffer_rows, versus: :o, bounds: true}
+    ).process_info
+    finder.add_tile(tile)
+  end
+  result = finder.process_info
+  puts result.metadata[:width] # => 130
+  puts result.metadata[:height] # => 130
+  puts result.points[0][:outer].first.inspect # => {:x=>10, :y=>4} # first point
+  puts result.points[0][:outer].size # => 1171 points
+```
 
-This reconstruction preserves polygon connectivity across stripe boundaries, so the final output is equivalent to tracing the entire image in a single pass.
+### Mode 4: End-to-end streaming
 
 <table>
-  <tr>
-    <td width="50%" style="padding: 0; background-color: white;">
-      <img src="docs/images/stripes/whole_0.png" width="100%"><br>
-      <img src="docs/images/stripes/whole_256.png" width="100%"><br>
-      <img src="docs/images/stripes/whole_512.png" width="100%"><br>
-      <img src="docs/images/stripes/whole_768.png" width="100%">
-    </td>
-    <td width="50%" align="center" style="vertical-align: middle; background-color: white;">
-      <strong>Full Topological Reconstruction</strong><br><br>
-      <img src="docs/images/stripes/whole.png" width="90%">
-    </td>
-  </tr>
-  <tr>
-    <td colspan="2" align="center" style="background-color: white;">
-      <em><b>Left:</b> Image split into 4 independent memory buffers (stripes).</em><br>
-      <em><b>Right:</b> Contrek ensures <b>perfect topological continuity</b> during merging.</em><br>
-      🔴 <b>Red:</b> Outer contours &nbsp;&nbsp; | &nbsp;&nbsp; 🟢 <b>Green:</b> Inner zones
-    </td>
-  </tr>
-</table>
+<tr>
+<td width="40%" valign="top">
+<img src="docs/images/modes/mode4.jpg" width="100%" alt="Mode 1">
+</td>
+<td width="60%" valign="top">
+This mode extends the incremental processing used in Mode 3 by streaming the output as well.
+The image is read and processed one tile at a time. Each new tile is immediately merged with the current state. As processing moves forward, whenever a geometry is complete and can no longer be affected by subsequent tiles, it is finalized and written directly to the SVG file.
+This limits both the amount of raster data kept in memory and the accumulation of generated vector geometries. It is particularly well suited to very large datasets or cases where the vector output itself can become significant in size.
+
+**Profile:** low speed; maximum memory efficiency.
+<br><br>
+<center><img src="docs/images/modes/mode4_panel.jpg" width="80%" alt="Mode 1"></center>
+</td></tr></table>
+
+Trace polygons from 2 in-memory pattern strings.
+
+```ruby
+  stripe1 = "     000       000" \
+            "    00 00     00 0" \
+            "   00   00   00  0" \
+            "    00 00   00   0" \
+            "     000   00    0" \
+            "          00     0"
+
+  stripe2 = "          00     0" \
+            " 000     00      0" \
+            "00 00   00       0" \
+            " 000   00        0" \
+            "      00         0" \
+            "     0000000000000"
+  # streaming to svg file pattern
+  stripes = [stripe1, stripe2]
+  shared_stream = Contrek::Cpp::CPPTempfile.new("output.svg")
+  step_finder = Contrek::Cpp::CPPSvgConcurrentStreamingMerger.new(
+    options:  {bounds: true, compress: {uniq: true, linear: true, douglas_peucker: true}},
+    stream_to: shared_stream,
+    total_width: 18,
+    total_height: 11
+  )
+  matcher = CPPValueNotMatcher.new(" ")
+  stripes.each do |stripe|
+    stripe_result = CPPPolygonFinder.new(CPPBitMap.new(stripe, 18),
+      matcher,
+      nil,
+      {versus: :a, bounds: true}).process_info
+    last = stripes.last == stripe
+    step_finder.add_tile(stripe_result, last)
+  end
+  result = step_finder.process_info
+  expect(result.metadata[:groups]).to eq(3)
+  expect(result.metadata[:width]).to eq(18)
+  expect(result.metadata[:height]).to eq(11)
+  shared_stream.rewind
+  puts shared_stream.read.inspect
+  # => "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"18\" height=\"11\">
+  #     <polygon points=\"3,3 5,5 8,5 10,3 8,0 5,0\" class=\"out\"/>
+  #     <polygon points=\"7,1 8,3 6,4 5,2\" class=\"in\"/>
+  #     <polygon points=\"5,11 18,11 18,0 15,0\" class=\"out\"/> ...
+```
 
 ## Benchmarking
 
@@ -166,7 +337,8 @@ Metadata includes timing information (expressed in milliseconds), the number of
   },
   :groups=>2,
   :named_sequence=>"",
-  :treemap=>[]
+  :treemap=>[],
+  :number_of_threads=>0
 }
 ```
 
@@ -600,7 +772,8 @@ Typical information includes
   benchmarks:{...},
   groups:1,
   treemap:[],
-  options:{...}
+  options:{...},
+  number_of_threads: 4
 }
 ```
 
@@ -717,27 +890,6 @@ Instead, it repeatedly combines smaller topologically consistent pieces into a l
 This strategy makes it possible to process images whose full size would otherwise exceed the available memory.
 
 
-## Merge order
-
-The merge order is intentionally left to the thread scheduler.
-
-For three stripes, both of the following execution orders are valid:
-
-```
-(B1 + B2) + B3
-```
-
-or
-
-```
-B1 + (B2 + B3)
-```
-
-Although the intermediate merge sequence changes, the reconstructed polygons remain geometrically equivalent.
-
-The exact ordering of coordinates may differ because polygon simplification is applied after merging.
-
-
 ## Polygon reconstruction
 
 Internally, polygons crossing a stripe boundary are temporarily divided into smaller pieces.

data/contrek.gemspec

@@ -32,6 +32,7 @@ Gem::Specification.new do |s|
 
   s.add_dependency "chunky_png", "~> 1.4"
   s.add_dependency "concurrent-ruby", "~> 1.3.5"
+  s.add_dependency "ffi", "~> 1.17"
   s.add_dependency "rice", "= 4.5.0"
 
   s.required_ruby_version = ">= 3.0.0"

data/ext/cpp_polygon_finder/PolygonFinder/src/ContrekApi.h

@@ -21,6 +21,11 @@
 
 namespace Contrek {
 
+enum class Versus {
+  A,  // Anticlockwise
+  O   // Clockwise
+};
+
 enum class MatchMode {
   NOT_COLOR,    // Tracks border of what is not target color
   EXACT_COLOR   // Tracks border of what exactly matchs target color
@@ -31,15 +36,22 @@ enum class Connectivity {
   OMNIDIRECTIONAL = 8    // 8 directions
 };
 
+
 struct Config {
-  int threads = 4;
-  int tiles = 2;
+  int threads = 0;
+  int tiles = 1;
+  Versus versus = Versus::A;
   bool compress_unique = false;
   bool compress_linear = false;
   bool compress_raster = false;
   bool compress_douglas_peucker = false;
   bool compress_visvalingam = false;
+  float compress_visvalingam_tolerance = 0.0f;
   bool treemap = false;
+  bool bounds = false;
+  bool named_sequences = false;
+  bool unsafe_mode = false;
+  bool deterministic = false;
   int32_t target_color = -1;
   MatchMode mode = MatchMode::NOT_COLOR;
   Connectivity connectivity_mode = Connectivity::ORTHOGONAL;
@@ -81,19 +93,33 @@ inline TraceContext trace(const std::string& image_path, const Config& cfg = Con
     ctx.matcher = std::make_unique<RGBMatcher>(color_to_match);
   }
 
+  std::string versus_str = (cfg.versus == Versus::O) ? "o" : "a";
   ctx.internal_args = {
-    {"versus", Identifier{"a"}},
+    {"versus", Identifier{versus_str}},
+    {"number_of_tiles", static_cast<int64_t>(cfg.tiles)},
+    {"treemap", cfg.treemap},
+    {"bounds", cfg.bounds},
+    {"named_sequences", cfg.named_sequences},
+    {"unsafe_mode", cfg.unsafe_mode},
+    {"deterministic", cfg.deterministic}
   };
-  Options compression_opts;
-
-  if (cfg.compress_unique) compression_opts["uniq"] = true;
-  if (cfg.compress_linear) compression_opts["linear"] = true;
-  if (cfg.compress_visvalingam) compression_opts["visvalingam"] = true;
-  if (cfg.compress_unique || cfg.compress_linear || cfg.compress_visvalingam) ctx.internal_args["compression"] = compression_opts;
-  ctx.internal_args["number_of_tiles"] = cfg.tiles;
   if (cfg.connectivity_mode == Connectivity::OMNIDIRECTIONAL) {
-    ctx.internal_args["connectivity"] = 8;
+    ctx.internal_args["connectivity"] = static_cast<int64_t>(8);
   }
+  Options compress_opts;
+  if (cfg.compress_unique) compress_opts["uniq"] = true;
+  if (cfg.compress_linear) compress_opts["linear"] = true;
+  if (cfg.compress_raster) compress_opts["raster"] = true;
+  if (cfg.compress_douglas_peucker) compress_opts["douglas_peucker"] = true;
+  if (cfg.compress_visvalingam) compress_opts["visvalingam"] = true;
+  if (cfg.compress_visvalingam_tolerance > 0.0f) {
+    compress_opts["visvalingam_tolerance"] = static_cast<double>(cfg.compress_visvalingam_tolerance);
+  }
+  if (cfg.compress_unique || cfg.compress_linear || cfg.compress_raster ||
+      cfg.compress_douglas_peucker || cfg.compress_visvalingam) {
+    ctx.internal_args["compress"] = compress_opts;
+  }
+
   ctx.finder = std::make_unique<Finder>(cfg.threads, ctx.bitmap.get(), ctx.matcher.get(), ctx.internal_args);
   ctx.result = std::unique_ptr<ProcessResult>(ctx.finder->process_info());