PyPatchMatch 2.2.0b1__tar.gz → 2.2.0b2__tar.gz

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/CLAUDE.md

@@ -62,6 +62,9 @@ Other entry points:
 - `uv run python scripts/render_readme_example.py` runs `py_example.py` and renders the
   picture at the top of the README, `examples/images/readme_example.jpg`; rerun it and
   commit the picture after changes that affect the fills.
+- `uv run --group gui gui/app.py` starts the Gradio GUI (paint holes, compare the fill).
+  No CI job installs gradio, so after API changes run its test with
+  `uv run --group gui pytest gui --no-cov` and try the GUI by hand.
 - CMake options: the `option()` lines of `CMakeLists.txt`. `PATCHMATCH_SANITIZE` turns
   off fast math, which lets the compiler drop checks.
 

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: PyPatchMatch
-Version: 2.2.0b1
+Version: 2.2.0b2
 Summary: This library implements the PatchMatch based inpainting algorithm.
 Keywords: inpainting,patchmatch,image-processing,opencv
 Author-Email: The InvokeAI Project <lincoln.stein@gmail.com>, Jiayuan Mao <maojiayuan@gmail.com>, Kyle Schouviller <kyle0654@hotmail.com>, Matthias Wild <mauwii@outlook.de>, Younesse ANDAM <younesse.andam@gmail.com>
@@ -41,9 +41,12 @@ Description-Content-Type: text/markdown
 This library implements the PatchMatch based inpainting algorithm. It provides both C++
 and Python interfaces. This implementation is heavily based on the implementation by
 Younesse ANDAM: [younesse-cv/PatchMatch](https://github.com/younesse-cv/PatchMatch),
-with some bug fixes, and updates.
+with some bug fixes, and updates. This repository continues
+[vacancy/PyPatchMatch](https://github.com/vacancy/PyPatchMatch) by Jiayuan Mao, through
+the forks [lkwq007/PyPatchMatch](https://github.com/lkwq007/PyPatchMatch) and
+[invoke-ai/PyPatchMatch](https://github.com/invoke-ai/PyPatchMatch).
 
-![Birdhouses removed from a photo by patchmatch.inpaint](https://raw.githubusercontent.com/mauwii/PyPatchMatch/v2.2.0b1/examples/images/readme_example.jpg)
+![Birdhouses removed from a photo by patchmatch.inpaint](https://raw.githubusercontent.com/mauwii/PyPatchMatch/v2.2.0b2/examples/images/readme_example.jpg)
 
 From left to right: the photo, the input of
 [examples/py_example.py](https://github.com/mauwii/PyPatchMatch/blob/main/examples/py_example.py)
@@ -100,7 +103,9 @@ The previous import path `from patchmatch import patch_match` keeps working.
 pixels in each direction. Larger patches follow larger structures but are much slower;
 the default of 15 compares 31x31 patches, the examples use 3. Below 5, the coarse levels
 that lay out the fill are also tried with a radius of 5, which is kept where it fits
-the surroundings better, so that repeating patterns continue across larger holes.
+the surroundings better, so that repeating patterns continue across larger holes. Above
+7, coarse levels narrower than six patches use the largest radius that fits, but at
+least 7: larger patches found no sources there and left the fill flat.
 
 `patchmatch.set_random_seed(seed)` sets the seed of the randomized search, modulo
 `2**32`, and `patchmatch.set_verbose(True)` prints the progress of the native code to
@@ -138,8 +143,14 @@ uv run pre-commit install      # enable ruff and the other hooks on commit
 uv run pytest                  # run the test suite
 uv run pytest -m "not e2e"     # only the fast unit tests
 uv run python scripts/evaluate_inpainting.py  # measure the fill quality
+uv run --group gui gui/app.py  # draw masks in the browser, http://127.0.0.1:7860
 ```
 
+The GUI tries new images quickly, without writing a script for each one, e.g. to pick
+new examples or to verify edge cases: load an image, paint the holes (every painted
+pixel is one) and compare the original with the fill. Its test runs with
+`uv run --group gui pytest gui --no-cov`.
+
 The evaluation cuts holes into the images of `examples/images`, compares the fills with
 the original content and writes `examples/images/evaluation.html`, which shows them side
 by side. `--baseline` evaluates another build of the library as well, e.g. of the main

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/README.md

@@ -11,7 +11,10 @@
 This library implements the PatchMatch based inpainting algorithm. It provides both C++
 and Python interfaces. This implementation is heavily based on the implementation by
 Younesse ANDAM: [younesse-cv/PatchMatch](https://github.com/younesse-cv/PatchMatch),
-with some bug fixes, and updates.
+with some bug fixes, and updates. This repository continues
+[vacancy/PyPatchMatch](https://github.com/vacancy/PyPatchMatch) by Jiayuan Mao, through
+the forks [lkwq007/PyPatchMatch](https://github.com/lkwq007/PyPatchMatch) and
+[invoke-ai/PyPatchMatch](https://github.com/invoke-ai/PyPatchMatch).
 
 ![Birdhouses removed from a photo by patchmatch.inpaint](examples/images/readme_example.jpg)
 
@@ -70,7 +73,9 @@ The previous import path `from patchmatch import patch_match` keeps working.
 pixels in each direction. Larger patches follow larger structures but are much slower;
 the default of 15 compares 31x31 patches, the examples use 3. Below 5, the coarse levels
 that lay out the fill are also tried with a radius of 5, which is kept where it fits
-the surroundings better, so that repeating patterns continue across larger holes.
+the surroundings better, so that repeating patterns continue across larger holes. Above
+7, coarse levels narrower than six patches use the largest radius that fits, but at
+least 7: larger patches found no sources there and left the fill flat.
 
 `patchmatch.set_random_seed(seed)` sets the seed of the randomized search, modulo
 `2**32`, and `patchmatch.set_verbose(True)` prints the progress of the native code to
@@ -108,8 +113,14 @@ uv run pre-commit install      # enable ruff and the other hooks on commit
 uv run pytest                  # run the test suite
 uv run pytest -m "not e2e"     # only the fast unit tests
 uv run python scripts/evaluate_inpainting.py  # measure the fill quality
+uv run --group gui gui/app.py  # draw masks in the browser, http://127.0.0.1:7860
 ```
 
+The GUI tries new images quickly, without writing a script for each one, e.g. to pick
+new examples or to verify edge cases: load an image, paint the holes (every painted
+pixel is one) and compare the original with the fill. Its test runs with
+`uv run --group gui pytest gui --no-cov`.
+
 The evaluation cuts holes into the images of `examples/images`, compares the fills with
 the original content and writes `examples/images/evaluation.html`, which shows them side
 by side. `--baseline` evaluates another build of the library as well, e.g. of the main

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/patchmatch/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '2.2.0b1'
-__version_tuple__ = version_tuple = (2, 2, 0, 'b1')
+__version__ = version = '2.2.0b2'
+__version_tuple__ = version_tuple = (2, 2, 0, 'b2')
 
-__commit_id__ = commit_id = 'gb66a4a1d8'
+__commit_id__ = commit_id = 'ge960c214c'

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/patchmatch/csrc/inpaint.cpp

@@ -92,10 +92,159 @@ namespace
         return false;
     }
 
+    // The gradient magnitude at a known pixel, summed over the channels, or -1 if it or a neighbor is unknown.
+    double gradient_magnitude(const MaskedImage &image, int y, int x)
+    {
+        const auto size = image.size();
+        const int up = std::max(y - 1, 0);
+        const int down = std::min(y + 1, size.height - 1);
+        const int left = std::max(x - 1, 0);
+        const int right = std::min(x + 1, size.width - 1);
+        if (!is_known(image, y, x) || !is_known(image, up, x) || !is_known(image, down, x) ||
+            !is_known(image, y, left) || !is_known(image, y, right))
+            return -1;
+        int magnitude = 0;
+        for (int c = 0; c < 3; ++c)
+            magnitude += std::abs(image.get_image(down, x)[c] - image.get_image(up, x)[c]) +
+                         std::abs(image.get_image(y, right)[c] - image.get_image(y, left)[c]);
+        return magnitude / 2.0;
+    }
+
+    // The mean gradient magnitude around each known pixel, -1 elsewhere or without one. Gradients
+    // across unknown pixels are left out, so the colors under the holes do not matter.
+    cv::Mat texture_energy(const MaskedImage &image)
+    {
+        constexpr int kRadius = 2;
+        const auto size = image.size();
+        // above and left of each pixel: the sum of the magnitudes and their number
+        cv::Mat sums(size.height + 1, size.width + 1, CV_64FC2, cv::Scalar::all(0));
+        for (int y = 0; y < size.height; ++y)
+        {
+            for (int x = 0; x < size.width; ++x)
+            {
+                const double magnitude = gradient_magnitude(image, y, x);
+                const cv::Vec2d measured = magnitude >= 0 ? cv::Vec2d(magnitude, 1) : cv::Vec2d();
+                sums.at<cv::Vec2d>(y + 1, x + 1) =
+                    measured + sums.at<cv::Vec2d>(y, x + 1) + sums.at<cv::Vec2d>(y + 1, x) - sums.at<cv::Vec2d>(y, x);
+            }
+        }
+
+        cv::Mat energy(size, CV_64F, cv::Scalar(-1));
+        for (int y = 0; y < size.height; ++y)
+        {
+            const int top = std::max(y - kRadius, 0);
+            const int bottom = std::min(y + kRadius + 1, size.height);
+            for (int x = 0; x < size.width; ++x)
+            {
+                const int left = std::max(x - kRadius, 0);
+                const int right = std::min(x + kRadius + 1, size.width);
+                const cv::Vec2d window = sums.at<cv::Vec2d>(bottom, right) - sums.at<cv::Vec2d>(top, right) -
+                                         sums.at<cv::Vec2d>(bottom, left) + sums.at<cv::Vec2d>(top, left);
+                if (is_known(image, y, x) && window[1] > 0)
+                    energy.at<double>(y, x) = window[0] / window[1];
+            }
+        }
+        return energy;
+    }
+
+    // The mean of the values >= 0 in the 2x2 block from (y, x), -1 if there is none.
+    double block_mean(const cv::Mat &values, int y, int x)
+    {
+        double sum = 0;
+        int count = 0;
+        for (int yy = y; yy < std::min(y + 2, values.rows); ++yy)
+        {
+            for (int xx = x; xx < std::min(x + 2, values.cols); ++xx)
+            {
+                if (values.at<double>(yy, xx) >= 0)
+                {
+                    sum += values.at<double>(yy, xx);
+                    ++count;
+                }
+            }
+        }
+        return count > 0 ? sum / count : -1;
+    }
+
+    // The value of coarse, which has half the resolution, at the center of pixel (y, x), bilinear.
+    double interpolate(const cv::Mat &coarse, int y, int x)
+    {
+        const double cy = std::clamp((y + 0.5) / 2 - 0.5, 0.0, coarse.rows - 1.0);
+        const double cx = std::clamp((x + 0.5) / 2 - 0.5, 0.0, coarse.cols - 1.0);
+        const auto y0 = static_cast<int>(cy);
+        const auto x0 = static_cast<int>(cx);
+        const int y1 = std::min(y0 + 1, coarse.rows - 1);
+        const int x1 = std::min(x0 + 1, coarse.cols - 1);
+        const double fy = cy - y0;
+        const double fx = cx - x0;
+        return (1 - fy) * ((1 - fx) * coarse.at<double>(y0, x0) + fx * coarse.at<double>(y0, x1)) +
+               fy * ((1 - fx) * coarse.at<double>(y1, x0) + fx * coarse.at<double>(y1, x1));
+    }
+
+    // Not std::all_of: libstdc++ subtracts the iterators, which divides by zero for an empty cv::Mat.
+    bool has_unknown(const cv::Mat &values)
+    {
+        for (int y = 0; y < values.rows; ++y)
+            for (int x = 0; x < values.cols; ++x)
+                if (values.at<double>(y, x) < 0)
+                    return true;
+        return false;
+    }
+
+    // Replaces the values below 0 from a pyramid of the others (pull-push), which carries the
+    // values around a hole smoothly into it.
+    void pull_push(cv::Mat &values)
+    {
+        if (!has_unknown(values))
+            return;
+        if (values.total() == 1)
+        {
+            values.at<double>(0, 0) = 0; // nothing is known
+            return;
+        }
+        cv::Mat coarse((values.rows + 1) / 2, (values.cols + 1) / 2, CV_64F);
+        for (int y = 0; y < coarse.rows; ++y)
+            for (int x = 0; x < coarse.cols; ++x)
+                coarse.at<double>(y, x) = block_mean(values, 2 * y, 2 * x);
+        pull_push(coarse);
+        for (int y = 0; y < values.rows; ++y)
+        {
+            for (int x = 0; x < values.cols; ++x)
+            {
+                if (values.at<double>(y, x) < 0)
+                    values.at<double>(y, x) = interpolate(coarse, y, x);
+            }
+        }
+    }
+
+    // The texture of the next level: the means of the 2x2 blocks.
+    cv::Mat halve(const cv::Mat &values, cv::Size size)
+    {
+        cv::Mat half(size, CV_64F);
+        for (int y = 0; y < size.height; ++y)
+            for (int x = 0; x < size.width; ++x)
+                half.at<double>(y, x) = block_mean(values, 2 * y, 2 * x);
+        return half;
+    }
+
+    cv::Mat to_bytes(const cv::Mat &values)
+    {
+        cv::Mat bytes(values.size(), CV_8U);
+        for (int y = 0; y < values.rows; ++y)
+            for (int x = 0; x < values.cols; ++x)
+                bytes.at<unsigned char>(y, x) = cv::saturate_cast<unsigned char>(values.at<double>(y, x));
+        return bytes;
+    }
+
     // A patch smaller than a structure cannot tell where it continues: on the coarsest level
     // with a hole, holes wider than the patches were filled from several shifted copies.
     constexpr int kCoarsePatchSize = 5;
 
+    // A patch wider than a sixth of a level found no sources there inside the image and outside
+    // the holes, and the fill stayed flat. Patches up to 7 keep their size: smaller ones lost the joints of bricks.
+    constexpr int kLevelSidesPerPatch = 6;
+    constexpr int kMinFittingPatchSize = 7;
+
     int nnf_iterations(int level)
     {
         return std::min(7, 1 + level);
@@ -191,6 +340,33 @@ namespace
         target_ptr[3] += weight;
     }
 
+    // A split vote: the weighted low frequencies and their weight, the detail of the best vote and its weight,
+    // and the weighted texture of the source pixels.
+    constexpr int kSplitChannels = 9;
+
+    // The mean color of the known pixels around each pixel, the part of the split votes that is averaged.
+    cv::Mat low_pass(const MaskedImage &image)
+    {
+        const auto size = image.size();
+        const auto known = [&image](int y, int x) { return is_known(image, y, x); };
+        const auto color = [&image](int y, int x) {
+            const auto *pixel = image.get_image(y, x);
+            return cv::Vec3d(pixel[0], pixel[1], pixel[2]);
+        };
+        cv::Mat low(size, CV_8UC3);
+        for (int y = 0; y < size.height; ++y)
+        {
+            for (int x = 0; x < size.width; ++x)
+            {
+                const cv::Vec3d mean = window_mean(size, y, x, 1, known, color);
+                auto *pixel = low.ptr<unsigned char>(y, x);
+                for (int c = 0; c < 3; ++c)
+                    pixel[c] = cv::saturate_cast<unsigned char>(mean[c]);
+            }
+        }
+        return low;
+    }
+
     // Replaces the vote of the pixel if the new one weighs more. It is stored like the
     // sums of _weighted_copy, so the maximization step divides it by its weight.
     inline void _best_copy(const MaskedImage &source, int ys, int xs, cv::Mat &target, int yt, int xt, double weight)
@@ -262,7 +438,8 @@ namespace
             bool upscaled, bool best_only, const HoleCounter *holes)
             : m_nnf(&nnf), m_vote(&vote), m_source(&source), m_holes(holes), m_source_size(nnf.source_size()),
               m_target_size(nnf.target_size()), m_source2target(source2target), m_upscaled(upscaled),
-              m_best_only(best_only)
+              m_best_only(best_only), m_split(vote.channels() == kSplitChannels),
+              m_low(m_split ? low_pass(source) : cv::Mat())
         {
         }
 
@@ -290,7 +467,7 @@ namespace
                         continue;
                     // Near ties go to the patch centered closest, not the first cast, which made blocks.
                     const double offset2 = static_cast<double>(di) * di + static_cast<double>(dj) * dj;
-                    cast_pair(ys, xs, yt, xt, m_best_only ? weight * (1 - 1e-5 * offset2) : weight);
+                    cast_pair(ys, xs, yt, xt, weight, weight * (1 - 1e-5 * offset2));
                 }
             }
         }
@@ -305,7 +482,7 @@ namespace
                 scale * (std::int64_t{y} + patch_size + 1) - 1, scale * (std::int64_t{x} + patch_size + 1) - 1);
         }
 
-        void cast_pair(int ys, int xs, int yt, int xt, double weight) const
+        void cast_pair(int ys, int xs, int yt, int xt, double weight, double best_weight) const
         {
             if (!m_source2target)
             {
@@ -314,22 +491,45 @@ namespace
             }
             if (!m_upscaled)
             {
-                copy(ys, xs, yt, xt, weight);
+                copy(ys, xs, yt, xt, weight, best_weight);
                 return;
             }
             for (int uy = 0; uy < 2; ++uy)
                 for (int ux = 0; ux < 2; ++ux)
-                    copy(2 * ys + uy, 2 * xs + ux, 2 * yt + uy, 2 * xt + ux, weight);
+                    copy(2 * ys + uy, 2 * xs + ux, 2 * yt + uy, 2 * xt + ux, weight, best_weight);
         }
 
-        void copy(int ys, int xs, int yt, int xt, double weight) const
+        void copy(int ys, int xs, int yt, int xt, double weight, double best_weight) const
         {
             if (m_best_only)
-                _best_copy(*m_source, ys, xs, *m_vote, yt, xt, weight);
+                _best_copy(*m_source, ys, xs, *m_vote, yt, xt, best_weight);
+            else if (m_split)
+                split_copy(ys, xs, yt, xt, weight, best_weight);
             else
                 _weighted_copy(*m_source, ys, xs, *m_vote, yt, xt, weight);
         }
 
+        // Adds the low frequencies and the texture of the source pixel to the sums, and keeps the rest if the
+        // vote weighs most.
+        void split_copy(int ys, int xs, int yt, int xt, double weight, double best_weight) const
+        {
+            if (!is_known(*m_source, ys, xs))
+                return;
+
+            const auto *color = m_source->get_image(ys, xs);
+            const auto *low = m_low.ptr<unsigned char>(ys, xs);
+            auto *vote = m_vote->ptr<double>(yt, xt);
+            for (int c = 0; c < 3; ++c)
+                vote[c] += low[c] * weight;
+            vote[3] += weight;
+            vote[8] += m_source->texture().at<unsigned char>(ys, xs) * weight;
+            if (best_weight <= vote[7])
+                return;
+            for (int c = 0; c < 3; ++c)
+                vote[4 + c] = color[c] - low[c];
+            vote[7] = best_weight;
+        }
+
         const NearestNeighborField *m_nnf;
         cv::Mat *m_vote;
         const MaskedImage *m_source;
@@ -339,7 +539,27 @@ namespace
         bool m_source2target;
         bool m_upscaled;
         bool m_best_only;
+        // cv::Mat::empty() is not inline in OpenCV 5, too slow to call for every vote
+        bool m_split;
+        cv::Mat m_low; // split votes only
     };
+
+    // The known texture, and in the holes twice the mean texture of the pixels that the split votes came from:
+    // the distances charge a deficit below half of it, so a fill expects the full texture of its sources.
+    cv::Mat voted_texture(const MaskedImage &source, const cv::Mat &vote)
+    {
+        cv::Mat texture = source.texture().clone();
+        for (int y = 0; y < texture.rows; ++y)
+        {
+            for (int x = 0; x < texture.cols; ++x)
+            {
+                const auto *sums = vote.ptr<double>(y, x);
+                if (is_hole(source, y, x) && sums[3] > 0)
+                    texture.at<unsigned char>(y, x) = cv::saturate_cast<unsigned char>(2 * sums[8] / sums[3]);
+            }
+        }
+        return texture;
+    }
 } // namespace
 
 /**
@@ -383,10 +603,25 @@ void Inpainting::_initialize_pyramid()
         m_pyramid.push_back(source);
     }
 
+    // The coarser levels cannot tell blurred from sharp sources by their colors, so every level
+    // compares the texture of the finest one, carried into the holes from their surroundings.
+    cv::Mat texture = texture_energy(m_pyramid[0]);
+    pull_push(texture);
+    for (std::size_t level = 0; level < m_pyramid.size(); ++level)
+    {
+        if (level > 0)
+            texture = halve(texture, m_pyramid[level].size());
+        m_pyramid[level].set_texture(to_bytes(texture));
+    }
+
     for (int level = 0; level < static_cast<int>(m_pyramid.size()); ++level)
     {
         if (has_hole(m_pyramid[level]))
             m_coarsest_hole_level = level;
+        const int side = std::min(m_pyramid[level].size().height, m_pyramid[level].size().width);
+        const int fitting = std::max(kMinFittingPatchSize, (side / kLevelSidesPerPatch - 1) / 2);
+        m_fitting_metrics.push_back(
+            fitting < m_distance_metric->patch_size() ? m_distance_metric->with_patch_size(fitting) : nullptr);
     }
     if (m_distance_metric->patch_size() < kCoarsePatchSize)
         m_coarse_metric = m_distance_metric->with_patch_size(kCoarsePatchSize);
@@ -431,12 +666,12 @@ cv::Mat Inpainting::run(bool verbose, unsigned int random_seed)
 MaskedImage Inpainting::_run_levels(
     int from, int to, MaskedImage target, const PatchDistanceMetric *metric, bool verbose)
 {
-    m_level_metric = metric;
     for (int level = from; level >= to; --level)
     {
         trace(verbose, "Inpainting level: ", level);
 
         const MaskedImage &source = m_pyramid[level];
+        m_level_metric = m_fitting_metrics[level] ? m_fitting_metrics[level].get() : metric;
         const bool coarsest = level == static_cast<int>(m_pyramid.size()) - 1;
         if (coarsest)
         {
@@ -448,10 +683,10 @@ MaskedImage Inpainting::_run_levels(
         if (level < 2)
             m_source2target.reset();
         else
-            m_source2target = coarsest ? NearestNeighborField(source, target, metric)
-                                       : NearestNeighborField(source, target, metric, *m_source2target);
-        m_target2source = coarsest ? NearestNeighborField(target, source, metric)
-                                   : NearestNeighborField(target, source, metric, m_target2source);
+            m_source2target = coarsest ? NearestNeighborField(source, target, m_level_metric)
+                                       : NearestNeighborField(source, target, m_level_metric, *m_source2target);
+        m_target2source = coarsest ? NearestNeighborField(target, source, m_level_metric)
+                                   : NearestNeighborField(target, source, m_level_metric, m_target2source);
 
         trace(verbose, "Initialization done.");
 
@@ -559,14 +794,10 @@ MaskedImage Inpainting::_expectation_maximization(
             new_target = target.clone();
         }
 
-        auto vote = cv::Mat(new_target.size(), CV_64FC4);
-        vote.setTo(cv::Scalar::all(0));
-
         // The weighted mean of all overlapping patches blurs the fill, so in the last
         // iteration of the finest level each pixel takes the value of its most similar
         // patch instead (similar to "Space-Time Completion of Video", which takes the
-        // mode of the votes). On coarser levels the mean is kept: it lays out the smooth
-        // structure that the finer levels refine, the best vote there makes seams.
+        // mode of the votes). On coarser levels the best vote makes seams.
         const bool best_only = level == 0 && iter_em == nr_iters_em - 1;
 
         // Known pixels are kept on the two finest levels only. On coarser levels those
@@ -576,6 +807,12 @@ MaskedImage Inpainting::_expectation_maximization(
         const int new_level = upscaled ? level - 1 : level;
         const bool keep_known = new_level <= 1;
 
+        // The other votes for these levels average only their low frequencies and add the
+        // detail of the best vote: with the plain mean the fill stayed smooth, and smooth
+        // fills match smooth sources best. On coarser levels the mean lays out the structure.
+        const bool split = keep_known && !best_only;
+        auto vote = cv::Mat(new_target.size(), CV_64FC(split ? kSplitChannels : 4), cv::Scalar::all(0));
+
         // Votes for best patch from NNF Source->Target (completeness) and Target->Source (coherence).
         if (m_source2target)
         {
@@ -588,6 +825,11 @@ MaskedImage Inpainting::_expectation_maximization(
         // Compile votes and update pixel values.
         _maximization_step(new_target, vote, new_source, keep_known);
         trace(verbose, "  Minimization step finished.");
+
+        // The fill expects the texture of its level: the known one, or that around the holes. On level 0
+        // that of the sources chosen on level 1: with the texture around the holes, the search there took
+        // blurred sources beside detailed surroundings, and parts of the fill stayed flat.
+        new_target.set_texture(upscaled && level == 1 ? voted_texture(new_source, vote) : new_source.texture());
     }
 
     return new_target;
@@ -628,7 +870,7 @@ void Inpainting::_expectation_step(
     const VoteCaster caster(
         nnf, source2target, vote, source, upscaled, best_only, holes.has_value() ? &holes.value() : nullptr);
     // The larger patches of m_coarse_metric would average more votes, which blurred the fills.
-    const int patch_size = m_distance_metric->patch_size();
+    const int patch_size = std::min(m_distance_metric->patch_size(), m_level_metric->patch_size());
     const auto &kDistance2Similarity = distance2similarity();
     const auto source_size = nnf.source_size();
     for (int i = 0; i < source_size.height; ++i)
@@ -645,12 +887,15 @@ void Inpainting::_expectation_step(
 void Inpainting::_maximization_step(
     MaskedImage &target, const cv::Mat &vote, const MaskedImage &source, bool keep_known) const
 {
+    constexpr std::array<double, 3> kNoDetail = {0, 0, 0};
+    const bool split = vote.channels() == kSplitChannels;
     auto target_size = target.size();
     for (int i = 0; i < target_size.height; ++i)
     {
         for (int j = 0; j < target_size.width; ++j)
         {
             const double *source_ptr = vote.ptr<double>(i, j);
+            const double *detail = split ? source_ptr + 4 : kNoDetail.data();
             unsigned char *target_ptr = target.get_mutable_image(i, j);
 
             if (target.is_globally_masked(i, j))
@@ -671,9 +916,9 @@ void Inpainting::_maximization_step(
             if (source_ptr[3] > 0)
             {
                 // All three before the stores: they may alias the votes, which changes the fast-math divisions.
-                const auto r = cv::saturate_cast<unsigned char>(source_ptr[0] / source_ptr[3]);
-                const auto g = cv::saturate_cast<unsigned char>(source_ptr[1] / source_ptr[3]);
-                const auto b = cv::saturate_cast<unsigned char>(source_ptr[2] / source_ptr[3]);
+                const auto r = cv::saturate_cast<unsigned char>(source_ptr[0] / source_ptr[3] + detail[0]);
+                const auto g = cv::saturate_cast<unsigned char>(source_ptr[1] / source_ptr[3] + detail[1]);
+                const auto b = cv::saturate_cast<unsigned char>(source_ptr[2] / source_ptr[3] + detail[2]);
                 target_ptr[0] = r;
                 target_ptr[1] = g;
                 target_ptr[2] = b;

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/patchmatch/csrc/inpaint.h

@@ -35,7 +35,9 @@ private:
     const PatchDistanceMetric *m_distance_metric;
     // Searches on the levels that lay out the fill, if patch_size is smaller than kCoarsePatchSize.
     std::unique_ptr<PatchDistanceMetric> m_coarse_metric;
+    // Per level: smaller patches where patch_size does not fit, otherwise nullptr.
+    std::vector<std::unique_ptr<PatchDistanceMetric>> m_fitting_metrics;
     int m_coarsest_hole_level = -1;
-    // Searches on the current level; the votes use patch_size.
+    // Searches on the current level; the votes use the smaller of its patch size and patch_size.
     const PatchDistanceMetric *m_level_metric = nullptr;
 };

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/patchmatch/csrc/masked_image.h

@@ -24,10 +24,12 @@ public:
     MaskedImage &operator=(MaskedImage &&) noexcept = default;
     ~MaskedImage() = default;
 
-    // Without the gradients, which would be outdated once the clone is changed.
+    // Without the gradients, which would be outdated once the clone is changed. It shares the texture.
     MaskedImage clone() const
     {
-        return MaskedImage(m_image.clone(), m_mask.clone(), m_global_mask.clone());
+        MaskedImage copy(m_image.clone(), m_mask.clone(), m_global_mask.clone());
+        copy.set_texture(m_texture);
+        return copy;
     }
 
     cv::Size size() const
@@ -61,6 +63,22 @@ public:
         return m_image_gradx;
     }
 
+    // The texture energy that the distances compare (CV_8UC1, see Inpainting), or empty.
+    // It is never changed, and the resampling does not carry it over.
+    const cv::Mat &texture() const
+    {
+        return m_texture;
+    }
+    bool has_texture() const
+    {
+        return m_has_texture;
+    }
+    void set_texture(const cv::Mat &texture)
+    {
+        m_texture = texture;
+        m_has_texture = !texture.empty();
+    }
+
     void init_global_mask_mat()
     {
         m_global_mask = cv::Mat(m_mask.size(), CV_8U);
@@ -118,8 +136,10 @@ private:
     cv::Mat m_image;
     cv::Mat m_mask;
     cv::Mat m_global_mask;
+    cv::Mat m_texture;
     // cv::Mat::empty() is not inline in OpenCV 5, too slow to call for every pixel
     bool m_has_global_mask = false;
+    bool m_has_texture = false;
     // computed on first use by compute_image_gradients
     mutable cv::Mat m_image_grady;
     mutable cv::Mat m_image_gradx;

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/patchmatch/csrc/nnf.cpp

@@ -52,7 +52,7 @@ namespace
     {
         ImagePair(const MaskedImage &source_image, const MaskedImage &target_image)
             : source(&source_image), target(&target_image), source_size(source_image.size()),
-              target_size(target_image.size())
+              target_size(target_image.size()), textured(source_image.has_texture() && target_image.has_texture())
         {
         }
 
@@ -65,6 +65,7 @@ namespace
         const MaskedImage *target;
         cv::Size source_size;
         cv::Size target_size;
+        bool textured;
     };
 
     // One row of a source and a target image, both inside their images. The gradients must be computed.
@@ -84,7 +85,9 @@ namespace
               m_sgy(images.source->grady().ptr<unsigned char>(ys, 0)),
               m_tgy(images.target->grady().ptr<unsigned char>(yt, 0)),
               m_sgx(images.source->gradx().ptr<unsigned char>(ys, 0)),
-              m_tgx(images.target->gradx().ptr<unsigned char>(yt, 0))
+              m_tgx(images.target->gradx().ptr<unsigned char>(yt, 0)),
+              m_st(images.textured ? images.source->texture().ptr<unsigned char>(ys) : nullptr),
+              m_tt(images.textured ? images.target->texture().ptr<unsigned char>(yt) : nullptr)
         {
         }
 
@@ -103,6 +106,10 @@ namespace
                 ssd += pow2(m_sgx[xs * 3 + c] - m_tgx[xt * 3 + c]);
                 ssd += pow2(m_sgy[xs * 3 + c] - m_tgy[xt * 3 + c]);
             }
+            // Smooth fills matched smooth sources and stayed flat, so a target pixel with less than half the
+            // texture of its source pixel costs extra. Not a two-sided difference: it removed mortar joints.
+            if (m_st)
+                ssd += 2 * pow2(std::max(m_st[xs] / 2 - m_tt[xt], 0));
             return ssd;
         }
 
@@ -119,6 +126,8 @@ namespace
         const unsigned char *m_tgy;
         const unsigned char *m_sgx;
         const unsigned char *m_tgx;
+        const unsigned char *m_st; // the textures, nullptr unless both images have one
+        const unsigned char *m_tt;
     };
 
     // The costs of the row of the patches around s and t.

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/pyproject.toml

@@ -44,6 +44,7 @@ Issues = "https://github.com/mauwii/PyPatchMatch/issues"
 coverage = ["gcovr"]
 # numpy and pillow for mypy, which runs without the project (--only-group dev)
 dev = [{ include-group = "test" }, "mypy", "numpy", "pillow", "pre-commit", "ruff"]
+gui = ["gradio"]
 test = ["hypothesis", "pytest", "pytest-cov"]
 
 [tool.scikit-build]
@@ -59,11 +60,13 @@ metadata.readme.provider = "scikit_build_core.metadata.fancy_pypi_readme"
 # The release wheels are built from the sdist, so it has to contain the tests
 # and their input images; _version.py is generated and therefore git-ignored.
 sdist.include = [
+    "examples/images/camera_damaged.png",
+    "examples/images/camera_damaged_mask.png",
     "examples/images/chelsea.png",
     "examples/images/forest_pruned.bmp",
     "patchmatch/_version.py",
 ]
-sdist.exclude = [".*", "examples", "sonar-project.properties", "uv.lock"]
+sdist.exclude = [".*", "examples", "gui", "sonar-project.properties", "uv.lock"]
 
 # Without a matching wheel, pip builds the sdist, which needs a C++ compiler, CMake and
 # OpenCV. If that fails, build a wheel without the native library instead of failing

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/scripts/evaluate_inpainting.py

@@ -3,13 +3,16 @@
 The holes are cut into images whose content is known: forest.bmp and the images listed
 in examples/images/SOURCES.md. forest_pruned.bmp and every ``*_pruned.*`` image in the
 git-ignored examples/images/local/ are added as object removals without a ground
-truth; their pure white pixels are the holes. grass-edge cuts a band at the right
-border, as in outpainting; brick-large and grass-large cut a circle of radius 242 into
-the center, 70 % of the image. pattern repeats almost exactly, so its error tells a fill
-that continues it (2 to 6) from a broken one; pattern-wide widens the hole to a radius
-of 60, where patch size 3 mixed the rings before 2.2 (error 37). forest-global and
-brick-global add a global mask, darkened in the report: the plant of
-examples/py_example_global_mask.py and the left third of brick.png.
+truth; their pure white pixels are the holes. camera restores the damage of
+camera_damaged.png, which camera_damaged_mask.png marks, against camera_aged.png; its
+error covers the larger holes and the torn corner, the scratches are narrower than the
+cells it compares. grass-edge cuts a band at the right border, as in outpainting;
+brick-large and grass-large cut a circle of radius 242 into the center, 70 % of the
+image. pattern repeats almost exactly, so its error tells a fill that continues it (2 to
+6) from a broken one; pattern-wide widens the hole to a radius of 60, where patch size 3
+mixed the rings before 2.2 (error 37). forest-global and brick-global add a global mask,
+darkened in the report: the plant of examples/py_example_global_mask.py and the left
+third of brick.png.
 
 Usage:
     uv run python scripts/evaluate_inpainting.py
@@ -116,6 +119,9 @@ def cases() -> Iterator[Case]:
     forest = load(IMAGES / "forest.bmp")
     pruned = load(IMAGES / "forest_pruned.bmp")
     yield Case("forest", pruned, white(pruned), None)
+    damaged = load(IMAGES / "camera_damaged.png")
+    damage = np.array(Image.open(IMAGES / "camera_damaged_mask.png")) > 0
+    yield Case("camera", damaged, damage, load(IMAGES / "camera_aged.png"))
     yield cut("trees", forest, ellipse(forest.shape, 70, 560, 45, 55))
     meadow = np.zeros(forest.shape[:2], dtype=bool)
     meadow[318:350, 380:470] = True

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b2}/tests/test_e2e.py

@@ -36,15 +36,21 @@ def surrounding(mask: np.ndarray, width: int = 10) -> np.ndarray:
 
 
 def assert_plausible_fill(
-    source: np.ndarray, result: np.ndarray, excluded: np.ndarray | None = None
+    source: np.ndarray,
+    result: np.ndarray,
+    excluded: np.ndarray | None = None,
+    holes: np.ndarray | None = None,
 ) -> None:
     """Check an inpainting result of ``source`` without a ground truth image.
 
-    ``excluded`` marks the pixels of a global mask, which are not filled.
+    ``excluded`` marks the pixels of a global mask, which are not filled. ``holes``
+    defaults to the white pixels of ``source``.
     """
     if excluded is None:
         excluded = np.zeros(source.shape[:2], dtype=bool)
-    holes = evaluation.white(source) & ~excluded
+    if holes is None:
+        holes = evaluation.white(source)
+    holes = holes & ~excluded
     assert holes.any()
     assert result.shape == source.shape
     assert result.dtype == np.uint8
@@ -130,10 +136,55 @@ def test_global_mask_is_not_used_as_source(pruned_image):
     np.testing.assert_array_equal(result[excluded], source[excluded])
 
 
-def test_fill_is_as_detailed_as_its_surroundings():
-    """The weighted mean of all overlapping patches blurred the fill (detail 0.69)."""
-    source, result = fill("forest_pruned.bmp", None, 3)
-    assert evaluation.measure(result, evaluation.white(source))["detail"] > 0.8
+def test_restore_aged_photo():
+    """Scratches, holes and a torn corner, light but not white, given as a mask."""
+    source = Image.open(IMAGES / "camera_damaged.png")
+    mask = Image.open(IMAGES / "camera_damaged_mask.png")
+
+    result = patchmatch.inpaint(source, mask, patch_size=3)
+
+    assert_plausible_fill(np.array(source), result, holes=np.array(mask) > 0)
+
+
+@pytest.mark.parametrize(
+    ("image", "hole", "patch_size", "limit"),
+    [
+        ("forest_pruned.bmp", None, 3, 0.8),
+        ("chelsea.png", evaluation.HOLES["chelsea"], 3, 0.7),
+        ("chelsea.png", evaluation.HOLES["chelsea"], 15, 0.4),
+    ],
+)
+def test_fill_is_as_detailed_as_its_surroundings(image, hole, patch_size, limit):
+    """The weighted mean of all overlapping patches blurred the fill (detail 0.69).
+
+    On the finer levels it kept the fill of the fur in chelsea smooth, which then
+    matched smooth sources best (detail 0.40). With patch size 15, the coarser levels
+    already took smooth sources, which look like fur once downsampled (detail 0.29).
+    """
+    source, result = fill(image, hole, patch_size)
+    assert evaluation.measure(result, evaluation.white(source))["detail"] > limit
+
+
+def test_no_part_of_the_fill_is_flat():
+    """The finest level linked the left of the hole in chelsea to the blurred fur at
+    the border of the image. That part stayed flat (detail 0.30 in the flattest 16x16
+    block) while the whole fill reached 0.70.
+    """
+    source, result = fill("chelsea.png", evaluation.HOLES["chelsea"], 7)
+    holes = evaluation.white(source)
+    inside = holes & ~evaluation.grow(~holes, 2)
+    around = evaluation.grow(holes, 24) & ~evaluation.grow(holes, 2)
+    ys, xs = np.nonzero(inside)
+    details = []
+    for y in range(ys.min(), ys.max() + 1, 16):
+        for x in range(xs.min(), xs.max() + 1, 16):
+            block = np.zeros_like(inside)
+            block[y : y + 16, x : x + 16] = True
+            block &= inside
+            if block.sum() > 128:
+                details.append(evaluation.neighbor_difference(result, block, block))
+    reference = evaluation.neighbor_difference(result, around, around)
+    assert min(details) / reference > 0.5
 
 
 @pytest.mark.parametrize(
@@ -189,13 +240,22 @@ def test_explicit_mask_matches_implicit_white_mask(pruned_image):
     np.testing.assert_array_equal(explicit, fill("forest_pruned.bmp", None, 3)[1])
 
 
-# Regions of pure background in forest_pruned.bmp as (y, x, height, width).
-BACKGROUND_REGIONS = [(20, 400, 60, 80), (40, 520, 60, 60)]
+# Regions of pure background in forest_pruned.bmp as (y, x, height, width), with the
+# patch size to fill them.
+BACKGROUND_REGIONS = [
+    ((20, 400, 60, 80), 3),
+    ((40, 520, 60, 60), 3),
+    ((25, 505, 90, 110), 15),
+]
 
 
-@pytest.mark.parametrize("region", BACKGROUND_REGIONS)
-def test_reconstructs_known_background(pruned_image, region):
-    """Cut a hole into known background and compare with the original pixels."""
+@pytest.mark.parametrize(("region", "patch_size"), BACKGROUND_REGIONS)
+def test_reconstructs_known_background(pruned_image, region, patch_size):
+    """Cut a hole into known background and compare with the original pixels.
+
+    With patch size 15, the coarser levels found no sources inside the image and
+    outside the hole, and the trees got a lighter patch (0.90 of the naive error).
+    """
     truth = np.array(pruned_image)
     y, x, height, width = region
     mask = np.zeros(truth.shape[:2], dtype=bool)
@@ -204,7 +264,7 @@ def test_reconstructs_known_background(pruned_image, region):
 
     source = truth.copy()
     source[mask] = 255
-    result = patchmatch.inpaint(source, mask.astype(np.uint8), patch_size=3)
+    result = patchmatch.inpaint(source, mask.astype(np.uint8), patch_size=patch_size)
 
     # baseline: fill the hole with the mean color of its surroundings
     naive = source.copy()