PyPatchMatch 2.2.0b1__tar.gz → 2.2.0b3__tar.gz

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/CLAUDE.md

@@ -62,6 +62,9 @@ Other entry points:
 - `uv run python scripts/render_readme_example.py` runs `py_example.py` and renders the
   picture at the top of the README, `examples/images/readme_example.jpg`; rerun it and
   commit the picture after changes that affect the fills.
+- `uv run --group gui gui/app.py` starts the Gradio GUI (paint holes, compare the fill).
+  No CI job installs gradio, so after API changes run its test with
+  `uv run --group gui pytest gui --no-cov` and try the GUI by hand.
 - CMake options: the `option()` lines of `CMakeLists.txt`. `PATCHMATCH_SANITIZE` turns
   off fast math, which lets the compiler drop checks.
 

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: PyPatchMatch
-Version: 2.2.0b1
+Version: 2.2.0b3
 Summary: This library implements the PatchMatch based inpainting algorithm.
 Keywords: inpainting,patchmatch,image-processing,opencv
 Author-Email: The InvokeAI Project <lincoln.stein@gmail.com>, Jiayuan Mao <maojiayuan@gmail.com>, Kyle Schouviller <kyle0654@hotmail.com>, Matthias Wild <mauwii@outlook.de>, Younesse ANDAM <younesse.andam@gmail.com>
@@ -41,9 +41,12 @@ Description-Content-Type: text/markdown
 This library implements the PatchMatch based inpainting algorithm. It provides both C++
 and Python interfaces. This implementation is heavily based on the implementation by
 Younesse ANDAM: [younesse-cv/PatchMatch](https://github.com/younesse-cv/PatchMatch),
-with some bug fixes, and updates.
+with some bug fixes, and updates. This repository continues
+[vacancy/PyPatchMatch](https://github.com/vacancy/PyPatchMatch) by Jiayuan Mao, through
+the forks [lkwq007/PyPatchMatch](https://github.com/lkwq007/PyPatchMatch) and
+[invoke-ai/PyPatchMatch](https://github.com/invoke-ai/PyPatchMatch).
 
-![Birdhouses removed from a photo by patchmatch.inpaint](https://raw.githubusercontent.com/mauwii/PyPatchMatch/v2.2.0b1/examples/images/readme_example.jpg)
+![Birdhouses removed from a photo by patchmatch.inpaint](https://raw.githubusercontent.com/mauwii/PyPatchMatch/v2.2.0b3/examples/images/readme_example.jpg)
 
 From left to right: the photo, the input of
 [examples/py_example.py](https://github.com/mauwii/PyPatchMatch/blob/main/examples/py_example.py)
@@ -100,7 +103,9 @@ The previous import path `from patchmatch import patch_match` keeps working.
 pixels in each direction. Larger patches follow larger structures but are much slower;
 the default of 15 compares 31x31 patches, the examples use 3. Below 5, the coarse levels
 that lay out the fill are also tried with a radius of 5, which is kept where it fits
-the surroundings better, so that repeating patterns continue across larger holes.
+the surroundings better, so that repeating patterns continue across larger holes. Above
+7, coarse levels narrower than six patches use the largest radius that fits, but at
+least 7: larger patches found no sources there and left the fill flat.
 
 `patchmatch.set_random_seed(seed)` sets the seed of the randomized search, modulo
 `2**32`, and `patchmatch.set_verbose(True)` prints the progress of the native code to
@@ -138,8 +143,14 @@ uv run pre-commit install      # enable ruff and the other hooks on commit
 uv run pytest                  # run the test suite
 uv run pytest -m "not e2e"     # only the fast unit tests
 uv run python scripts/evaluate_inpainting.py  # measure the fill quality
+uv run --group gui gui/app.py  # draw masks in the browser, http://127.0.0.1:7860
 ```
 
+The GUI tries new images quickly, without writing a script for each one, e.g. to pick
+new examples or to verify edge cases: load an image, paint the holes (every painted
+pixel is one) and compare the original with the fill. Its test runs with
+`uv run --group gui pytest gui --no-cov`.
+
 The evaluation cuts holes into the images of `examples/images`, compares the fills with
 the original content and writes `examples/images/evaluation.html`, which shows them side
 by side. `--baseline` evaluates another build of the library as well, e.g. of the main

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/README.md

@@ -11,7 +11,10 @@
 This library implements the PatchMatch based inpainting algorithm. It provides both C++
 and Python interfaces. This implementation is heavily based on the implementation by
 Younesse ANDAM: [younesse-cv/PatchMatch](https://github.com/younesse-cv/PatchMatch),
-with some bug fixes, and updates.
+with some bug fixes, and updates. This repository continues
+[vacancy/PyPatchMatch](https://github.com/vacancy/PyPatchMatch) by Jiayuan Mao, through
+the forks [lkwq007/PyPatchMatch](https://github.com/lkwq007/PyPatchMatch) and
+[invoke-ai/PyPatchMatch](https://github.com/invoke-ai/PyPatchMatch).
 
 ![Birdhouses removed from a photo by patchmatch.inpaint](examples/images/readme_example.jpg)
 
@@ -70,7 +73,9 @@ The previous import path `from patchmatch import patch_match` keeps working.
 pixels in each direction. Larger patches follow larger structures but are much slower;
 the default of 15 compares 31x31 patches, the examples use 3. Below 5, the coarse levels
 that lay out the fill are also tried with a radius of 5, which is kept where it fits
-the surroundings better, so that repeating patterns continue across larger holes.
+the surroundings better, so that repeating patterns continue across larger holes. Above
+7, coarse levels narrower than six patches use the largest radius that fits, but at
+least 7: larger patches found no sources there and left the fill flat.
 
 `patchmatch.set_random_seed(seed)` sets the seed of the randomized search, modulo
 `2**32`, and `patchmatch.set_verbose(True)` prints the progress of the native code to
@@ -108,8 +113,14 @@ uv run pre-commit install      # enable ruff and the other hooks on commit
 uv run pytest                  # run the test suite
 uv run pytest -m "not e2e"     # only the fast unit tests
 uv run python scripts/evaluate_inpainting.py  # measure the fill quality
+uv run --group gui gui/app.py  # draw masks in the browser, http://127.0.0.1:7860
 ```
 
+The GUI tries new images quickly, without writing a script for each one, e.g. to pick
+new examples or to verify edge cases: load an image, paint the holes (every painted
+pixel is one) and compare the original with the fill. Its test runs with
+`uv run --group gui pytest gui --no-cov`.
+
 The evaluation cuts holes into the images of `examples/images`, compares the fills with
 the original content and writes `examples/images/evaluation.html`, which shows them side
 by side. `--baseline` evaluates another build of the library as well, e.g. of the main

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/patchmatch/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '2.2.0b1'
-__version_tuple__ = version_tuple = (2, 2, 0, 'b1')
+__version__ = version = '2.2.0b3'
+__version_tuple__ = version_tuple = (2, 2, 0, 'b3')
 
-__commit_id__ = commit_id = 'gb66a4a1d8'
+__commit_id__ = commit_id = 'gc0f3bbfe6'

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/patchmatch/csrc/inpaint.cpp

@@ -92,10 +92,159 @@ namespace
         return false;
     }
 
+    // The gradient magnitude at a known pixel, summed over the channels, or -1 if it or a neighbor is unknown.
+    double gradient_magnitude(const MaskedImage &image, int y, int x)
+    {
+        const auto size = image.size();
+        const int up = std::max(y - 1, 0);
+        const int down = std::min(y + 1, size.height - 1);
+        const int left = std::max(x - 1, 0);
+        const int right = std::min(x + 1, size.width - 1);
+        if (!is_known(image, y, x) || !is_known(image, up, x) || !is_known(image, down, x) ||
+            !is_known(image, y, left) || !is_known(image, y, right))
+            return -1;
+        int magnitude = 0;
+        for (int c = 0; c < 3; ++c)
+            magnitude += std::abs(image.get_image(down, x)[c] - image.get_image(up, x)[c]) +
+                         std::abs(image.get_image(y, right)[c] - image.get_image(y, left)[c]);
+        return magnitude / 2.0;
+    }
+
+    // The mean gradient magnitude around each known pixel, -1 elsewhere or without one. Gradients
+    // across unknown pixels are left out, so the colors under the holes do not matter.
+    cv::Mat texture_energy(const MaskedImage &image)
+    {
+        constexpr int kRadius = 2;
+        const auto size = image.size();
+        // above and left of each pixel: the sum of the magnitudes and their number
+        cv::Mat sums(size.height + 1, size.width + 1, CV_64FC2, cv::Scalar::all(0));
+        for (int y = 0; y < size.height; ++y)
+        {
+            for (int x = 0; x < size.width; ++x)
+            {
+                const double magnitude = gradient_magnitude(image, y, x);
+                const cv::Vec2d measured = magnitude >= 0 ? cv::Vec2d(magnitude, 1) : cv::Vec2d();
+                sums.at<cv::Vec2d>(y + 1, x + 1) =
+                    measured + sums.at<cv::Vec2d>(y, x + 1) + sums.at<cv::Vec2d>(y + 1, x) - sums.at<cv::Vec2d>(y, x);
+            }
+        }
+
+        cv::Mat energy(size, CV_64F, cv::Scalar(-1));
+        for (int y = 0; y < size.height; ++y)
+        {
+            const int top = std::max(y - kRadius, 0);
+            const int bottom = std::min(y + kRadius + 1, size.height);
+            for (int x = 0; x < size.width; ++x)
+            {
+                const int left = std::max(x - kRadius, 0);
+                const int right = std::min(x + kRadius + 1, size.width);
+                const cv::Vec2d window = sums.at<cv::Vec2d>(bottom, right) - sums.at<cv::Vec2d>(top, right) -
+                                         sums.at<cv::Vec2d>(bottom, left) + sums.at<cv::Vec2d>(top, left);
+                if (is_known(image, y, x) && window[1] > 0)
+                    energy.at<double>(y, x) = window[0] / window[1];
+            }
+        }
+        return energy;
+    }
+
+    // The mean of the values >= 0 in the 2x2 block from (y, x), -1 if there is none.
+    double block_mean(const cv::Mat &values, int y, int x)
+    {
+        double sum = 0;
+        int count = 0;
+        for (int yy = y; yy < std::min(y + 2, values.rows); ++yy)
+        {
+            for (int xx = x; xx < std::min(x + 2, values.cols); ++xx)
+            {
+                if (values.at<double>(yy, xx) >= 0)
+                {
+                    sum += values.at<double>(yy, xx);
+                    ++count;
+                }
+            }
+        }
+        return count > 0 ? sum / count : -1;
+    }
+
+    // The value of coarse, which has half the resolution, at the center of pixel (y, x), bilinear.
+    double interpolate(const cv::Mat &coarse, int y, int x)
+    {
+        const double cy = std::clamp((y + 0.5) / 2 - 0.5, 0.0, coarse.rows - 1.0);
+        const double cx = std::clamp((x + 0.5) / 2 - 0.5, 0.0, coarse.cols - 1.0);
+        const auto y0 = static_cast<int>(cy);
+        const auto x0 = static_cast<int>(cx);
+        const int y1 = std::min(y0 + 1, coarse.rows - 1);
+        const int x1 = std::min(x0 + 1, coarse.cols - 1);
+        const double fy = cy - y0;
+        const double fx = cx - x0;
+        return (1 - fy) * ((1 - fx) * coarse.at<double>(y0, x0) + fx * coarse.at<double>(y0, x1)) +
+               fy * ((1 - fx) * coarse.at<double>(y1, x0) + fx * coarse.at<double>(y1, x1));
+    }
+
+    // Not std::all_of: libstdc++ subtracts the iterators, which divides by zero for an empty cv::Mat.
+    bool has_unknown(const cv::Mat &values)
+    {
+        for (int y = 0; y < values.rows; ++y)
+            for (int x = 0; x < values.cols; ++x)
+                if (values.at<double>(y, x) < 0)
+                    return true;
+        return false;
+    }
+
+    // Replaces the values below 0 from a pyramid of the others (pull-push), which carries the
+    // values around a hole smoothly into it.
+    void pull_push(cv::Mat &values)
+    {
+        if (!has_unknown(values))
+            return;
+        if (values.total() == 1)
+        {
+            values.at<double>(0, 0) = 0; // nothing is known
+            return;
+        }
+        cv::Mat coarse((values.rows + 1) / 2, (values.cols + 1) / 2, CV_64F);
+        for (int y = 0; y < coarse.rows; ++y)
+            for (int x = 0; x < coarse.cols; ++x)
+                coarse.at<double>(y, x) = block_mean(values, 2 * y, 2 * x);
+        pull_push(coarse);
+        for (int y = 0; y < values.rows; ++y)
+        {
+            for (int x = 0; x < values.cols; ++x)
+            {
+                if (values.at<double>(y, x) < 0)
+                    values.at<double>(y, x) = interpolate(coarse, y, x);
+            }
+        }
+    }
+
+    // The texture of the next level: the means of the 2x2 blocks.
+    cv::Mat halve(const cv::Mat &values, cv::Size size)
+    {
+        cv::Mat half(size, CV_64F);
+        for (int y = 0; y < size.height; ++y)
+            for (int x = 0; x < size.width; ++x)
+                half.at<double>(y, x) = block_mean(values, 2 * y, 2 * x);
+        return half;
+    }
+
+    cv::Mat to_bytes(const cv::Mat &values)
+    {
+        cv::Mat bytes(values.size(), CV_8U);
+        for (int y = 0; y < values.rows; ++y)
+            for (int x = 0; x < values.cols; ++x)
+                bytes.at<unsigned char>(y, x) = cv::saturate_cast<unsigned char>(values.at<double>(y, x));
+        return bytes;
+    }
+
     // A patch smaller than a structure cannot tell where it continues: on the coarsest level
     // with a hole, holes wider than the patches were filled from several shifted copies.
     constexpr int kCoarsePatchSize = 5;
 
+    // A patch wider than a sixth of a level found no sources there inside the image and outside
+    // the holes, and the fill stayed flat. Patches up to 7 keep their size: smaller ones lost the joints of bricks.
+    constexpr int kLevelSidesPerPatch = 6;
+    constexpr int kMinFittingPatchSize = 7;
+
     int nnf_iterations(int level)
     {
         return std::min(7, 1 + level);
@@ -191,6 +340,33 @@ namespace
         target_ptr[3] += weight;
     }
 
+    // A split vote: the weighted low frequencies and their weight, the detail of the best vote and its weight,
+    // and the weighted texture of the source pixels.
+    constexpr int kSplitChannels = 9;
+
+    // The mean color of the known pixels around each pixel, the part of the split votes that is averaged.
+    cv::Mat low_pass(const MaskedImage &image)
+    {
+        const auto size = image.size();
+        const auto known = [&image](int y, int x) { return is_known(image, y, x); };
+        const auto color = [&image](int y, int x) {
+            const auto *pixel = image.get_image(y, x);
+            return cv::Vec3d(pixel[0], pixel[1], pixel[2]);
+        };
+        cv::Mat low(size, CV_8UC3);
+        for (int y = 0; y < size.height; ++y)
+        {
+            for (int x = 0; x < size.width; ++x)
+            {
+                const cv::Vec3d mean = window_mean(size, y, x, 1, known, color);
+                auto *pixel = low.ptr<unsigned char>(y, x);
+                for (int c = 0; c < 3; ++c)
+                    pixel[c] = cv::saturate_cast<unsigned char>(mean[c]);
+            }
+        }
+        return low;
+    }
+
     // Replaces the vote of the pixel if the new one weighs more. It is stored like the
     // sums of _weighted_copy, so the maximization step divides it by its weight.
     inline void _best_copy(const MaskedImage &source, int ys, int xs, cv::Mat &target, int yt, int xt, double weight)
@@ -262,7 +438,8 @@ namespace
             bool upscaled, bool best_only, const HoleCounter *holes)
             : m_nnf(&nnf), m_vote(&vote), m_source(&source), m_holes(holes), m_source_size(nnf.source_size()),
               m_target_size(nnf.target_size()), m_source2target(source2target), m_upscaled(upscaled),
-              m_best_only(best_only)
+              m_best_only(best_only), m_split(vote.channels() == kSplitChannels),
+              m_low(m_split ? low_pass(source) : cv::Mat())
         {
         }
 
@@ -290,7 +467,7 @@ namespace
                         continue;
                     // Near ties go to the patch centered closest, not the first cast, which made blocks.
                     const double offset2 = static_cast<double>(di) * di + static_cast<double>(dj) * dj;
-                    cast_pair(ys, xs, yt, xt, m_best_only ? weight * (1 - 1e-5 * offset2) : weight);
+                    cast_pair(ys, xs, yt, xt, weight, weight * (1 - 1e-5 * offset2));
                 }
             }
         }
@@ -305,7 +482,7 @@ namespace
                 scale * (std::int64_t{y} + patch_size + 1) - 1, scale * (std::int64_t{x} + patch_size + 1) - 1);
         }
 
-        void cast_pair(int ys, int xs, int yt, int xt, double weight) const
+        void cast_pair(int ys, int xs, int yt, int xt, double weight, double best_weight) const
         {
             if (!m_source2target)
             {
@@ -314,22 +491,45 @@ namespace
             }
             if (!m_upscaled)
             {
-                copy(ys, xs, yt, xt, weight);
+                copy(ys, xs, yt, xt, weight, best_weight);
                 return;
             }
             for (int uy = 0; uy < 2; ++uy)
                 for (int ux = 0; ux < 2; ++ux)
-                    copy(2 * ys + uy, 2 * xs + ux, 2 * yt + uy, 2 * xt + ux, weight);
+                    copy(2 * ys + uy, 2 * xs + ux, 2 * yt + uy, 2 * xt + ux, weight, best_weight);
         }
 
-        void copy(int ys, int xs, int yt, int xt, double weight) const
+        void copy(int ys, int xs, int yt, int xt, double weight, double best_weight) const
         {
             if (m_best_only)
-                _best_copy(*m_source, ys, xs, *m_vote, yt, xt, weight);
+                _best_copy(*m_source, ys, xs, *m_vote, yt, xt, best_weight);
+            else if (m_split)
+                split_copy(ys, xs, yt, xt, weight, best_weight);
             else
                 _weighted_copy(*m_source, ys, xs, *m_vote, yt, xt, weight);
         }
 
+        // Adds the low frequencies and the texture of the source pixel to the sums, and keeps the rest if the
+        // vote weighs most.
+        void split_copy(int ys, int xs, int yt, int xt, double weight, double best_weight) const
+        {
+            if (!is_known(*m_source, ys, xs))
+                return;
+
+            const auto *color = m_source->get_image(ys, xs);
+            const auto *low = m_low.ptr<unsigned char>(ys, xs);
+            auto *vote = m_vote->ptr<double>(yt, xt);
+            for (int c = 0; c < 3; ++c)
+                vote[c] += low[c] * weight;
+            vote[3] += weight;
+            vote[8] += m_source->texture().at<unsigned char>(ys, xs) * weight;
+            if (best_weight <= vote[7])
+                return;
+            for (int c = 0; c < 3; ++c)
+                vote[4 + c] = color[c] - low[c];
+            vote[7] = best_weight;
+        }
+
         const NearestNeighborField *m_nnf;
         cv::Mat *m_vote;
         const MaskedImage *m_source;
@@ -339,7 +539,27 @@ namespace
         bool m_source2target;
         bool m_upscaled;
         bool m_best_only;
+        // cv::Mat::empty() is not inline in OpenCV 5, too slow to call for every vote
+        bool m_split;
+        cv::Mat m_low; // split votes only
     };
+
+    // The known texture, and in the holes twice the mean texture of the pixels that the split votes came from:
+    // the distances charge a deficit below half of it, so a fill expects the full texture of its sources.
+    cv::Mat voted_texture(const MaskedImage &source, const cv::Mat &vote)
+    {
+        cv::Mat texture = source.texture().clone();
+        for (int y = 0; y < texture.rows; ++y)
+        {
+            for (int x = 0; x < texture.cols; ++x)
+            {
+                const auto *sums = vote.ptr<double>(y, x);
+                if (is_hole(source, y, x) && sums[3] > 0)
+                    texture.at<unsigned char>(y, x) = cv::saturate_cast<unsigned char>(2 * sums[8] / sums[3]);
+            }
+        }
+        return texture;
+    }
 } // namespace
 
 /**
@@ -383,10 +603,25 @@ void Inpainting::_initialize_pyramid()
         m_pyramid.push_back(source);
     }
 
+    // The coarser levels cannot tell blurred from sharp sources by their colors, so every level
+    // compares the texture of the finest one, carried into the holes from their surroundings.
+    cv::Mat texture = texture_energy(m_pyramid[0]);
+    pull_push(texture);
+    for (std::size_t level = 0; level < m_pyramid.size(); ++level)
+    {
+        if (level > 0)
+            texture = halve(texture, m_pyramid[level].size());
+        m_pyramid[level].set_texture(to_bytes(texture));
+    }
+
     for (int level = 0; level < static_cast<int>(m_pyramid.size()); ++level)
     {
         if (has_hole(m_pyramid[level]))
             m_coarsest_hole_level = level;
+        const int side = std::min(m_pyramid[level].size().height, m_pyramid[level].size().width);
+        const int fitting = std::max(kMinFittingPatchSize, (side / kLevelSidesPerPatch - 1) / 2);
+        m_fitting_metrics.push_back(
+            fitting < m_distance_metric->patch_size() ? m_distance_metric->with_patch_size(fitting) : nullptr);
     }
     if (m_distance_metric->patch_size() < kCoarsePatchSize)
         m_coarse_metric = m_distance_metric->with_patch_size(kCoarsePatchSize);
@@ -431,12 +666,12 @@ cv::Mat Inpainting::run(bool verbose, unsigned int random_seed)
 MaskedImage Inpainting::_run_levels(
     int from, int to, MaskedImage target, const PatchDistanceMetric *metric, bool verbose)
 {
-    m_level_metric = metric;
     for (int level = from; level >= to; --level)
     {
         trace(verbose, "Inpainting level: ", level);
 
         const MaskedImage &source = m_pyramid[level];
+        m_level_metric = m_fitting_metrics[level] ? m_fitting_metrics[level].get() : metric;
         const bool coarsest = level == static_cast<int>(m_pyramid.size()) - 1;
         if (coarsest)
         {
@@ -448,10 +683,10 @@ MaskedImage Inpainting::_run_levels(
         if (level < 2)
             m_source2target.reset();
         else
-            m_source2target = coarsest ? NearestNeighborField(source, target, metric)
-                                       : NearestNeighborField(source, target, metric, *m_source2target);
-        m_target2source = coarsest ? NearestNeighborField(target, source, metric)
-                                   : NearestNeighborField(target, source, metric, m_target2source);
+            m_source2target = coarsest ? NearestNeighborField(source, target, m_level_metric)
+                                       : NearestNeighborField(source, target, m_level_metric, *m_source2target);
+        m_target2source = coarsest ? NearestNeighborField(target, source, m_level_metric)
+                                   : NearestNeighborField(target, source, m_level_metric, m_target2source);
 
         trace(verbose, "Initialization done.");
 
@@ -559,14 +794,10 @@ MaskedImage Inpainting::_expectation_maximization(
             new_target = target.clone();
         }
 
-        auto vote = cv::Mat(new_target.size(), CV_64FC4);
-        vote.setTo(cv::Scalar::all(0));
-
         // The weighted mean of all overlapping patches blurs the fill, so in the last
         // iteration of the finest level each pixel takes the value of its most similar
         // patch instead (similar to "Space-Time Completion of Video", which takes the
-        // mode of the votes). On coarser levels the mean is kept: it lays out the smooth
-        // structure that the finer levels refine, the best vote there makes seams.
+        // mode of the votes). On coarser levels the best vote makes seams.
         const bool best_only = level == 0 && iter_em == nr_iters_em - 1;
 
         // Known pixels are kept on the two finest levels only. On coarser levels those
@@ -576,6 +807,12 @@ MaskedImage Inpainting::_expectation_maximization(
         const int new_level = upscaled ? level - 1 : level;
         const bool keep_known = new_level <= 1;
 
+        // The other votes for these levels average only their low frequencies and add the
+        // detail of the best vote: with the plain mean the fill stayed smooth, and smooth
+        // fills match smooth sources best. On coarser levels the mean lays out the structure.
+        const bool split = keep_known && !best_only;
+        auto vote = cv::Mat(new_target.size(), CV_64FC(split ? kSplitChannels : 4), cv::Scalar::all(0));
+
         // Votes for best patch from NNF Source->Target (completeness) and Target->Source (coherence).
         if (m_source2target)
         {
@@ -588,6 +825,11 @@ MaskedImage Inpainting::_expectation_maximization(
         // Compile votes and update pixel values.
         _maximization_step(new_target, vote, new_source, keep_known);
         trace(verbose, "  Minimization step finished.");
+
+        // The fill expects the texture of its level: the known one, or that around the holes. On level 0
+        // that of the sources chosen on level 1: with the texture around the holes, the search there took
+        // blurred sources beside detailed surroundings, and parts of the fill stayed flat.
+        new_target.set_texture(upscaled && level == 1 ? voted_texture(new_source, vote) : new_source.texture());
     }
 
     return new_target;
@@ -628,7 +870,7 @@ void Inpainting::_expectation_step(
     const VoteCaster caster(
         nnf, source2target, vote, source, upscaled, best_only, holes.has_value() ? &holes.value() : nullptr);
     // The larger patches of m_coarse_metric would average more votes, which blurred the fills.
-    const int patch_size = m_distance_metric->patch_size();
+    const int patch_size = std::min(m_distance_metric->patch_size(), m_level_metric->patch_size());
     const auto &kDistance2Similarity = distance2similarity();
     const auto source_size = nnf.source_size();
     for (int i = 0; i < source_size.height; ++i)
@@ -645,12 +887,15 @@ void Inpainting::_expectation_step(
 void Inpainting::_maximization_step(
     MaskedImage &target, const cv::Mat &vote, const MaskedImage &source, bool keep_known) const
 {
+    constexpr std::array<double, 3> kNoDetail = {0, 0, 0};
+    const bool split = vote.channels() == kSplitChannels;
     auto target_size = target.size();
     for (int i = 0; i < target_size.height; ++i)
     {
         for (int j = 0; j < target_size.width; ++j)
         {
             const double *source_ptr = vote.ptr<double>(i, j);
+            const double *detail = split ? source_ptr + 4 : kNoDetail.data();
             unsigned char *target_ptr = target.get_mutable_image(i, j);
 
             if (target.is_globally_masked(i, j))
@@ -671,9 +916,9 @@ void Inpainting::_maximization_step(
             if (source_ptr[3] > 0)
             {
                 // All three before the stores: they may alias the votes, which changes the fast-math divisions.
-                const auto r = cv::saturate_cast<unsigned char>(source_ptr[0] / source_ptr[3]);
-                const auto g = cv::saturate_cast<unsigned char>(source_ptr[1] / source_ptr[3]);
-                const auto b = cv::saturate_cast<unsigned char>(source_ptr[2] / source_ptr[3]);
+                const auto r = cv::saturate_cast<unsigned char>(source_ptr[0] / source_ptr[3] + detail[0]);
+                const auto g = cv::saturate_cast<unsigned char>(source_ptr[1] / source_ptr[3] + detail[1]);
+                const auto b = cv::saturate_cast<unsigned char>(source_ptr[2] / source_ptr[3] + detail[2]);
                 target_ptr[0] = r;
                 target_ptr[1] = g;
                 target_ptr[2] = b;

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/patchmatch/csrc/inpaint.h

@@ -35,7 +35,9 @@ private:
     const PatchDistanceMetric *m_distance_metric;
     // Searches on the levels that lay out the fill, if patch_size is smaller than kCoarsePatchSize.
     std::unique_ptr<PatchDistanceMetric> m_coarse_metric;
+    // Per level: smaller patches where patch_size does not fit, otherwise nullptr.
+    std::vector<std::unique_ptr<PatchDistanceMetric>> m_fitting_metrics;
     int m_coarsest_hole_level = -1;
-    // Searches on the current level; the votes use patch_size.
+    // Searches on the current level; the votes use the smaller of its patch size and patch_size.
     const PatchDistanceMetric *m_level_metric = nullptr;
 };

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/patchmatch/csrc/masked_image.cpp

@@ -152,16 +152,15 @@ MaskedImage MaskedImage::upsample(int new_w, int new_h, const cv::Mat &new_globa
     return ret;
 }
 
-void MaskedImage::compute_image_gradients() const
+void MaskedImage::compute_features() const
 {
-    if (m_image_grad_computed)
+    if (m_features_computed)
     {
         return;
     }
 
     const auto size = m_image.size();
-    m_image_grady = cv::Mat(size, CV_8UC3);
-    m_image_gradx = cv::Mat(size, CV_8UC3);
+    m_features = cv::Mat(size, CV_8UC(kFeatureChannels));
 
     // The border replicates its pixels, so that patches over the outermost line compare it as well.
     for (int y = 0; y < size.height; ++y)
@@ -179,15 +178,20 @@ void MaskedImage::compute_image_gradients() const
             const unsigned char *above = get_image(up, x);
             const unsigned char *next = get_image(y, right);
             const unsigned char *previous = get_image(y, left);
-            auto *grady = m_image_grady.ptr<unsigned char>(y, x);
-            auto *gradx = m_image_gradx.ptr<unsigned char>(y, x);
+            auto *features = m_features.ptr<unsigned char>(y, x);
+            std::fill_n(features, kFeatureChannels, 0);
+            const unsigned char *color = get_image(y, x);
             for (int c = 0; c < 3; ++c)
             {
-                grady[c] = static_cast<unsigned char>(skip_y ? 128 : below[c] / 2 - above[c] / 2 + 128);
-                gradx[c] = static_cast<unsigned char>(skip_x ? 128 : next[c] / 2 - previous[c] / 2 + 128);
+                features[c] = color[c];
+                features[3 + c] = static_cast<unsigned char>(skip_x ? 128 : next[c] / 2 - previous[c] / 2 + 128);
+                features[6 + c] = static_cast<unsigned char>(skip_y ? 128 : below[c] / 2 - above[c] / 2 + 128);
             }
+            if (m_has_texture)
+                features[kTextureFeature] = m_texture.at<unsigned char>(y, x);
+            features[kMaskedFeature] = static_cast<unsigned char>(is_masked(y, x) || is_globally_masked(y, x));
         }
     }
 
-    m_image_grad_computed = true;
+    m_features_computed = true;
 }

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/patchmatch/csrc/masked_image.h

@@ -24,10 +24,12 @@ public:
     MaskedImage &operator=(MaskedImage &&) noexcept = default;
     ~MaskedImage() = default;
 
-    // Without the gradients, which would be outdated once the clone is changed.
+    // Without the gradients, which would be outdated once the clone is changed. It shares the texture.
     MaskedImage clone() const
     {
-        return MaskedImage(m_image.clone(), m_mask.clone(), m_global_mask.clone());
+        MaskedImage copy(m_image.clone(), m_mask.clone(), m_global_mask.clone());
+        copy.set_texture(m_texture);
+        return copy;
     }
 
     cv::Size size() const
@@ -50,15 +52,33 @@ public:
     {
         return m_has_global_mask;
     }
-    const cv::Mat &grady() const
+    // Per pixel the colors, the gradients along x and y, the texture and whether it is masked, packed so that
+    // a distance reads one place per pixel.
+    static constexpr int kFeatureChannels = 12;
+    static constexpr int kColorAndGradientFeatures = 9;
+    static constexpr int kTextureFeature = 9;
+    static constexpr int kMaskedFeature = 10;
+    const cv::Mat &features() const
     {
-        assert(m_image_grad_computed);
-        return m_image_grady;
+        assert(m_features_computed);
+        return m_features;
     }
-    const cv::Mat &gradx() const
+
+    // The texture energy that the distances compare (CV_8UC1, see Inpainting), or empty.
+    // It is never changed, and the resampling does not carry it over.
+    const cv::Mat &texture() const
+    {
+        return m_texture;
+    }
+    bool has_texture() const
+    {
+        return m_has_texture;
+    }
+    void set_texture(const cv::Mat &texture)
     {
-        assert(m_image_grad_computed);
-        return m_image_gradx;
+        m_texture = texture;
+        m_has_texture = !texture.empty();
+        m_features_computed = false;
     }
 
     void init_global_mask_mat()
@@ -67,12 +87,14 @@ public:
         m_global_mask.setTo(cv::Scalar(0));
         m_has_global_mask = !m_global_mask.empty();
         m_hole_sums_computed = false;
+        m_features_computed = false;
     }
     void set_global_mask_mat(const cv::Mat &other)
     {
         m_global_mask = other;
         m_has_global_mask = !other.empty();
         m_hole_sums_computed = false;
+        m_features_computed = false;
     }
 
     bool is_masked(int y, int x) const
@@ -87,16 +109,19 @@ public:
     {
         m_mask.at<unsigned char>(y, x) = static_cast<unsigned char>(value);
         m_hole_sums_computed = false;
+        m_features_computed = false;
     }
     void set_global_mask(int y, int x, bool value)
     {
         m_global_mask.at<unsigned char>(y, x) = static_cast<unsigned char>(value);
         m_hole_sums_computed = false;
+        m_features_computed = false;
     }
     void clear_mask()
     {
         m_mask.setTo(cv::Scalar(0));
         m_hole_sums_computed = false;
+        m_features_computed = false;
     }
 
     const unsigned char *get_image(int y, int x) const
@@ -112,18 +137,19 @@ public:
     MaskedImage downsample() const;
     MaskedImage upsample(int new_w, int new_h) const;
     MaskedImage upsample(int new_w, int new_h, const cv::Mat &new_global_mask) const;
-    void compute_image_gradients() const;
+    void compute_features() const;
 
 private:
     cv::Mat m_image;
     cv::Mat m_mask;
     cv::Mat m_global_mask;
+    cv::Mat m_texture;
     // cv::Mat::empty() is not inline in OpenCV 5, too slow to call for every pixel
     bool m_has_global_mask = false;
-    // computed on first use by compute_image_gradients
-    mutable cv::Mat m_image_grady;
-    mutable cv::Mat m_image_gradx;
-    mutable bool m_image_grad_computed = false;
+    bool m_has_texture = false;
+    // computed on first use by compute_features
+    mutable cv::Mat m_features;
+    mutable bool m_features_computed = false;
     // computed on first use by contains_mask: the masked pixels outside the global mask
     // above and left of each pixel. Copies share them with the masks, so change a mask only
     // through an image that nothing has copied yet, as downsample and upsample do.

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/patchmatch/csrc/nnf.cpp

@@ -3,6 +3,7 @@
 #include <algorithm>
 #include <cassert>
 #include <cmath>
+#include <cstddef>
 #include <cstdint>
 #include <random>
 #include <typeinfo>
@@ -52,7 +53,7 @@ namespace
     {
         ImagePair(const MaskedImage &source_image, const MaskedImage &target_image)
             : source(&source_image), target(&target_image), source_size(source_image.size()),
-              target_size(target_image.size())
+              target_size(target_image.size()), textured(source_image.has_texture() && target_image.has_texture())
         {
         }
 
@@ -65,60 +66,47 @@ namespace
         const MaskedImage *target;
         cv::Size source_size;
         cv::Size target_size;
+        bool textured;
     };
 
-    // One row of a source and a target image, both inside their images. The gradients must be computed.
+    // One row of a source and a target image, both inside their images. The features must be computed.
     class RowPair
     {
     public:
         RowPair(const ImagePair &images, int ys, int yt)
             : m_source_width(images.source_size.width), m_target_width(images.target_size.width),
-              m_si(images.source->image().ptr<unsigned char>(ys, 0)),
-              m_ti(images.target->image().ptr<unsigned char>(yt, 0)),
-              m_sm(images.source->mask().ptr<unsigned char>(ys, 0)),
-              m_tm(images.target->mask().ptr<unsigned char>(yt, 0)),
-              m_sgm(
-                  images.source->has_global_mask() ? images.source->global_mask().ptr<unsigned char>(ys, 0) : nullptr),
-              m_tgm(
-                  images.target->has_global_mask() ? images.target->global_mask().ptr<unsigned char>(yt, 0) : nullptr),
-              m_sgy(images.source->grady().ptr<unsigned char>(ys, 0)),
-              m_tgy(images.target->grady().ptr<unsigned char>(yt, 0)),
-              m_sgx(images.source->gradx().ptr<unsigned char>(ys, 0)),
-              m_tgx(images.target->gradx().ptr<unsigned char>(yt, 0))
+              m_source(images.source->features().ptr<unsigned char>(ys)),
+              m_target(images.target->features().ptr<unsigned char>(yt)), m_textured(images.textured)
         {
         }
 
         // The SSD over the colors and gradients, kSSDScale where a pixel is masked or outside its image.
         int cost(int xs, int xt) const
         {
-            // The bounds first: the masks are read only inside the image.
-            if (xs < 0 || xs >= m_source_width || xt < 0 || xt >= m_target_width || m_sm[xs] || m_tm[xt] ||
-                (m_sgm && m_sgm[xs]) || (m_tgm && m_tgm[xt]))
+            // The bounds first: the features are read only inside the image.
+            if (xs < 0 || xs >= m_source_width || xt < 0 || xt >= m_target_width)
+                return PatchSSDDistanceMetric::kSSDScale;
+            const unsigned char *s = m_source + static_cast<std::ptrdiff_t>(xs) * MaskedImage::kFeatureChannels;
+            const unsigned char *t = m_target + static_cast<std::ptrdiff_t>(xt) * MaskedImage::kFeatureChannels;
+            if (s[MaskedImage::kMaskedFeature] || t[MaskedImage::kMaskedFeature])
                 return PatchSSDDistanceMetric::kSSDScale;
 
             int ssd = 0;
-            for (int c = 0; c < 3; ++c)
-            {
-                ssd += pow2(m_si[xs * 3 + c] - m_ti[xt * 3 + c]);
-                ssd += pow2(m_sgx[xs * 3 + c] - m_tgx[xt * 3 + c]);
-                ssd += pow2(m_sgy[xs * 3 + c] - m_tgy[xt * 3 + c]);
-            }
+            for (int k = 0; k < MaskedImage::kColorAndGradientFeatures; ++k)
+                ssd += pow2(s[k] - t[k]);
+            // Smooth fills matched smooth sources and stayed flat, so a target pixel with less than half the
+            // texture of its source pixel costs extra. Not a two-sided difference: it removed mortar joints.
+            if (m_textured)
+                ssd += 2 * pow2(std::max(s[MaskedImage::kTextureFeature] / 2 - t[MaskedImage::kTextureFeature], 0));
             return ssd;
         }
 
     private:
         int m_source_width;
         int m_target_width;
-        const unsigned char *m_si;
-        const unsigned char *m_ti;
-        const unsigned char *m_sm;
-        const unsigned char *m_tm;
-        const unsigned char *m_sgm;
-        const unsigned char *m_tgm;
-        const unsigned char *m_sgy;
-        const unsigned char *m_tgy;
-        const unsigned char *m_sgx;
-        const unsigned char *m_tgx;
+        const unsigned char *m_source;
+        const unsigned char *m_target;
+        bool m_textured; // false unless both images have a texture
     };
 
     // The costs of the row of the patches around s and t.
@@ -159,8 +147,8 @@ namespace
         const double limit =
             (bound + 1.0) * PatchSSDDistanceMetric::kSSDScale * side * side / PatchDistanceMetric::kDistanceScale;
 
-        source.compute_image_gradients();
-        target.compute_image_gradients();
+        source.compute_features();
+        target.compute_features();
 
         const ImagePair images(source, target);
         for (int dy = -patch_size; dy <= patch_size; ++dy)

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/pyproject.toml

@@ -44,6 +44,7 @@ Issues = "https://github.com/mauwii/PyPatchMatch/issues"
 coverage = ["gcovr"]
 # numpy and pillow for mypy, which runs without the project (--only-group dev)
 dev = [{ include-group = "test" }, "mypy", "numpy", "pillow", "pre-commit", "ruff"]
+gui = ["gradio"]
 test = ["hypothesis", "pytest", "pytest-cov"]
 
 [tool.scikit-build]
@@ -59,11 +60,13 @@ metadata.readme.provider = "scikit_build_core.metadata.fancy_pypi_readme"
 # The release wheels are built from the sdist, so it has to contain the tests
 # and their input images; _version.py is generated and therefore git-ignored.
 sdist.include = [
+    "examples/images/camera_damaged.png",
+    "examples/images/camera_damaged_mask.png",
     "examples/images/chelsea.png",
     "examples/images/forest_pruned.bmp",
     "patchmatch/_version.py",
 ]
-sdist.exclude = [".*", "examples", "sonar-project.properties", "uv.lock"]
+sdist.exclude = [".*", "examples", "gui", "sonar-project.properties", "uv.lock"]
 
 # Without a matching wheel, pip builds the sdist, which needs a C++ compiler, CMake and
 # OpenCV. If that fails, build a wheel without the native library instead of failing

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/scripts/evaluate_inpainting.py

@@ -3,13 +3,16 @@
 The holes are cut into images whose content is known: forest.bmp and the images listed
 in examples/images/SOURCES.md. forest_pruned.bmp and every ``*_pruned.*`` image in the
 git-ignored examples/images/local/ are added as object removals without a ground
-truth; their pure white pixels are the holes. grass-edge cuts a band at the right
-border, as in outpainting; brick-large and grass-large cut a circle of radius 242 into
-the center, 70 % of the image. pattern repeats almost exactly, so its error tells a fill
-that continues it (2 to 6) from a broken one; pattern-wide widens the hole to a radius
-of 60, where patch size 3 mixed the rings before 2.2 (error 37). forest-global and
-brick-global add a global mask, darkened in the report: the plant of
-examples/py_example_global_mask.py and the left third of brick.png.
+truth; their pure white pixels are the holes. camera restores the damage of
+camera_damaged.png, which camera_damaged_mask.png marks, against camera_aged.png; its
+error covers the larger holes and the torn corner, the scratches are narrower than the
+cells it compares. grass-edge cuts a band at the right border, as in outpainting;
+brick-large and grass-large cut a circle of radius 242 into the center, 70 % of the
+image. pattern repeats almost exactly, so its error tells a fill that continues it (2 to
+6) from a broken one; pattern-wide widens the hole to a radius of 60, where patch size 3
+mixed the rings before 2.2 (error 37). forest-global and brick-global add a global mask,
+darkened in the report: the plant of examples/py_example_global_mask.py and the left
+third of brick.png.
 
 Usage:
     uv run python scripts/evaluate_inpainting.py
@@ -20,8 +23,8 @@ Usage:
 workflow "Fill quality" redirects it to the job summary.
 
 Besides the table, it writes examples/images/evaluation.html (git-ignored): the holes,
-the fills (of --baseline too) and the originals side by side. It embeds the images of
-examples/images/local/ as well, so keep it out of the repository.
+the fills (of --baseline too) and the originals side by side, cut to the holes. It
+embeds the images of examples/images/local/ as well, so keep it out of the repository.
 
 The e2e tests use measure() to check the seam and the detail of forest_pruned.bmp.
 
@@ -31,7 +34,11 @@ Columns, means over the seeds:
     detail  color difference of neighbors inside the holes, relative to their
             surroundings; 0.85 to 1.1 in the original images, lower is blurrier
     error   mean color error against the original on images reduced to 1/8, which
-            compares the structure of the fill rather than its pixels
+            compares the structure of the fill rather than its pixels; it rates a flat
+            fill better than a texture whose structures are shifted
+    edges   difference of the gradient directions from the original, in histograms
+            weighted by magnitude per 32x32 cell of the holes; lowest for the right
+            texture, also when shifted, highest for a flat fill
     fills   with --baseline: whether the fills of all seeds are byte-identical to the
             baseline's, as they have to be after a change that only makes it faster
 """
@@ -70,7 +77,7 @@ HOLES = {
     "chelsea": (52, 215, 24, 40),
     "pattern": (320, 320, 40, 40),
 }
-COLUMNS = ("seam", "detail", "error")
+COLUMNS = ("seam", "detail", "error", "edges")
 
 
 class Case(NamedTuple):
@@ -116,6 +123,9 @@ def cases() -> Iterator[Case]:
     forest = load(IMAGES / "forest.bmp")
     pruned = load(IMAGES / "forest_pruned.bmp")
     yield Case("forest", pruned, white(pruned), None)
+    damaged = load(IMAGES / "camera_damaged.png")
+    damage = np.array(Image.open(IMAGES / "camera_damaged_mask.png")) > 0
+    yield Case("camera", damaged, damage, load(IMAGES / "camera_aged.png"))
     yield cut("trees", forest, ellipse(forest.shape, 70, 560, 45, 55))
     meadow = np.zeros(forest.shape[:2], dtype=bool)
     meadow[318:350, 380:470] = True
@@ -176,6 +186,25 @@ def coarse_error(image: np.ndarray, truth: np.ndarray, holes: np.ndarray) -> flo
     return float(np.abs(reduce(image)[cells] - reduce(truth)[cells]).mean())
 
 
+EDGE_CELL = 32
+EDGE_BINS = 8
+
+
+def edge_error(image: np.ndarray, truth: np.ndarray, holes: np.ndarray) -> float:
+    """Difference of the gradient directions, weighted by magnitude, per cell of holes."""
+    ys, xs = np.nonzero(holes)
+    cells = (ys // EDGE_CELL) * (holes.shape[1] // EDGE_CELL + 1) + xs // EDGE_CELL
+    length = (int(cells.max()) + 1) * EDGE_BINS
+
+    def histograms(array: np.ndarray) -> np.ndarray:
+        gy, gx = np.gradient(array.astype(float).mean(axis=2))
+        bins = (np.arctan2(gy, gx) % np.pi * EDGE_BINS / np.pi).astype(int) % EDGE_BINS
+        weights = np.hypot(gx, gy)[holes]
+        return np.bincount(cells * EDGE_BINS + bins[holes], weights, length)
+
+    return float(np.abs(histograms(image) - histograms(truth)).sum() / len(ys))
+
+
 def measure(
     result: np.ndarray, holes: np.ndarray, truth: np.ndarray | None = None
 ) -> dict[str, float]:
@@ -196,6 +225,7 @@ def measure(
     }
     if truth is not None:
         values["error"] = coarse_error(result, truth, holes)
+        values["edges"] = edge_error(result, truth, holes)
     return values
 
 
@@ -280,19 +310,13 @@ figure { margin: 0; cursor: zoom-in; }
 figcaption { font-size: 13px; color: var(--muted); margin-top: 4px;
   font-variant-numeric: tabular-nums; }
 figcaption b { color: var(--fg); font-weight: 600; margin-right: 6px; }
-.frame { position: relative; overflow: hidden; width: 100%;
-  aspect-ratio: var(--crop); background: var(--line); }
-.frame img { position: absolute; display: block; max-width: none; height: auto;
-  left: var(--left); top: var(--top); width: var(--width); }
-body.whole .frame { aspect-ratio: var(--full); }
-body.whole .frame img { left: 0; top: 0; width: 100%; }
+figure img { display: block; width: 100%; height: auto; }
 body.pixels img { image-rendering: pixelated; }
 #viewer { position: fixed; inset: 0; z-index: 10; display: flex;
   flex-direction: column; align-items: center; justify-content: center; gap: 10px;
   background: rgb(0 0 0 / 0.9); cursor: zoom-out; }
 #viewer[hidden] { display: none; }
-#viewer .frame { width: min(94vw, calc(86vh * var(--crop))); cursor: pointer; }
-body.whole #viewer .frame { width: min(94vw, calc(86vh * var(--full))); }
+#viewer img { width: 94vw; height: 86vh; object-fit: contain; cursor: pointer; }
 #viewer p { color: #ddd; font-size: 14px; margin: 0; }
 """
 
@@ -304,7 +328,7 @@ function show(i) {
   index = (i + figures.length) % figures.length;
   const figure = figures[index];
   viewer.querySelector(".slot").replaceChildren(
-    figure.querySelector(".frame").cloneNode(true));
+    figure.querySelector("img").cloneNode());
   viewer.querySelector("p").textContent = figure.dataset.title
     + "  (" + (index + 1) + "/" + figures.length + ")";
 }
@@ -316,7 +340,7 @@ for (const figure of document.querySelectorAll("figure")) {
   });
 }
 viewer.addEventListener("click", (event) => {
-  if (event.target.closest(".frame")) show(index + 1);
+  if (event.target.closest("img")) show(index + 1);
   else viewer.hidden = true;
 });
 document.addEventListener("keydown", (event) => {
@@ -327,12 +351,10 @@ document.addEventListener("keydown", (event) => {
   else return;
   event.preventDefault();
 });
-for (const name of ["whole", "pixels"]) {
-  const box = document.getElementById(name);
-  const apply = () => document.body.classList.toggle(name, box.checked);
-  box.addEventListener("change", apply);
-  apply();  // browsers restore the checkboxes on reload
-}
+const pixels = document.getElementById("pixels");
+const apply = () => document.body.classList.toggle("pixels", pixels.checked);
+pixels.addEventListener("change", apply);
+apply();  // browsers restore the checkbox on reload
 """
 
 
@@ -356,18 +378,6 @@ def crop_box(holes: np.ndarray) -> tuple[int, int, int, int]:
     return int(x0), int(y0), int(x1 - x0), int(y1 - y0)
 
 
-def frame(image: np.ndarray, box: tuple[int, int, int, int]) -> str:
-    """The image, cut to ``box`` by CSS unless the whole images are shown."""
-    height, width = image.shape[:2]
-    x, y, w, h = box
-    style = (
-        f"--crop:{w / h:.5f};--full:{width / height:.5f};--width:{100 * width / w:.4f}%;"
-        f"--left:{-100 * x / w:.4f}%;--top:{-100 * y / h:.4f}%"
-    )
-    source = f"data:image/png;base64,{png(image)}"
-    return f'<div class="frame" style="{style}"><img alt="" src="{source}"></div>'
-
-
 def verdict(key: str, old: float, new: float) -> str:
     """CSS class of ``new`` compared with ``old``: better, worse or none."""
     if abs(new - old) <= (0.1 if key == "time" else 0.02) * abs(old):
@@ -406,7 +416,7 @@ def caption(label: str, result: Result | None) -> str:
 
 
 def section(case: Case, results: list[Result], labels: list[str]) -> str:
-    box = crop_box(case.holes)
+    x, y, w, h = crop_box(case.holes)
     shown = case.image.copy()
     if case.global_mask is not None:
         shown[case.global_mask] //= 2
@@ -416,7 +426,8 @@ def section(case: Case, results: list[Result], labels: list[str]) -> str:
         versions.append(("original", case.truth, None))
     figures = "".join(
         f'<figure data-title="{html.escape(case.name)}: {label}">'
-        f"{frame(image, box)}<figcaption>{caption(label, result)}</figcaption></figure>"
+        f'<img alt="" src="data:image/png;base64,{png(image[y : y + h, x : x + w])}">'
+        f"<figcaption>{caption(label, result)}</figcaption></figure>"
         for label, image, result in versions
     )
     height, width = case.holes.shape
@@ -450,7 +461,6 @@ def write_report(
             f"<title>patchmatch evaluation</title><style>{STYLE}</style></head><body>"
             f"<main><header><h1>patchmatch evaluation</h1>{meta}</header>"
             '<div class="controls">'
-            '<label><input type="checkbox" id="whole"> whole images</label>'
             '<label><input type="checkbox" id="pixels"> show pixels</label>'
             '<p class="hint">Click an image to enlarge it; click it, &larr; or &rarr; '
             "switches between the versions, Esc closes.</p></div>"
@@ -493,8 +503,8 @@ def markdown(rows: list[tuple[Case, list[Result]]], title: str) -> str:
     lines += [
         "",
         "Means over the seeds. seam and detail are best at 1.0, as in the original "
-        "images, error at 0. With a baseline: baseline → current, 🟢 better and 🔴 worse "
-        "by more than 2 % (time: 10 %).",
+        "images, error and edges at 0. With a baseline: baseline → current, 🟢 better "
+        "and 🔴 worse by more than 2 % (time: 10 %).",
     ]
     return "\n".join(lines)
 
@@ -536,7 +546,7 @@ def main() -> None:
     out = sys.stderr if args.markdown else sys.stdout
     print(f"patch_size={args.patch_size} seeds={args.seeds}", file=out)
     fills = "fills" if args.baseline else ""
-    header = f"{'case':14} {'seam':16}{'detail':16}{'error':16}{'time':16}{fills}"
+    header = f"{'case':14} {''.join(f'{key:16}' for key in COLUMNS)}{'time':16}{fills}"
     print(header.rstrip(), file=out)
     rows = []
     for case in cases():

{pypatchmatch-2.2.0b1 → pypatchmatch-2.2.0b3}/tests/test_e2e.py

@@ -36,15 +36,21 @@ def surrounding(mask: np.ndarray, width: int = 10) -> np.ndarray:
 
 
 def assert_plausible_fill(
-    source: np.ndarray, result: np.ndarray, excluded: np.ndarray | None = None
+    source: np.ndarray,
+    result: np.ndarray,
+    excluded: np.ndarray | None = None,
+    holes: np.ndarray | None = None,
 ) -> None:
     """Check an inpainting result of ``source`` without a ground truth image.
 
-    ``excluded`` marks the pixels of a global mask, which are not filled.
+    ``excluded`` marks the pixels of a global mask, which are not filled. ``holes``
+    defaults to the white pixels of ``source``.
     """
     if excluded is None:
         excluded = np.zeros(source.shape[:2], dtype=bool)
-    holes = evaluation.white(source) & ~excluded
+    if holes is None:
+        holes = evaluation.white(source)
+    holes = holes & ~excluded
     assert holes.any()
     assert result.shape == source.shape
     assert result.dtype == np.uint8
@@ -130,10 +136,55 @@ def test_global_mask_is_not_used_as_source(pruned_image):
     np.testing.assert_array_equal(result[excluded], source[excluded])
 
 
-def test_fill_is_as_detailed_as_its_surroundings():
-    """The weighted mean of all overlapping patches blurred the fill (detail 0.69)."""
-    source, result = fill("forest_pruned.bmp", None, 3)
-    assert evaluation.measure(result, evaluation.white(source))["detail"] > 0.8
+def test_restore_aged_photo():
+    """Scratches, holes and a torn corner, light but not white, given as a mask."""
+    source = Image.open(IMAGES / "camera_damaged.png")
+    mask = Image.open(IMAGES / "camera_damaged_mask.png")
+
+    result = patchmatch.inpaint(source, mask, patch_size=3)
+
+    assert_plausible_fill(np.array(source), result, holes=np.array(mask) > 0)
+
+
+@pytest.mark.parametrize(
+    ("image", "hole", "patch_size", "limit"),
+    [
+        ("forest_pruned.bmp", None, 3, 0.8),
+        ("chelsea.png", evaluation.HOLES["chelsea"], 3, 0.7),
+        ("chelsea.png", evaluation.HOLES["chelsea"], 15, 0.4),
+    ],
+)
+def test_fill_is_as_detailed_as_its_surroundings(image, hole, patch_size, limit):
+    """The weighted mean of all overlapping patches blurred the fill (detail 0.69).
+
+    On the finer levels it kept the fill of the fur in chelsea smooth, which then
+    matched smooth sources best (detail 0.40). With patch size 15, the coarser levels
+    already took smooth sources, which look like fur once downsampled (detail 0.29).
+    """
+    source, result = fill(image, hole, patch_size)
+    assert evaluation.measure(result, evaluation.white(source))["detail"] > limit
+
+
+def test_no_part_of_the_fill_is_flat():
+    """The finest level linked the left of the hole in chelsea to the blurred fur at
+    the border of the image. That part stayed flat (detail 0.30 in the flattest 16x16
+    block) while the whole fill reached 0.70.
+    """
+    source, result = fill("chelsea.png", evaluation.HOLES["chelsea"], 7)
+    holes = evaluation.white(source)
+    inside = holes & ~evaluation.grow(~holes, 2)
+    around = evaluation.grow(holes, 24) & ~evaluation.grow(holes, 2)
+    ys, xs = np.nonzero(inside)
+    details = []
+    for y in range(ys.min(), ys.max() + 1, 16):
+        for x in range(xs.min(), xs.max() + 1, 16):
+            block = np.zeros_like(inside)
+            block[y : y + 16, x : x + 16] = True
+            block &= inside
+            if block.sum() > 128:
+                details.append(evaluation.neighbor_difference(result, block, block))
+    reference = evaluation.neighbor_difference(result, around, around)
+    assert min(details) / reference > 0.5
 
 
 @pytest.mark.parametrize(
@@ -189,13 +240,22 @@ def test_explicit_mask_matches_implicit_white_mask(pruned_image):
     np.testing.assert_array_equal(explicit, fill("forest_pruned.bmp", None, 3)[1])
 
 
-# Regions of pure background in forest_pruned.bmp as (y, x, height, width).
-BACKGROUND_REGIONS = [(20, 400, 60, 80), (40, 520, 60, 60)]
+# Regions of pure background in forest_pruned.bmp as (y, x, height, width), with the
+# patch size to fill them.
+BACKGROUND_REGIONS = [
+    ((20, 400, 60, 80), 3),
+    ((40, 520, 60, 60), 3),
+    ((25, 505, 90, 110), 15),
+]
 
 
-@pytest.mark.parametrize("region", BACKGROUND_REGIONS)
-def test_reconstructs_known_background(pruned_image, region):
-    """Cut a hole into known background and compare with the original pixels."""
+@pytest.mark.parametrize(("region", "patch_size"), BACKGROUND_REGIONS)
+def test_reconstructs_known_background(pruned_image, region, patch_size):
+    """Cut a hole into known background and compare with the original pixels.
+
+    With patch size 15, the coarser levels found no sources inside the image and
+    outside the hole, and the trees got a lighter patch (0.90 of the naive error).
+    """
     truth = np.array(pruned_image)
     y, x, height, width = region
     mask = np.zeros(truth.shape[:2], dtype=bool)
@@ -204,7 +264,7 @@ def test_reconstructs_known_background(pruned_image, region):
 
     source = truth.copy()
     source[mask] = 255
-    result = patchmatch.inpaint(source, mask.astype(np.uint8), patch_size=3)
+    result = patchmatch.inpaint(source, mask.astype(np.uint8), patch_size=patch_size)
 
     # baseline: fill the hole with the mean color of its surroundings
     naive = source.copy()