structN2V¶
Overview¶
structN2V [1] is a self-supervised denoising method based on Noise2Void [2,3]. The method was developed to remove pixel-noises from images, even in the presence of spatial noise correlations.
It retains the same training scheme (i.e. training on noisy images by randomly masking pixels), and differs from Noise2Void by the use of an additional mask covering the extent of the spatial noise correlation.
Pixel-wise independent noise¶
It might happen that the noise in your images is not pixel-wise independent. In this case, the network can learn the amount of noise in the masked pixels from its neighboring pixels. Noise2Void might then introduce small artefacts or simply reinforce the correlated pattern in the denoised image.
Correlations in the noise can sometimes be observed with sCMOS camera or point-scanning microscopy methods. To estimate whether the noise is correlated, one can perform image autocorrelation and inspect the shape of the central peak in the distribution. The next figure shows two examples of the same image with artificial Gaussian noise. In the second one, we introduced a correlation in the Gaussian noise, leading to an horizontal line in the autocorrelation image.
Fig 1.: Structured noise. Top row: image with uncorrelated noise (left) and its autocorrelation (close-up on the zero-shift component). Bottom row: same images with correlations introduced in the noise component. CC-BY.
N2V vs structN2V masking¶
In order to obfuscate the pixel values whose noise might be correlated, structN2V introduces a mask spanning the extent of the correlation. In order to estimate the size and direction of the correlation, one simply needs to examine the autocorrelation image (see section above).
However, as in Noise2Void, masking pixels by setting their values to zero will confuse the network by introducing unrealistic values. To circumvent this issue, structN2V replaces the masked pixels by values uniformly sampled between the minimum and maximum pixel values of the patch.
Fig 2.: Noise2Void vs structN2V manipulation, with a toy example (top row) and a real-world one (bottom row). In both algorithms, randomly selected pixels in the input patch (left) are selected and their value replaced. Noise2Void (center) replaces the values by the value of one of the neighboring pixel, while structN2V (right) additionally masks neighboring pixels using a uniformly sampled value comprised between the min and max patch value to prevent the network from learning from the noise correlations. CC-BY.
References¶
[1] Coleman Broaddus, Alexander Krull, Martin Weigert, Uwe Schmidt, and Gene Myers. " Removing structured noise with self-supervised blind-spot networks." ISBI, 2020. link
[2] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. "Noise2Void - learning denoising from single noisy images." CVPR, 2019. link
[3] Joshua Batson, and Loic Royer. "Noise2Self: Blind denoising by self-supervision." MLR, 2019. link