Browse > Article
http://dx.doi.org/10.1186/s42649-020-00041-8

Noise2Atom: unsupervised denoising for scanning transmission electron microscopy images  

Feng Wang (Electron Microscopy Center, Empa, Swiss Federal Laboratories for Materials Science and Technology)
Trond R. Henninen (Electron Microscopy Center, Empa, Swiss Federal Laboratories for Materials Science and Technology)
Debora Keller (Electron Microscopy Center, Empa, Swiss Federal Laboratories for Materials Science and Technology)
Rolf Erni (Electron Microscopy Center, Empa, Swiss Federal Laboratories for Materials Science and Technology)
Publication Information
Applied Microscopy / v.50, no., 2020 , pp. 23.1-23.9 More about this Journal
Abstract
We propose an effective deep learning model to denoise scanning transmission electron microscopy (STEM) image series, named Noise2Atom, to map images from a source domain 𝓢 to a target domain 𝓒, where 𝓢 is for our noisy experimental dataset, and 𝓒 is for the desired clear atomic images. Noise2Atom uses two external networks to apply additional constraints from the domain knowledge. This model requires no signal prior, no noise model estimation, and no paired training images. The only assumption is that the inputs are acquired with identical experimental configurations. To evaluate the restoration performance of our model, as it is impossible to obtain ground truth for our experimental dataset, we propose consecutive structural similarity (CSS) for image quality assessment, based on the fact that the structures remain much the same as the previous frame(s) within small scan intervals. We demonstrate the superiority of our model by providing evaluation in terms of CSS and visual quality on different experimental datasets.
Keywords
Denoising; STEM images; Deep learning; Unsupervised learning;
Citations & Related Records
연도 인용수 순위
  • Reference
1 D. Keller, T. R. Henninen, R. Erni, Formation of gold nanoparticles in a free-standing ionic liquid triggered by heat and electron irradiation. Micron. 117, 16-21 (2019). Publisher: Elsevier
2 E. J. Kirkland, R. F. Loane, J. Silcox, Simulation of annular dark field STEM images using a modified multislice method. Ultramicroscopy. 23(1), 77-96 (1987). Publisher: Elsevier
3 F. Kokkinos, S. Lefkimmiatis, Iterative Joint Image Demosaicking and Denoising Using a Residual Denoising Network. IEEE Trans. Image Process. 28(8), 4177-4188 (2019). ISSN 1941-0042. https://doi.org/10.1109/TIP.2019.2905991   DOI
4 A. Krizhevsky, I. Sutskever, G. E. Hinton, in Advances in Neural Information Processing Systems, vol. 25, ed. by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger. ImageNet Classification with Deep Convolutional Neural Networks (Curran Associates, Inc., 2012), pp. 1097-1105. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neuralnetwor.Accessed 24 Sept 2020
5 A. Krull, T. Vicar, F. Jug, Probabilistic Noise2Void: Unsupervised Content-Aware Denoising (2019b). arXiv:1906.00651
6 J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, T. Aila, Noise2noise: Learning image restoration without clean data (2018). arXiv preprint arXiv:1803.04189
7 Y. Chang, L. Yan, H. Fang, S. Zhong, W. Liao, HSI-DeNet: Hyperspectral Image Restoration via Convolutional Neural Network. IEEE Trans. Geosci. Remote Sens. 57(2), 667-682 (2019). ISSN 1558-0644. https://doi.org/10.1109/TGRS.2018.2859203   DOI
8 T. R. Henninen, M. Bon, F. Wang, D. Passerone, R. Erni, in Angewandte Chemie International Edition. The Structure of Sub-nm Platinum Clusters at Elevated Temperatures, (2019). ISSN 14337851. https://doi.org/10.1002/anie.201911068   DOI
9 Y. Zhang, Y. Zhu, E. Nichols, Q. Wang, S. Zhang, C. Smith, S. Howard, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. A poisson-gaussian denoising dataset with real fluorescence microscopy images, (2019), pp. 11710-11718. https://doi.org/10.1109/cvpr.2019.01198   DOI
10 K. He, X. Zhang, S. Ren, J. Sun, in Proceedings of the IEEE conference on computer vision and pattern recognition. Deep residual learning for image recognition, (2016), pp. 770-778. https://doi.org/10.1109/cvpr.2016.90   DOI
11 V. S. Kadimesetty, S. Gutta, S. Ganapathy, P. K. Yalavarthy, Convolutional neural network-based robust denoising of low-dose computed tomography perfusion maps. IEEE Trans. Radiat. Plasma Med. Sci. 3(2), 137-152 (2018)
12 F. Wang, A. Eljarrat, J. Muller, T. R. Henninen, R. Erni, C. T. Koch, Multi-resolution convolutional neural networks for inverse problems. Sci. Rep. 10(1), 1-11 (2020). ISSN 2045-2322. https://doi.org/10.1038/s41598-020-62484-z   DOI
13 K. Lin, T. H. Li, S. Liu, G. Li, in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. Real photographs denoising with noise domain adaptation and attentive generative adversarial network, (2019). https://doi.org/10.1109/cvprw.2019.00221   DOI
14 A. Lucas, M. Iliadis, R. Molina, A. K. Katsaggelos, Using deep neural networks for inverse problems in imaging: Beyond analytical methods. IEEE Signal Process. Mag. 35, 20-36 (2018)
15 T. Kim, M. Cha, H. Kim, J. K. Lee, J. Kim, in Proceedings of the 34th International Conference on Machine Learning, vol. 70, ed. by D. Precup, Y.W. Teh. Learning to Discover Cross-Domain Relations with Generative Adversarial Networks (PMLR, International Convention Centre, Sydney, 2017), pp. 1857-1865. http://proceedings.mlr.press/v70/kim17a.html. Accessed 24 Sept 2020
16 O. Ronneberger, P. Fischer, T. Brox, in Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015, Lecture Notes in Computer Science, ed. by N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi. U-Net: Convolutional Networks for Biomedical Image Segmentation (Cham. Springer International Publishing, 2015), pp. 234-241. ISBN 978-3-319-24574-4. https://doi.org/10.1007/978-3-319-24574-4_28   DOI
17 T.-O. Buchholz, A. Krull, R. Shahidi, G. Pigino, G. Jekely, F. Jug, Content-aware image restoration for electron microscopy.Methods Cell Biol. 152, 277-289 (2019b). ISSN 0091-679X. https://doi.org/10.1016/bs.mcb.2019.05.001   DOI
18 M. Prakash, M. Lalit, P. Tomancak, A. Krul, F. Jug, in 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). Fully Unsupervised Probabilistic Noise2Void, (2020), pp. 154-158. https://doi.org/10.1109/ISBI45749.2020.9098612   DOI
19 Y. Song, Y. Zhu, X. Du, Dynamic Residual Dense Network for Image Denoising. Sensors. 19(17), 3809 (2019). https://doi.org/10.3390/s19173809. https://www.mdpi.com/1424-8220/19/17/3809   DOI
20 Z. Yi, H. Zhang, P. Tan, M. Gong, in Proceedings of the IEEE International Conference on Computer Vision. Dualgan: Unsupervised dual learning for image-to-image translation, (2017), pp. 2849-2857. https://doi.org/10.1109/iccv.2017.310   DOI
21 K. Zhang, W. Zuo, L. Zhang, FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising. IEEE Trans. Image Process. 27(9), 4608-4622 (2018). ISSN 1057-7149, 1941-0042. https://doi.org/10.1109/TIP.2018.2839891   DOI
22 J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, in Proceedings of the IEEE International Conference on Computer Vision. Unpaired image-to-image translation using cycle-consistent adversarial networks, (2017), pp. 2223-2232. https://doi.org/10.1109/iccv.2017.244   DOI
23 P. Liu, H. Zhang, K. Zhang, L. Lin, W. Zuo, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Multi-level wavelet-CNN for image restoration, (2018), pp. 773-782. https://doi.org/10.1109/cvprw.2018.00121   DOI
24 F. Chollet, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Xception: Deep Learning with Depthwise Separable Convolutions (IEEE, Honolulu, 2017), pp. 1800-1807. ISBN, 978-1-5386-0457-1. https://doi.org/10.1109/CVPR.2017.195   DOI
25 Z. Guo, Y. Sun, M. Jian, X. Zhang, Deep Residual Network with Sparse Feedback for Image Restoration. Appl. Sci. 8(12), 2417 (2018)
26 T. R. Henninen, D. Keller, R. Erni, Structure matters - Direct in-situ observation of cluster nucleation at atomic scale in a liquid phase. In review (2020)
27 B. Mildenhall, J. T. Barron, J. Chen, D. Sharlet, R. Ng, R. Carroll, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Burst denoising with kernel prediction networks, (2018), pp. 2502-2510. https://doi.org/10.1109/cvpr.2018.00265   DOI
28 N. Moran, D. Schmidt, Y. Zhong, P. Coady, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Noisier2Noise: Learning to Denoise from Unpaired Noisy Data, (2020), pp. 12064-12072. https://doi.org/10.1109/cvpr42600.2020.01208   DOI
29 M. Ran, J. Hu, Y. Chen, H. Chen, H. Sun, J. Zhou, Y. Zhang, Denoising of 3D magnetic resonance images using a residual encoder-decoder Wasserstein generative adversarial network. Med. Image Anal. 55, 165-180 (2019)
30 Y. Su, Q. Lian, X. Zhang, B. Shi, X. Fan, Multi-scale cross-path concatenation residual network for Poisson denoising. IET Image Process. 13(8), 1295-1303 (2019)
31 D. Ulyanov, A. Vedaldi, V. Lempitsky, Instance normalization: The missing ingredient for fast stylization, (2016). arXiv preprint arXiv:1607.08022
32 J. Wu, Z. Huang, J. Thoma, D. Acharya, L. Van Gool, in Proceedings of the European Conference on Computer Vision (ECCV). Wasserstein divergence for gans, (2018), pp. 653-668. https://doi.org/10.1007/978-3-030-01228-1_40   DOI
33 W. Xie, Y. Li, X. Jia, Deep convolutional networks with residual learning for accurate spectral-spatial denoising. Neurocomputing. 312, 372-381 (2018)
34 T. Furnival, R. K. Leary, P. A. Midgley, Denoising time-resolved microscopy image sequences with singular value thresholding. Ultramicroscopy. 178, 112-124 (2017). ISSN 03043991. https://doi.org/10.1016/j.ultramic.2016.05.005   DOI
35 A. Krull, T.-O. Buchholz, F. Jug, in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Noise2void - learning denoising from single noisy images, (2019a). https://doi.org/10.1109/cvpr.2019.00223   DOI
36 M. Arjovsky, S. Chintala, L. Bottou, Wasserstein GAN (2017). arXiv:1701.07875
37 J. Batson, L. Royer, Noise2Self: Blind Denoising by Self-Supervision (2019). arXiv:1901.11365
38 T.-O. Buchholz, M. Jordan, G. Pigino, F. Jug, in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). Cryo-CARE: Content-Aware Image Restoration for Cryo-Transmission Electron Microscopy Data (IEEE, Venice, 2019a), pp. 502-506. ISBN 978-1-5386-3641-1. https://doi.org/10.1109/ISBI.2019.8759519   DOI
39 K. Cao, T. Zoberbier, J. Biskupek, A. Botos, R. L. McSweeney, A. Kurtoglu, C. T. Stoppiello, A. V. Markevich, E. Besley, T. W. Chamberlain, Comparison of atomic scale dynamics for the middle and late transition metal nanocatalysts. Nat. Commun. 9(1), 1-10 (2018). Publisher: Nature Publishing Group
40 C. Dwyer, R. Erni, J. Etheridge, Measurement of effective source distribution and its importance for quantitative interpretation of STEM images. Ultramicroscopy. 110(8), 952-957 (2010). ISSN 0304-3991. https://doi.org/10.1016/j.ultramic.2010.01.007. http://www.sciencedirect.com/science/article/pii/S0304399110000100   DOI