TopoSinGAN: Learning a Topology-Aware Generative Model from a Single Image
Abstract
:1. Introduction
1.1. SinGAN
1.2. The Need for Topological Accuracy
2. Materials and Methods
2.1. Topology Loss Algorithm
- I
- Soft Thresholding with Sigmoid Function ()To detect terminals, the input mask (M) undergoes a custom sigmoid function , resulting in a soft binarization of the tensor. acts as a soft thresholding mechanism ensuring continuity across the entire domain. The mathematical representation is as follows:
- II
- Detection of Terminal Nodes via ConvolutionAfter applying the sigmoid function, the soft-binarized mask is convolved with a set of eight fixed kernels, , which are depicted in Figure 2. These kernels are specifically designed to target the eight main cardinal and intercardinal directions in a 2D space (north, northeast, east, southeast, south, southwest, west, northwest). Empirically, the use of these kernels has proven effective in identifying discontinuities, thereby enhancing the model’s ability to evaluate topological performance. This evaluation is integral to the core of the developed loss function, enabling the application of corrective penalties during the training phase for improved topological accuracy.
- III
- Loss ComputationThe topology loss is computed as:The final gradient of the topology loss with respect to the original mask M is:Since both components are differentiable, the overall gradient is well-defined, and the optimization function can be represented as:
2.2. Evaluation
2.2.1. Node Topology Clustering (NTC)
2.2.2. Modified Fréchet Inception Distance (FID)
2.3. Experimental Setup
2.3.1. Agricultural Fields
2.3.2. Dendrology
2.4. System Setup
2.4.1. System Configurations
2.4.2. Training Procedure
3. Results
3.1. NTC-Based Graph Classification
3.2. TopoSinGAN Performance Evaluation Using NTC
3.3. Modified FID Evaluation
3.4. Comparative Efficiency Analysis
3.5. Hyperparameter Tuning of for CREMI Experiment
4. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
GAN | Generative adversarial network |
GT | Ground truth |
TD | Terminal distance |
NTC | Node topology clustering |
PH | Persistence homology |
STD | Standard deviation |
GIS | Geographic information science |
TDA | Topological data analysis |
MSI | Minnesota supercomputing institute |
FID | Fréchet inception distance |
RGB | Red, green, blue |
Appendix A. Demonstration of Independence of NTC Metric on the Placement of the Randomly Added Terminal Node
Cluster Type | Agricultural Fields | Dendrology | ||
---|---|---|---|---|
Mean | STD | Mean | STD | |
HH | 22.14 | 2.98 | 20.04 | 1.71 |
LL | 18.91 | 1.84 | 18.76 | 1.78 |
LH | 0.017 | 0.22 | 0 | 0 |
HL | 0 | 0 | 0 | 0 |
Appendix B. Extreme Cases Considered for Graph Classification
Appendix C. Demonstrating the Non-Disruptive Effect of Topology Loss on SinGAN Learning
SinGAN | TopoSinGAN | |||
---|---|---|---|---|
Mean | STD | Mean | STD | |
Agricultural | 0.9873 | 0.0023 | 0.9895 | 0.0001 |
Dendrology | 0.9892 | 0.0011 | 0.9891 | 0.0004 |
References
- Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative Adversarial Nets. Adv. Neural Inf. Process. Syst. 2014, 27. [Google Scholar]
- Xu, M.; Yoon, S.; Fuentes, A.; Park, D.S. A Comprehensive Survey of Image Augmentation Techniques for Deep Learning. Pattern Recognit. 2023, 137, 109347. [Google Scholar] [CrossRef]
- Liu, H.; Wan, Z.; Huang, W.; Song, Y.; Han, X.; Liao, J. PD-GAN: Probabilistic Diverse GAN for Image Inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 9371–9381. [Google Scholar]
- Ren, Y.; Wu, J.; Zhang, P.; Zhang, M.; Xiao, X.; He, Q.; Wang, R.; Zheng, M.; Pan, X. UGC: Unified GAN Compression for Efficient Image-to-Image Translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 2–6 October 2023; pp. 17281–17291. [Google Scholar]
- Mahapatra, D.; Ge, Z. Training Data Independent Image Registration with GANs Using Transfer Learning and Segmentation Information. In Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), Venice, Italy, 8–11 April 2019; pp. 709–713. [Google Scholar] [CrossRef]
- Jain, M.; Meegan, C.; Dev, S. Using GANs to Augment Data for Cloud Image Segmentation Task. In Proceedings of the 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, Brussels, Belgium, 11–16 July 2021; pp. 3452–3455. [Google Scholar] [CrossRef]
- Zhaoa, Z.; Wang, Y.; Liu, K.; Yang, H.; Sun, Q.; Qiao, H. Semantic Segmentation by Improved Generative Adversarial Networks. arXiv 2021, arXiv:2104.09917. [Google Scholar]
- Majurski, M.; Manescu, P.; Padi, S.; Schaub, N.; Hotaling, N.; Simon, C.; Bajcsy, P. Cell Image Segmentation Using Generative Adversarial Networks, Transfer Learning, and Augmentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 16–17 June 2019. [Google Scholar]
- Thambawita, V.; Salehi, P.; Sheshkal, S.A.; Hicks, S.A.; Hammer, H.L.; Parasa, S.; de Lange, T.; Halvorsen, P.; Riegler, M.A. Singan-Seg: Synthetic Training Data Generation for Medical Image Segmentation. PLoS ONE 2022, 17, e0267976. [Google Scholar] [CrossRef] [PubMed]
- Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; Fei-Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar] [CrossRef]
- You, C.; Li, G.; Zhang, Y.; Zhang, X.; Shan, H.; Li, M.; Ju, S.; Zhao, Z.; Zhang, Z.; Cong, W.; et al. CT Super-Resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble (GAN-CIRCLE). IEEE Trans. Med. Imaging 2020, 39, 188–203. [Google Scholar] [CrossRef]
- Dy, J.B.; Virtusio, J.J.; Tan, D.S.; Lin, Y.-X.; Ilao, J.; Chen, Y.-Y.; Hua, K.-L. MCGAN: Mask Controlled Generative Adversarial Network for Image Retargeting. Neural Comput. Appl. 2023, 35, 10497–10509. [Google Scholar] [CrossRef]
- Shaham, T.R.; Dekel, T.; Michaeli, T. SinGAN: Learning a Generative Model From a Single Natural Image. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea, 27 October–2 November 2019; pp. 4570–4580. [Google Scholar] [CrossRef]
- Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A.C.; Lo, W.-Y.; et al. Segment Anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 2–3 October 2023; pp. 4015–4026. [Google Scholar]
- Mo, Y.; Wu, Y.; Yang, X.; Liu, F.; Liao, Y. Review the State-Of-The-Art Technologies of Semantic Segmentation Based on Deep Learning. Neurocomputing 2022, 493, 626–646. [Google Scholar] [CrossRef]
- Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; Courville, A.C. Improved Training of Wasserstein GANs. Adv. Neural Inf. Process. Syst. 2017, 30, 5767–5777. [Google Scholar]
- Liu, C.; Ma, B.; Ban, X.; Xie, Y.; Wang, H.; Xue, W.; Ma, J.; Xu, K. Enhancing Boundary Segmentation for Topological Accuracy with Skeleton-Based Methods. arXiv 2024, arXiv:2404.18539. [Google Scholar]
- Mosinska, A.; Marquez-Neila, P.; Kozinski, M.; Fua, P. Beyond the Pixel-Wise Loss for Topology-Aware Delineation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 3136–3145. [Google Scholar]
- Hu, X. Structure-Aware Image Segmentation with Homotopy Warping. Adv. Neural Inf. Process. Syst. 2022, 35, 24046–24059. [Google Scholar]
- Costea, D.; Marcu, A.; Leordeanu, M.; Slusanschi, E. Creating Roadmaps in Aerial Images with Generative Adversarial Networks and Smoothing-Based Optimization. In Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy, 22–29 October 2017. [Google Scholar]
- Park, E. Refining Inferred Road Maps Using GANs. Ph.D. Thesis, Massachusetts Institute of Technology, Cambridge, MA, USA, 2019. [Google Scholar]
- Guo, X.; Zhou, R. Data Augmentation Method for Extracting Partially Occluded Roads From High Spatial Resolution Remote Sensing Images. IEEE Access 2023, 11, 79232–79239. [Google Scholar] [CrossRef]
- Patel, H.; Farrelly, C.; Hathaway, Q.A.; Rozenblit, J.Z.; Deepa, D.; Singh, Y.; Chaudhary, A.; Himeur, Y.; Mansoor, W.; Atalls, S. Topology-Aware GAN (TopoGAN): Transforming Medical Imaging Advances. In Proceedings of the 2023 Tenth International Conference on Social Networks Analysis, Management and Security (SNAMS), Abu Dhabi, United Arab Emirates, 21–24 November 2023; pp. 1–3. [Google Scholar] [CrossRef]
- Arjovsky, M.; Chintala, S.; Bottou, L. Wasserstein Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017; pp. 214–223. [Google Scholar]
- Lim, J.H.; Ye, J.C. Geometric GAN. arXiv 2017, arXiv:1705.02894. [Google Scholar]
- Qi, G.-J.; Zhang, L.; Hu, H.; Edraki, M.; Wang, J.; Hua, X.-S. Global versus Localized Generative Adversarial Nets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 1517–1525. [Google Scholar]
- Kossaifi, J.; Tran, L.; Panagakis, Y.; Pantic, M. GAGAN: Geometry-Aware Generative Adversarial Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 878–887. [Google Scholar]
- Fu, H.; Gong, M.; Wang, C.; Batmanghelich, K.; Zhang, K.; Tao, D. Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 2427–2436. [Google Scholar]
- Chen, W.; Yu, S.; Wu, J.; Ma, K.; Bian, C.; Chu, C.; Shen, L.; Zheng, Y. TR-GAN: Topology ranking GAN with triplet loss for retinal artery/vein classification. In Proceedings of the Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, 4–8 October 2020; pp. 616–625. [Google Scholar]
- Chen, W.; Yu, S.; Ma, K.; Ji, W.; Bian, C.; Chu, C.; Shen, L.; Zheng, Y. TW-GAN: Topology and width aware GAN for retinal artery/vein classification. Med. Image Anal. 2022, 77, 102340. [Google Scholar] [CrossRef]
- Liu, W.; Chen, P.Y.; Yu, F.; Suzumura, T.; Hu, G. Learning graph topological features via GAN. IEEE Access 2019, 7, 21834–21843. [Google Scholar] [CrossRef]
- Wang, F.; Liu, H.; Samaras, D.; Chen, C. TopoGAN: A Topology-Aware Generative Adversarial Network. In Computer Vision—ECCV 2020; Springer International Publishing: Cham, Switzerland, 2020; pp. 118–136. [Google Scholar] [CrossRef]
- Hu, X.; Li, F.; Samaras, D.; Chen, C. Topology-Preserving Deep Image Segmentation. Adv. Neural Inf. Process. Syst. 2019, 32, abs/1906.05404. [Google Scholar]
- Clough, J.R.; Byrne, N.; Oksuz, I.; Zimmer, V.A.; Schnabel, J.A.; King, A.P. A Topological Loss Function for Deep-Learning Based Image Segmentation Using Persistent Homology. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 44, 8766–8778. [Google Scholar] [CrossRef]
- Bao, J.; Wang, Z.; Wang, J.; Yan, C. Persistent Homology Based Generative Adversarial Network. In Proceedings of the VISIGRAPP (4: VISAPP), Lisbon, Portugal, 19–21 February 2023; pp. 196–203. [Google Scholar]
- Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the Inception Architecture for Computer Vision. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2818–2826. [Google Scholar] [CrossRef]
- Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; Chen, X. Improved Techniques for Training GANs. Adv. Neural Inf. Process. Syst. 2016, 29. [Google Scholar]
- Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; Hochreiter, S. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Adv. Neural Inf. Process. Syst. 2017, 30. [Google Scholar]
- Horak, D.; Yu, S.; Salimi-Khorshidi, G. Topology Distance: A Topology-Based Approach for Evaluating Generative Adversarial Networks. AAAI 2021, 35, 7721–7728. [Google Scholar] [CrossRef]
- Cerri, A.; Di Fabio, B.; Jabłoński, G.; Medri, F. Comparing Shapes through Multi-Scale Approximations of the Matching Distance. Comput. Vis. Image Underst. 2014, 121, 43–56. [Google Scholar] [CrossRef]
- Sheehy, D.; Kisielius, O.; Cavanna, N.J. Computing the Shift-Invariant Bottleneck Distance for Persistence Diagrams. In Proceedings of the Canadian Conference on Computational Geometry, Winnipeg, MB, Canada, 8–10 August 2018; pp. 78–84. [Google Scholar]
- Bouttier, J.; Di Francesco, P.; Guitter, E. Geodesic Distance in Planar Graphs. Nucl. Phys. B 2003, 663, 535–567. [Google Scholar] [CrossRef]
- Anselin, L. Local Indicators of Spatial association—LISA. Geogr. Anal. 1995, 27, 93–115. [Google Scholar] [CrossRef]
- Hearst, M.A.; Dumais, S.T.; Osuna, E.; Platt, J.; Scholkopf, B. Support Vector Machines. IEEE Intell. Syst. Their Appl. 1998, 13, 18–28. [Google Scholar] [CrossRef]
- Aliakbary, S.; Habibi, J.; Movaghar, A. Feature Extraction from Degree Distribution for Comparison and Analysis of Complex Networks. Comput. J. 2015, 58, 2079–2091. [Google Scholar] [CrossRef]
- Attar, N.; Aliakbary, S. Classification of Complex Networks Based on Similarity of Topological Network Features. Chaos 2017, 27, 091102. [Google Scholar] [CrossRef]
- Paul, E.; Rényi, A. On the Central Limit Theorem for Samples from a Finite Population. Sel. Pap. Alfréd Rényi 1959, 353, 49–61. [Google Scholar]
- Watts, D.; Strogatz, S. Collective Dynamics of “small-World” Networks. Nature 1998, 393, 440–442. [Google Scholar] [CrossRef]
- Costa, L.d.F.; Rodrigues, F.A.; Travieso, G.; Villas Boas, P.R. Characterization of Complex Networks: A Survey of Measurements. Adv. Phys. 2007, 56, 167–242. [Google Scholar] [CrossRef]
- Newman, M.E.J. Modularity and Community Structure in Networks. Proc. Natl. Acad. Sci. USA 2006, 103, 8577–8582. [Google Scholar] [CrossRef]
- Miccai Challenge on Circuit Reconstruction from Electron Microscopy Images. Available online: http://cremi.org/ (accessed on 23 October 2024).
- Griffin, D.; Porter, S.T.; Trumper, M.L.; Carlson, K.E.; Crawford, D.J.; Schwalen, D.; McFadden, C.H. Gigapixel Macro Photography of Tree Rings. Tree-Ring Res. 2021, 77, 86–94. [Google Scholar] [CrossRef]
- Hacke, U.G.; Spicer, R.; Schreiber, S.G.; Plavcová, L. An Ecophysiological and Developmental Perspective on Variation in Vessel Diameter. Plant Cell Environ. 2017, 40, 831–845. [Google Scholar] [CrossRef] [PubMed]
- Research Computing, Research and Innovation Office. Minnesota Supercomputing Institute (MSI)-Agate Cluster. Available online: https://msi.umn.edu/about-msi-services/high-performance-computing/agate (accessed on 23 October 2024).
- Li, Y.; Xiao, N.; Ouyang, W. Improved Generative Adversarial Networks with Reconstruction Loss. Neurocomputing 2019, 323, 363–372. [Google Scholar] [CrossRef]
- Yaveroğlu, Ö.N.; Milenković, T.; Pržulj, N. Proper Evaluation of Alignment-Free Network Comparison Methods. Bioinformatics 2015, 31, 2697–2704. [Google Scholar] [CrossRef]
- Yaveroğlu, Ö.N.; Malod-Dognin, N.; Davis, D.; Levnajic, Z.; Janjic, V.; Karapandza, R.; Stojmirovic, A.; Pržulj, N. Revealing the Hidden Language of Complex Networks. Sci. Rep. 2014, 4, 4547. [Google Scholar] [CrossRef] [PubMed]
- Przulj, N. Biological Network Comparison Using Graphlet Degree Distribution. Bioinformatics 2007, 23, e177–e183. [Google Scholar] [CrossRef]
- Kuchaiev, O.; Stevanović, A.; Hayes, W.; Pržulj, N. GraphCrunch 2: Software Tool for Network Modeling, Alignment and Clustering. BMC Bioinform. 2011, 12, 24. [Google Scholar] [CrossRef] [PubMed]
- Ahmadkhani, M.; Shook, E. TopoSinGAN Github Repository. Available online: https://github.com/mohsenumn/TopoSinGAN (accessed on 23 October 2024).
SinGAN | TopoSinGAN | |||
---|---|---|---|---|
Mean | STD | Mean | STD | |
Agricultural | 15.15 | 3.41 | 3.94 | 1.81 |
Dendrology | 14.55 | 3.07 | 2.44 | 1.35 |
SinGAN | WGAN | TopoGAN | TopoSinGAN | |||||
---|---|---|---|---|---|---|---|---|
Mean | STD | Mean | STD | Mean | STD | Mean | STD | |
CREMI | 15.46 | 2.86 | 15.76 | 2.77 | 11.05 | 2.80 | 8.15 | 2.01 |
SinGAN | TopoSinGAN | |||
---|---|---|---|---|
Mean | STD | Mean | STD | |
Agricultural | 0.2485 | 0.0086 | 0.1914 | 0.0083 |
Dendrology | 0.0014 | 0.0013 | 0.0014 | 0.0018 |
Input Dimensions | Pyramid Scales | SinGAN | TopoSinGAN |
---|---|---|---|
(Minutes) | (Minutes) | ||
175 × 240 × 4 | 8 | 20.93 | 22.90 |
350 × 400 × 4 | 11 | 65.13 | 69.88 |
500 × 500 × 4 | 12 | 108.11 | 114.58 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ahmadkhani, M.; Shook, E. TopoSinGAN: Learning a Topology-Aware Generative Model from a Single Image. Appl. Sci. 2024, 14, 9944. https://doi.org/10.3390/app14219944
Ahmadkhani M, Shook E. TopoSinGAN: Learning a Topology-Aware Generative Model from a Single Image. Applied Sciences. 2024; 14(21):9944. https://doi.org/10.3390/app14219944
Chicago/Turabian StyleAhmadkhani, Mohsen, and Eric Shook. 2024. "TopoSinGAN: Learning a Topology-Aware Generative Model from a Single Image" Applied Sciences 14, no. 21: 9944. https://doi.org/10.3390/app14219944
APA StyleAhmadkhani, M., & Shook, E. (2024). TopoSinGAN: Learning a Topology-Aware Generative Model from a Single Image. Applied Sciences, 14(21), 9944. https://doi.org/10.3390/app14219944