{"type":"ScholarlyArticle","title":"STS-GAN: Can We Synthesize Solid Texture with High Fidelity from Arbitrary 2D Exemplar?","authors":[{"name":"Xin Zhao","slug":"xin-zhao","given":"Xin","family":"Zhao","affiliation":"Shandong Provincial Key Laboratory of Preparation and Measurement of Building Materials , University of Jinan , Jinan 250022 , China"},{"name":"Jifeng Guo","slug":"jifeng-guo","given":"Jifeng","family":"Guo","affiliation":"School of Computer Science and Engineering , South China University of Technology , Guangzhou 510641 , China"},{"name":"Lin Wang","slug":"lin-wang","given":"Lin","family":"Wang","affiliation":"Shandong Provincial Key Laboratory of Network Based Intelligent Computing , University of Jinan , Jinan 250022 , China; Quan Cheng Laboratory , Jinan 250100 , China"},{"name":"Fanqi Li","slug":"fanqi-li","given":"Fanqi","family":"Li","affiliation":"Shandong Provincial Key Laboratory of Network Based Intelligent Computing , University of Jinan , Jinan 250022 , China; Shandong Qiuqi Analysis Instrument Co., Ltd. , Jinan 250024 , China"},{"name":"Jiahao Li","slug":"jiahao-li","given":"Jiahao","family":"Li","affiliation":"Shandong Provincial Key Laboratory of Network Based Intelligent Computing , University of Jinan , Jinan 250022 , China"},{"name":"Junteng Zheng","slug":"junteng-zheng","given":"Junteng","family":"Zheng","affiliation":"David R. Cheriton School of Computer Science , University of Waterloo , Waterloo , ON N2L 3G1 , Canada"},{"name":"Bo Yang","slug":"bo-yang","given":"Bo","family":"Yang","affiliation":"Quan Cheng Laboratory , Jinan 250100 , China; Shandong Provincial Key Laboratory of Network Based Intelligent Computing , University of Jinan , Jinan 250022 , China"}],"abstract":"Solid texture synthesis (STS), an effective way to extend a 2D exemplar to a 3D solid volume, exhibits advantages in computational photography. However, existing methods generally fail to accurately learn arbitrary textures, which may result in the failure to synthesize solid textures with high fidelity. In this paper, we propose a novel generative adversarial nets-based framework (STS-GAN) to extend the given 2D exemplar to arbitrary 3D solid textures. In STS-GAN, multi-scale 2D texture discriminators evaluate the similarity between the given 2D exemplar and slices from the generated 3D texture, promoting the 3D texture generator synthesizing realistic solid textures. Finally, experiments demonstrate that the proposed method can generate high-fidelity solid textures with similar visual characteristics to the 2D exemplar.","doi":"10.24963/ijcai.2023/196","url":"https://ijcai.org/proceedings/2023/196","pdf":"https://ijcai.org/proceedings/2023/0196.pdf","markdown":"https://ijcai.org/proceedings/2023/196.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/196","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":196,"pages":{"first":1768,"last":1776},"keywords":["Computer Vision: CV: Computational photography","Computer Vision: CV: Neural generative models, auto encoders, GANs","Computer Vision: CV: 3D computer vision"],"videos":[],"references":[{"position":1,"raw":"[Ashton et al., 2020] Tristan N Ashton, Donna Post Guillen, and William H Harris. An algorithm to generate synthetic 3d microstructures from 2d exemplars. JOM, 72(1):65-74, 2020.","title":"An algorithm to generate synthetic 3d microstructures from 2d exemplars","authors":"Donna Post Tristan N Ashton, William H Guillen, Harris","year":"2020","venue":"JOM","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bergmann et al., 2017] Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. Learning texture manifolds with the periodic spatial gan. In Proceedings of the 34th Interna- tional Conference on Machine Learning-Volume 70, pages 469-477, 2017.","title":"Learning texture manifolds with the periodic spatial gan","authors":"Urs Bergmann, Nikolay Jetchev, Roland Vollgraf","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Wang, 2010] Jiating Chen and Bin Wang. High quality solid texture synthesis using position and index his- togram matching. The Visual Computer, 26(4):253-262, 2010.","title":"High quality solid texture synthesis using position and index histogram matching","authors":"Chen, Wang ;, Jiating Chen, Bin Wang","year":"2010","venue":"The Visual Computer","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Di Lillo et al., 2007] Antonella Di Lillo, Giovanni Motta, and James A Storer. Texture classification based on dis- criminative features extracted in the frequency domain. In 2007 IEEE International Conference on Image Process- ing, volume 2, pages II -53-II -56, 2007.","title":"Texture classification based on discriminative features extracted in the frequency domain","authors":"Antonella Di Lillo, Giovanni Motta, James A Storer","year":"2007","venue":"IEEE International Conference on Image Processing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Efros and Leung, 1999] Alexei A Efros and Thomas K Le- ung. Texture synthesis by non-parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1033-1038, 1999.","title":"Texture synthesis by non-parametric sampling","authors":"Efros, Leung ;, Thomas K Leung","year":"1999","venue":"Proceedings of the seventh IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fayolle et al., 2021] PA Fayolle, Leigh McLoughlin, M Sanchez, G Pasko, and A Pasko. Modeling and visual- ization of multi-material volumes. Scientific Visualization, 13(2):117-148, 2021.","title":"Modeling and visualization of multi-material volumes","authors":"Fayolle","year":"2021","venue":"Scientific Visualization","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Georgiadis et al., 2013] Georgios Georgiadis, Alessandro Chiuso, and Stefano Soatto. Texture compression. In 2013 Data Compression Conference, pages 221-230, 2013.","title":"Texture compression","authors":"Georgios Georgiadis, Alessandro Chiuso, Stefano Soatto","year":"2013","venue":"2013 Data Compression Conference","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ghazanfarpour and Dischler, 1995] Djamchid Ghazanfar- pour and JeanMichel Dischler. Spectral analysis for automatic 3-d texture generation. Computers & Graphics, 19(3):413-422, 1995.","title":"Djamchid Ghazanfarpour and JeanMichel Dischler. Spectral analysis for automatic 3-d texture generation","authors":"Dischler Ghazanfarpour","year":"1995","venue":"Computers & Graphics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in Neural Informa- tion Processing Systems, volume 27, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gulrajani et al., 2017] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, vol- ume 30, 2017.","title":"Improved training of wasserstein gans","authors":"Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Aaron C Vincent Dumoulin, Courville","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gutierrez et al., 2020] Jorge Gutierrez, Julien Rabin, Bruno Galerne, and Thomas Hurtut. On demand solid texture synthesis using deep 3d networks. Computer Graphics Fo- rum, 39(1):511-530, 2020.","title":"On demand solid texture synthesis using deep 3d networks","authors":"Jorge Gutierrez, Julien Rabin, Bruno Galerne, Thomas Hurtut","year":"2020","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Heeger and Bergen, 1995] David J Heeger and James R Bergen. Pyramid-based texture analysis/synthesis. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 229-238, 1995.","title":"Pyramid-based texture analysis/synthesis","authors":"J David, Heeger, James R Bergen","year":"1995","venue":"Proceedings of the 22nd annual conference on Computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Henzler et al., 2020] Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Learning a neural 3d texture space from 2d exemplars. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 8356-8364, 2020.","title":"Learning a neural 3d texture space from 2d exemplars","authors":"Henzler","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Iwasaki et al., 2017] Kei Iwasaki, Yoshinori Dobashi, and Makoto Okabe. Example-based synthesis of three- dimensional clouds from photographs. In Proceedings of the Computer Graphics International Conference, pages 1-6, 2017.","title":"Example-based synthesis of threedimensional clouds from photographs","authors":"Kei Iwasaki, Yoshinori Dobashi, Makoto Okabe","year":"2017","venue":"Proceedings of the Computer Graphics International Conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jagnow et al., 2004] Robert Jagnow, Julie Dorsey, and Holly Rushmeier. Stereological techniques for solid tex- tures. ACM Transactions on Graphics (TOG), 23(3):329- 335, 2004.","title":"Stereological techniques for solid textures","authors":"Robert Jagnow, Julie Dorsey, Holly Rushmeier","year":"2004","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd In- ternational Conference on Learning Representations, San Diego, CA, USA, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"3rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kopf et al., 2007] Johannes Kopf, Chi-Wing Fu, Daniel Cohen-Or, Oliver Deussen, Dani Lischinski, and Tien- Tsin Wong. Solid texture synthesis from 2d exemplars. In ACM SIGGRAPH 2007 Papers, page 2-es, 2007.","title":"Solid texture synthesis from 2d exemplars","authors":"Johannes Kopf, Chi-Wing Fu, Daniel Cohen-Or, Oliver Deussen, Dani Lischinski, Tien-Tsin Wong","year":"2007","venue":"ACM SIGGRAPH 2007 Papers","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Laursen et al., 2011] Lasse Farnung Laursen, Bjarne Kjær Ersbøll, and Jakob Andreas Bærentzen. Anisotropic 3d texture synthesis with application to volume rendering. In WSCG' 2011 Communication Papers Proceedings, pages 49-57, 2011.","title":"Anisotropic 3d texture synthesis with application to volume rendering","authors":"Laursen","year":"2011","venue":"WSCG' 2011 Communication Papers Proceedings","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mariethoz and Lefebvre, 2014] Gregoire Mariethoz and Sylvain Lefebvre. Bridges between multiple-point geo- statistics and texture synthesis: Review and guidelines for future research. Computers & Geosciences, 66:66-80, 2014.","title":"Bridges between multiple-point geostatistics and texture synthesis: Review and guidelines for future research","authors":"Lefebvre Mariethoz, Gregoire Mariethoz, Sylvain Lefebvre","year":"2014","venue":"Computers & Geosciences","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mark et al., 2015] Benjamin Mark, Tudor Berechet, Tobias Mahlmann, and Julian Togelius. Procedural generation of 3d caves for games on the gpu. In Proceedings of the 10th International Conference on the Foundations of Dig- ital Games, Pacific Grove, CA, USA, 2015.","title":"Procedural generation of 3d caves for games on the gpu","authors":"Benjamin Mark, Tudor Berechet, Tobias Mahlmann, Julian Togelius","year":"2015","venue":"Proceedings of the 10th International Conference on the Foundations of Digital Games","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peachey, 1985] Darwyn R Peachey. Solid texturing of com- plex surfaces. In Proceedings of the 12th annual con- ference on Computer graphics and interactive techniques, pages 279-286, 1985.","title":"Solid texturing of complex surfaces","authors":"R Darwyn, Peachey","year":"1985","venue":"Proceedings of the 12th annual conference on Computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Perlin, 1985] Ken Perlin. An image synthesizer. ACM Sig- graph Computer Graphics, 19(3):287-296, 1985.","title":"An image synthesizer","authors":"Ken Perlin","year":"1985","venue":"ACM Siggraph Computer Graphics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Portenier et al., 2020] Tiziano Portenier, Siavash Arjo- mand Bigdeli, and Orcun Goksel. Gramgan: Deep 3d tex- ture synthesis from 2d exemplars. In Advances in Neural Information Processing Systems, volume 33, pages 6994- 7004, 2020.","title":"Tiziano Portenier, Siavash Arjomand Bigdeli, and Orcun Goksel. Gramgan: Deep 3d texture synthesis from 2d exemplars","authors":"Portenier","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shaham et al., 2019] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning a generative model from a single natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4570-4580, 2019.","title":"Singan: Learning a generative model from a single natural image","authors":"Shaham","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Con- ference on Learning Representations, San Diego, CA, USA, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"3rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tadi Bani and Fekri-Ershad, 2019] Neda Tadi Bani and Shervan Fekri-Ershad. Content-based image retrieval based on combination of texture and colour information extracted in spatial and frequency domains. The electronic library, 37(4):650-666, 2019.","title":"Neda Tadi Bani and Shervan Fekri-Ershad. Content-based image retrieval based on combination of texture and colour information extracted in spatial and frequency domains","authors":"Tadi Bani, Fekri-Ershad","year":"2019","venue":"The electronic library","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tong et al., 2002] Xin Tong, Jingdan Zhang, Ligang Liu, Xi Wang, Baining Guo, and Heung-Yeung Shum. Synthe- sis of bidirectional texture functions on arbitrary surfaces. ACM Transactions on Graphics, 21(3):665-672, 2002.","title":"Synthesis of bidirectional texture functions on arbitrary surfaces","authors":"Xin Tong, Jingdan Zhang, Ligang Liu, Xi Wang, Baining Guo, Heung-Yeung Shum","year":"2002","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Turner and Kalidindi, 2016] David M. Turner and Surya R. Kalidindi. Statistical construction of 3-d microstructures from 2-d exemplars collected on oblique sections. Acta Materialia, 102:136-148, 2016.","title":"Statistical construction of 3-d microstructures from 2-d exemplars collected on oblique sections","authors":"David M Turner, Surya R Kalidindi","year":"2016","venue":"Acta Materialia","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ulyanov et al., 2016] D Ulyanov, V Lebedev, A Vedaldi, and V Lempitsky. Texture networks: Feed-forward syn- thesis of textures and stylized images. In Proceedings of the 33rd International Conference on International Con- ference on Machine Learning, volume 48, pages 1349- 1357, 2016.","title":"Texture networks: Feed-forward synthesis of textures and stylized images","authors":"Ulyanov, Lebedev, Vedaldi, Lempitsky","year":"2016","venue":"Proceedings of the 33rd International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2018] Na Wang, Guodong Chen, and Huiyan Zeng. An optimization algorithm of space anisotropic hep- atic artery solid texture synthesis. Current Medical Imag- ing, 14(4):609-616, 2018.","title":"An optimization algorithm of space anisotropic hepatic artery solid texture synthesis","authors":"Na Wang, Guodong Chen, Huiyan Zeng","year":"2018","venue":"Current Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wei and Levoy, 2000] Li-Yi Wei and Marc Levoy. Fast tex- ture synthesis using tree-structured vector quantization. In Proceedings of the 27th annual conference on Com- puter graphics and interactive techniques, SIGGRAPH '00, page 479-488, 2000.","title":"Fast texture synthesis using tree-structured vector quantization","authors":"Li-Yi Wei, Marc Levoy","year":"2000","venue":"Proceedings of the 27th annual conference on Computer graphics and interactive techniques, SIGGRAPH '00","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wei, 2002] Li-Yi Wei. Texture synthesis by fixed neighbor- hood searching. PhD thesis, Stanford University, 2002.","title":"Texture synthesis by fixed neighborhood searching","authors":"Li-Yi Wei","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}