{"type":"ScholarlyArticle","title":"MetaJND: A Meta-Learning Approach for Just Noticeable Difference Estimation","authors":[{"name":"Miaohui Wang","slug":"miaohui-wang","given":"Miaohui","family":"Wang","affiliation":"Guangdong Key Laboratory of Intelligent Information Processing , Shenzhen University"},{"name":"Yukuan Zhu","slug":"yukuan-zhu","given":"Yukuan","family":"Zhu","affiliation":"Guangdong Key Laboratory of Intelligent Information Processing , Shenzhen University"},{"name":"Rong Zhang","slug":"rong-zhang","given":"Rong","family":"Zhang","affiliation":"Guangdong Key Laboratory of Intelligent Information Processing , Shenzhen University"},{"name":"Wuyuan Xie","slug":"wuyuan-xie","given":"Wuyuan","family":"Xie","affiliation":"College of Computer Science and Software Engineering , Shenzhen University"}],"abstract":"The modeling of just noticeable difference (JND) in supervised learning for visual signals has made significant progress. However, existing JND models often suffer from limited generalization due to the need for large-scale training data and their constraints to certain image types. Moreover, these models primarily focus on a single RGB modality, ignoring the potential complementary impacts of multiple modalities. To address these challenges, we propose a new meta-learning approach for the JND modeling, called MetaJND. We introduce two key visual-sensitive modalities like saliency and depth, and leverage a self-attention mechanism for effective interdependence of multi-modal features. Additionally, we incorporate meta-learning for the modality alignment, facilitating dynamic weight generation. Furthermore, we perform hierarchical fusion through multi-layer channel and spatial feature rectification. Experimental results on four benchmark datasets demonstrate the effectiveness of our MetaJND. Moreover, we have also evaluated its performance in compression and watermarking applications, observing higher bit-rate savings and better watermark hiding capabilities.","doi":"10.24963/ijcai.2024/349","url":"https://ijcai.org/proceedings/2024/349","pdf":"https://ijcai.org/proceedings/2024/0349.pdf","markdown":"https://ijcai.org/proceedings/2024/349.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/349","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Humans and AI","track":"Main Track","number":349,"pages":{"first":3151,"last":3159},"keywords":["Humans and AI: HAI: Cognitive modeling","Humans and AI: HAI: Cognitive systems","Humans and AI: HAI: Applications"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2022] Yuanchao Bai, Xu Yang, Xianming Liu, Junjun Jiang, Yaowei Wang, Xiangyang Ji, and Wen Gao. Towards end-to-end image compression and analysis with transformers. In AAAI Conference on Artificial Intelli- gence (AAAI), volume 36, pages 104-112, 2022.","title":"Towards end-to-end image compression and analysis with transformers","authors":"Yuanchao Bai, Xu Yang, Xianming Liu, Junjun Jiang, Yaowei Wang, Xiangyang Ji, Wen Gao","year":"2022","venue":"AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Wu, 2019] Zhenzhong Chen and Wei Wu. Asym- metric foveated just-noticeable-difference model for im- ages with visual field inhomogeneities. IEEE Trans- actions on Circuits and Systems for Video Technology, 30(11):4064-4074, 2019.","title":"Asymmetric foveated just-noticeable-difference model for images with visual field inhomogeneities","authors":"Zhenzhong Chen, Wei Wu","year":"2019","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2021] Zhengxue Cheng, Ting Fu, Jiapeng Hu, Li Guo, Shihao Wang, Xiongxin Zhao, Dajiang Zhou, and Yang Song. Perceptual Image Compression using Rela- tivistic Average Least Squares GANs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1895-1900, 2021.","title":"Perceptual Image Compression using Relativistic Average Least Squares GANs","authors":"Zhengxue Cheng, Ting Fu, Jiapeng Hu, Li Guo, Shihao Wang, Xiongxin Zhao, Dajiang Zhou, Yang Song","year":"2021","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fang et al., 2023] Han Fang, Zhaoyang Jia, Hang Zhou, Zehua Ma, and Weiming Zhang. Encoded feature en- hancement in watermarking network for distortion in real scenes. IEEE Transactions on Multimedia, 25:2648-2660, 2023.","title":"Encoded feature enhancement in watermarking network for distortion in real scenes","authors":"Han Fang, Zhaoyang Jia, Hang Zhou, Zehua Ma, Weiming Zhang","year":"2023","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2022] Shuai He, Yongchang Zhang, Rui Xie, Dongxiang Jiang, and Anlong Ming. Rethinking image aesthetics assessment: Models, datasets and benchmarks. In International Conference on International Joint Con- ferences on Artificial Intelligence (IJCAI), pages 942-948, 2022.","title":"Rethinking image aesthetics assessment: Models, datasets and benchmarks","authors":"Shuai He, Yongchang Zhang, Rui Xie, Dongxiang Jiang, Anlong Ming","year":"2022","venue":"International Conference on International Joint Conferences on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2022/132"},{"position":6,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 7132-7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2019] Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun. Meta-SR: A magnification-arbitrary network for super-resolution. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 1575-1584, 2019.","title":"Meta-SR: A magnification-arbitrary network for super-resolution","authors":"Hu","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang et al., 2022] Qiuping Jiang, Zhentao Liu, Shiqi Wang, Feng Shao, and Weisi Lin. Toward top-down just notice- able difference estimation of natural images. IEEE Trans- actions on Image Processing, 31:3697-3712, 2022.","title":"Toward top-down just noticeable difference estimation of natural images","authors":"Jiang","year":"2022","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jin et al., 2021] Jian Jin, Xingxing Zhang, Xin Fu, Huan Zhang, Weisi Lin, Jian Lou, and Yao Zhao. Just noticeable difference for deep machine vision. IEEE Transactions on Circuits and Systems for Video Technology, 32(6):3452- 3461, 2021.","title":"Just noticeable difference for deep machine vision","authors":"Jian Jin, Xingxing Zhang, Xin Fu, Huan Zhang, Weisi Lin, Jian Lou, Yao Zhao","year":"2021","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin and Ghinea, 2022] Weisi Lin and Gheorghita Ghinea. Progress and opportunities in modelling just-noticeable difference (JND) for multimedia. IEEE Transactions on Multimedia, 24:3706-3721, 2022.","title":"Progress and opportunities in modelling just-noticeable difference (JND) for multimedia","authors":"Lin, Ghinea ;, Weisi Lin, Gheorghita Ghinea","year":"2022","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2022] Hanhe Lin, Guangan Chen, Mohsen Je- nadeleh, Vlad Hosu, Ulf-Dietrich Reips, Raouf Hamzaoui, and Dietmar Saupe. Large-scale crowdsourced subjective assessment of picturewise just noticeable difference. IEEE Transactions on Circuits and Systems for Video Technol- ogy, 32(9):5859-5873, 2022.","title":"Large-scale crowdsourced subjective assessment of picturewise just noticeable difference","authors":"Lin","year":"2022","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2010] Anmin Liu, Weisi Lin, Manoranjan Paul, Chenwei Deng, and Fan Zhang. Just noticeable difference for images with decomposition model for separating edge and textured regions. IEEE Transactions on Circuits and Systems for Video Technology, 20(11):1648-1652, 2010.","title":"Just noticeable difference for images with decomposition model for separating edge and textured regions","authors":"Liu","year":"2010","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2019] Huanhua Liu, Yun Zhang, Huan Zhang, Chunling Fan, Sam Kwong, C-C Jay Kuo, and Xiaoping Fan. Deep learning-based picture-wise just noticeable dis- tortion prediction model for image compression. IEEE Transactions on Image Processing, 29:641-656, 2019.","title":"Deep learning-based picture-wise just noticeable distortion prediction model for image compression","authors":"Huanhua Liu, Yun Zhang, Huan Zhang, Chunling Fan, Sam Kwong, C-C Jay Kuo, Xiaoping Fan","year":"2019","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE International Conference on Computer Vision (ICCV), pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2023] Yaxuan Liu, Jian Jin, Yuan Xue, and Weisi Lin. The First Comprehensive Dataset with Multiple Distortion Types for Visual Just-Noticeable Differences. arXiv preprint arXiv:2303.02562, pages 1-6, 2023.","title":"The First Comprehensive Dataset with Multiple Distortion Types for Visual Just-Noticeable Differences","authors":"Yaxuan Liu, Jian Jin, Yuan Xue, Weisi Lin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2022] Ruijun Ma, Shuyi Li, Bob Zhang, Leyuan Fang, and Zhengming Li. Flexible and generalized real photograph denoising exploiting dual meta attention. IEEE Transactions on Cybernetics, pages 1-13, 2022.","title":"Flexible and generalized real photograph denoising exploiting dual meta attention","authors":"Ruijun Ma, Shuyi Li, Bob Zhang, Leyuan Fang, Zhengming Li","year":"2022","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nami et al., 2022] Sanaz Nami, Farhad Pakdaman, Mah- moud Reza Hashemi, and Shervin Shirmohammadi. BL- JUNIPER: A CNN-assisted framework for perceptual video coding leveraging block-level JND. IEEE Trans- actions on Multimedia, pages 1-16, 2022.","title":"BL-JUNIPER: A CNN-assisted framework for perceptual video coding leveraging block-level JND","authors":"Sanaz Nami, Farhad Pakdaman, Mahmoud Reza Hashemi, Shervin Shirmohammadi","year":"2022","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nikzad et al., 2021] Mohammad Nikzad, Yongsheng Gao, and Jun Zhou. Attention-based Pyramid Dilated Lattice Network for Blind Image Denoising. In International Con- ference on International Joint Conferences on Artificial In- telligence (IJCAI), pages 931-937, 2021.","title":"Attention-based Pyramid Dilated Lattice Network for Blind Image Denoising","authors":"Mohammad Nikzad, Yongsheng Gao, Jun Zhou","year":"2021","venue":"International Conference on International Joint Conferences on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2021/129"},{"position":19,"raw":"[Qi et al., 2023] Linfeng Qi, Jiahao Li, Bin Li, Houqiang Li, and Yan Lu. Motion Information Propagation for Neural Video Compression. In IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 6111-6120, 2023.","title":"Motion Information Propagation for Neural Video Compression","authors":"Qi","year":"2023","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qin et al., 2019] Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jager- sand. Basnet: Boundary-aware salient object detection. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 7479-7489, 2019.","title":"Basnet: Boundary-aware salient object detection","authors":"Qin","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qu et al., 2023] Xinghua Qu, Xiang Yin, Pengfei Wei, Lu Lu, and Zejun Ma. AudioQR: deep neural audio wa- termarks for QR code. In International Conference on International Joint Conferences on Artificial Intelligence (IJCAI), pages 6192-6200, 2023.","title":"AudioQR: deep neural audio watermarks for QR code","authors":"Qu","year":"2023","venue":"International Conference on International Joint Conferences on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2023/687"},{"position":22,"raw":"[Ranftl et al., 2021] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense predic- tion. In IEEE International Conference on Computer Vi- sion (ICCV), pages 12179-12188, 2021.","title":"René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction","authors":"Ranftl","year":"2021","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shen et al., 2020] Xuelin Shen, Zhangkai Ni, Wenhan Yang, Xinfeng Zhang, Shiqi Wang, and Sam Kwong. Just noticeable distortion profile inference: A patch-level struc- tural visibility learning approach. IEEE Transactions on Image Processing, 30:26-38, 2020.","title":"Just noticeable distortion profile inference: A patch-level structural visibility learning approach","authors":"Shen","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tian et al., 2020] Tao Tian, Hanli Wang, Lingxuan Zuo, C- C Jay Kuo, and Sam Kwong. Just noticeable difference level prediction for perceptual image compression. IEEE Transactions on Broadcasting, 66(3):690-700, 2020.","title":"Just noticeable difference level prediction for perceptual image compression","authors":"Tao Tian, Hanli Wang, Lingxuan Zuo, C-C Jay Kuo, Sam Kwong","year":"2020","venue":"IEEE Transactions on Broadcasting","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2016] Haiqiang Wang, Weihao Gan, Sudeng Hu, Joe Yuchieh Lin, Lina Jin, Longguang Song, Ping Wang, Ioannis Katsavounidis, Anne Aaron, and C-C Jay Kuo. MCL-JCV: a JND-based H. 264/AVC video quality assessment dataset. In IEEE International Conference on Image Processing (ICIP), pages 1509-1513, 2016.","title":"MCL-JCV: a JND-based H. 264/AVC video quality assessment dataset","authors":"Wang","year":"2016","venue":"IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Menglu Wang, Xueyang Fu, Zepei Sun, and Zheng-Jun Zha. JPEG artifacts removal via compres- sion quality ranker-guided networks. In International Con- ference on International Joint Conferences on Artificial In- telligence (IJCAI), pages 566-572, 2021.","title":"JPEG artifacts removal via compression quality ranker-guided networks","authors":"Wang","year":"2021","venue":"International Conference on International Joint Conferences on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2020/79"},{"position":27,"raw":"[Wang et al., 2022] Miaohui Wang, Zhuowei Xu, Xueqin Liu, Jian Xiong, and Wuyuan Xie. Perceptually quasi- lossless compression of screen content data via visibility modeling and deep forecasting. IEEE Transactions on In- dustrial Informatics, 18(10):6865-6875, 2022.","title":"Perceptually quasilossless compression of screen content data via visibility modeling and deep forecasting","authors":"Wang","year":"2022","venue":"IEEE Transactions on Industrial Informatics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2013] Jinjian Wu, Guangming Shi, Weisi Lin, Anmin Liu, and Fei Qi. Just noticeable difference estima- tion for images with free-energy principle. IEEE Transac- tions on Multimedia, 15(7):1705-1710, 2013.","title":"Just noticeable difference estimation for images with free-energy principle","authors":"Wu","year":"2013","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2017] Jinjian Wu, Leida Li, Weisheng Dong, Guangming Shi, Weisi Lin, and C-C Jay Kuo. Enhanced just noticeable difference model for images with pattern complexity. IEEE Transactions on Image Processing, 26(6):2682-2693, 2017.","title":"Enhanced just noticeable difference model for images with pattern complexity","authors":"Jinjian Wu, Leida Li, Weisheng Dong, Guangming Shi, Weisi Lin, C-C Jay Kuo","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2020] Yuhao Wu, Weiping Ji, and Jinjian Wu. Unsupervised deep learning for just noticeable difference estimation. In IEEE International Conference on Multi- media & Expo Workshops (ICMEW), pages 1-6, 2020.","title":"Unsupervised deep learning for just noticeable difference estimation","authors":"Yuhao Wu, Weiping Ji, Jinjian Wu","year":"2020","venue":"IEEE International Conference on Multimedia & Expo Workshops (ICMEW)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xie et al., 2023] Wuyuan Xie, Shukang Wang, Sukun Tian, Lirong Huang, Ye Liu, and Miaohui Wang. Just Noticeable Visual Redundancy Forecasting: A Deep Multimodal- driven Approach. AAAI Conference on Artificial Intelli- gence (AAAI), pages 2965-2973, 2023.","title":"Just Noticeable Visual Redundancy Forecasting: A Deep Multimodaldriven Approach","authors":"Wuyuan Xie, Shukang Wang, Sukun Tian, Lirong Huang, Ye Liu, Miaohui Wang","year":"2023","venue":"AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2005] Xiaokang Yang, Weisi Lin, Zhongkhang Lu, Ee Ping Ong, and Susu Yao. Just noticeable distortion model and its applications in video coding. Elsevier Signal processing: Image Communication, 20(7):662-680, 2005.","title":"Just noticeable distortion model and its applications in video coding","authors":"Xiaokang Yang, Weisi Lin, Zhongkhang Lu, Ee, Ping Ong, Susu Yao","year":"2005","venue":"Elsevier Signal processing: Image Communication","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Ye et al., 2021] Shuquan Ye, Dongdong Chen, Songfang Han, Ziyu Wan, and Jing Liao. Meta-PU: An arbitrary- scale upsampling network for point cloud. IEEE Transactions on Visualization and Computer Graphics, 28(9):3206-3218, 2021.","title":"Meta-PU: An arbitraryscale upsampling network for point cloud","authors":"Ye","year":"2021","venue":"IEEE Transactions on Visualization and Computer Graphics","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yin et al., 2023] Haibing Yin, Hongkui Wang, Li Yu, Jun- hui Liang, and Guangtao Zhai. Feedforward and Feedback Modulations Based Foveated JND Estimation for Images. ACM Transactions on Multimedia Computing, Communi- cations and Applications, 19(5):1-23, 2023.","title":"Feedforward and Feedback Modulations Based Foveated JND Estimation for Images","authors":"Yin","year":"2023","venue":"ACM Transactions on Multimedia Computing, Communications and Applications","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 586-595, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, Oliver Wang","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2021] Gongjie Zhang, Zhipeng Luo, Kaiwen Cui, and Shijian Lu. Meta-DETR: Image-level few-shot object detection with inter-class correlation exploitation. arXiv preprint arXiv:2103.11731, pages 1-14, 2021.","title":"Meta-DETR: Image-level few-shot object detection with inter-class correlation exploitation","authors":"Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}