{"type":"ScholarlyArticle","title":"3E-Solver: An Effortless, Easy-to-Update, and End-to-End Solver with Semi-Supervised Learning for Breaking Text-Based Captchas","authors":[{"name":"Xianwen Deng","slug":"xianwen-deng","given":"Xianwen","family":"Deng","affiliation":"Shanghai Jiao Tong University , Shanghai , China"},{"name":"Ruijie Zhao","slug":"ruijie-zhao","given":"Ruijie","family":"Zhao","affiliation":"Shanghai Jiao Tong University , Shanghai , China"},{"name":"Yanhao Wang","slug":"yanhao-wang","given":"Yanhao","family":"Wang","affiliation":"NIO Security Research , Shanghai , China"},{"name":"Libo Chen","slug":"libo-chen","given":"Libo","family":"Chen","affiliation":"Shanghai Jiao Tong University , Shanghai , China"},{"name":"Yijun Wang","slug":"yijun-wang","given":"Yijun","family":"Wang","affiliation":"NIO Security Research , Shanghai , China"},{"name":"Zhi Xue","slug":"zhi-xue","given":"Zhi","family":"Xue","affiliation":"Shanghai Jiao Tong University , Shanghai , China"}],"abstract":"Text-based captchas are the most widely used security mechanism currently. Due to the limitations and specificity of the segmentation algorithm, the early segmentation-based attack method has been unable to deal with the current captchas with newly introduced security features (e.g., occluding lines and overlapping). Recently, some works have designed captcha solvers based on deep learning methods with powerful feature extraction capabilities, which have greater generality and higher accuracy. However, these works still suffer from two main intrinsic limitations: (1) many labor costs are required to label the training data, and (2) the solver cannot be updated with unlabeled data to recognize captchas more accurately. In this paper, we present a novel solver using improved FixMatch for semi-supervised captcha recognition to tackle these problems. Specifically, we first build an end-to-end baseline model to effectively break text-based captchas by leveraging encoder-decoder architecture and attention mechanism. Then we construct our solver with a few labeled samples and many unlabeled samples by improved FixMatch, which introduces teacher forcing, adaptive batch normalization, and consistency loss to achieve more effective training. Experiment results show that our solver outperforms state-of-the-arts by a large margin on current captcha schemes. We hope that our work can help security experts to revisit the design and usability of text-based captchas. The source code of this work is available at https://github.com/SJTU-dxw/3E-Solver-CAPTCHA.","doi":"10.24963/ijcai.2022/530","url":"https://ijcai.org/proceedings/2022/530","pdf":"https://ijcai.org/proceedings/2022/0530.pdf","markdown":"https://ijcai.org/proceedings/2022/530.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/530","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":530,"pages":{"first":3817,"last":3824},"keywords":["Multidisciplinary Topics and Applications: Security and Privacy","Multidisciplinary Topics and Applications: Other"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986769","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984912","length_sec":352,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bursztein et al., 2011] Elie Bursztein, Matthieu Martin, and John Mitchell. Text-based captcha strengths and weak- nesses. In CCS, pages 125-138, 2011.","title":"Text-based captcha strengths and weaknesses","authors":"Bursztein","year":"2011","venue":"CCS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Converse, 2005] Tim Converse. Captcha generation as a web service. In Human Interactive Proofs, pages 82-96, 2005.","title":"Captcha generation as a web service","authors":"Tim Converse","year":"2005","venue":"Human Interactive Proofs","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gao et al., 2010] Haichang Gao, Honggang Liu, Dan Yao, Xiyang Liu, and Uwe Aickelin. An audio captcha to dis- tinguish humans from computers. In International Sym- posium on Electronic Commerce and Security, pages 265- 269, 2010.","title":"An audio captcha to distinguish humans from computers","authors":"Gao","year":"2010","venue":"International Symposium on Electronic Commerce and Security","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2013] Haichang Gao, Wei Wang, Jiao Qi, Xuqin Wang, Xiyang Liu, and Jeff Yan. The robustness of hollow captchas. In CCS, pages 1075-1086, 2013.","title":"The robustness of hollow captchas","authors":"Gao","year":"2013","venue":"CCS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2015] Haichang Gao, Xuqin Wang, Fang Cao, Zhengya Zhang, Lei Lei, Jiao Qi, and Xiyang Liu. Ro- bustness of text-based completely automated public turing test to tell computers and humans apart. IET Information Security, 10(1):45-52, 2015.","title":"Robustness of text-based completely automated public turing test to tell computers and humans apart","authors":"Gao","year":"2015","venue":"IET Information Security","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2017] Haichang Gao, Mengyun Tang, Yi Liu, Ping Zhang, and Xiyang Liu. Research on the security of microsoft's two-layer captcha. IEEE Transactions on In- formation Forensics and Security, 12(7):1671-1685, 2017.","title":"Research on the security of microsoft's two-layer captcha","authors":"Gao","year":"2017","venue":"IEEE Transactions on Information Forensics and Security","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[George et al., 2017] Dileep George, Wolfgang Lehrach, Ken Kansky, Miguel Lázaro-Gredilla, Christopher Laan, Bhaskara Marthi, Xinghua Lou, Zhaoshi Meng, Yi Liu, Huayan Wang, et al. A generative vision model that trains with high data efficiency and breaks text-based captchas. Science, 358(6368), 2017.","title":"A generative vision model that trains with high data efficiency and breaks text-based captchas","authors":"Dileep George, Wolfgang Lehrach, Ken Kansky, Miguel Lázaro-Gredilla, Christopher Laan, Bhaskara Marthi, Xinghua Lou, Zhaoshi Meng, Yi Liu, Huayan Wang","year":null,"venue":"Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gossweiler et al., 2009] Rich Gossweiler, Maryam Kamvar, and Shumeet Baluja. What's up captcha? a captcha based on image orientation. In WWW, pages 841-850, 2009.","title":"What's up captcha? a captcha based on image orientation","authors":"Gossweiler","year":"2009","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Le et al., 2017] Tuan Anh Le, Atilim Giines ¸Baydin, Robert Zinkov, and Frank Wood. Using synthetic data to train neural networks is model-based reasoning. In IJCNN, pages 3514-3521, 2017.","title":"Using synthetic data to train neural networks is model-based reasoning","authors":"Anh Tuan, Atilim Le, Robert Giines ¸baydin, Frank Zinkov, Wood","year":"2017","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2021] Chunhui Li, Xingshu Chen, Haizhou Wang, Peiming Wang, Yu Zhang, and Wenxian Wang. End- to-end attack on text-based captchas based on cycle- consistent generative adversarial network. Neurocomput- ing, 433:223-236, 2021.","title":"Endto-end attack on text-based captchas based on cycleconsistent generative adversarial network","authors":"Chunhui Li, Xingshu Chen, Haizhou Wang, Peiming Wang, Yu Zhang, Wenxian Wang","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mori and Malik, 2003] Greg Mori and Jitendra Malik. Rec- ognizing objects in adversarial clutter: Breaking a visual captcha. In CVPR, 2003.","title":"Recognizing objects in adversarial clutter: Breaking a visual captcha","authors":"Malik ; Mori, Greg Mori, Jitendra Malik","year":"2003","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recogni- tion challenge. International Journal of Computer Vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Stark et al., 2015] Fabian Stark, Caner Hazırbas, Rudolph Triebel, and Daniel Cremers. Captcha recognition with ac- tive deep learning. In Workshop New Challenges in Neural Computation, volume 2015, page 94, 2015.","title":"Captcha recognition with active deep learning","authors":"Fabian Stark, Caner Hazırbas, Rudolph Triebel, Daniel Cremers","year":"2015","venue":"Workshop New Challenges in Neural Computation","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tang et al., 2018] Mengyun Tang, Haichang Gao, Yang Zhang, Yi Liu, Ping Zhang, and Ping Wang. Research on deep learning techniques in breaking text-based captchas and designing image-based captcha. IEEE Transactions on Information Forensics and Security, 13(10):2522-2537, 2018.","title":"Research on deep learning techniques in breaking text-based captchas and designing image-based captcha","authors":"Tang","year":"2018","venue":"IEEE Transactions on Information Forensics and Security","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tarvainen and Valpola, 2017] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. In NIPS, pages 1195-1204, 2017.","title":"Mean teachers are better role models: Weightaveraged consistency targets improve semi-supervised deep learning results","authors":"Valpola Tarvainen, Antti Tarvainen, Harri Valpola","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tian and Xiong, 2020] Sheng Tian and Tao Xiong. A generic solver combining unsupervised learning and rep- resentation learning for breaking text-based captchas. In WWW, pages 860-871, 2020.","title":"A generic solver combining unsupervised learning and representation learning for breaking text-based captchas","authors":"Sheng Tian, Tao Xiong","year":"2020","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Von Ahn et al., 2003] Luis Von Ahn, Manuel Blum, Nicholas J Hopper, and John Langford. Captcha: Using hard ai problems for security. In International Confer- ence on The Theory and Applications of Cryptographic Techniques, pages 294-311, 2003.","title":"Captcha: Using hard ai problems for security","authors":"Luis Von Ahn, Manuel Blum, Nicholas J Hopper, John Langford","year":"2003","venue":"International Conference on The Theory and Applications of Cryptographic Techniques","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yan and El Ahmad, 2007] Jeff Yan and Ahmad Salah El Ahmad. Breaking visual captchas with naive pattern recognition algorithms. In Annual Computer Security Ap- plications Conference (ACSAC), pages 279-291, 2007.","title":"Breaking visual captchas with naive pattern recognition algorithms","authors":"Yan, El Ahmad, ;, Jeff Yan, Ahmad Salah, El Ahmad","year":"2007","venue":"Annual Computer Security Applications Conference (ACSAC)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yan and El Ahmad, 2008] Jeff Yan and Ahmad Salah El Ahmad. A low-cost attack on a microsoft captcha. In CCS, pages 543-554, 2008.","title":"A low-cost attack on a microsoft captcha","authors":"Yan, El Ahmad, Jeff Yan, Ahmad Salah, El Ahmad","year":"2008","venue":"CCS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ye et al., 2018] Guixin Ye, Zhanyong Tang, Dingyi Fang, Zhanxing Zhu, Yansong Feng, Pengfei Xu, Xiaojiang Chen, and Zheng Wang. Yet another text captcha solver: A generative adversarial network based approach. In CCS, pages 332-348, 2018.","title":"Yet another text captcha solver: A generative adversarial network based approach","authors":"Ye","year":"2018","venue":"CCS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhan et al., 2018] Hongjian Zhan, Shujing Lyu, and Yue Lu. Handwritten digit string recognition using convolu- tional neural network. In International Conference on Pat- tern Recognition (ICPR), pages 3729-3734, 2018.","title":"Handwritten digit string recognition using convolutional neural network","authors":"Hongjian Zhan, Shujing Lyu, Yue Lu","year":"2018","venue":"International Conference on Pattern Recognition (ICPR)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zi et al., 2019] Yang Zi, Haichang Gao, Zhouhang Cheng, and Yi Liu. An end-to-end attack on text captchas. IEEE Transactions on Information Forensics and Security, 15:753-766, 2019.","title":"An end-to-end attack on text captchas","authors":"Yang Zi, Haichang Gao, Zhouhang Cheng, Yi Liu","year":"2019","venue":"IEEE Transactions on Information Forensics and Security","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}