Few-shot learning for dermatological conditions with Lesion Area Aware Swin Transformer
Yonggong Ren
School of Computer and Information Technology, Liaoning Normal University, Dalian, China
Search for more papers by this authorWenqiang Xu
School of Computer and Information Technology, Liaoning Normal University, Dalian, China
Search for more papers by this authorYuanxin Mao
School of Computer and Information Technology, Liaoning Normal University, Dalian, China
Search for more papers by this authorYuechu Wu
School of Computer and Information Technology, Liaoning Normal University, Dalian, China
Search for more papers by this authorCorresponding Author
Bo Fu
School of Computer and Information Technology, Liaoning Normal University, Dalian, China
Correspondence
Bo Fu, School of Computer and Information Technology, Liaoning Normal University, Dalian, China.
Email: [email protected]
Dang N. H. Thanh, Department of Information Technology, School of Business Information Technology, University of Economics Ho Chi Minh City, Ho Chi Minh City, Vietnam.
Email: [email protected]
Search for more papers by this authorCorresponding Author
Dang N. H. Thanh
Department of Information Technology, School of Business Information Technology, University of Economics Ho Chi Minh City, Ho Chi Minh City, Vietnam
Correspondence
Bo Fu, School of Computer and Information Technology, Liaoning Normal University, Dalian, China.
Email: [email protected]
Dang N. H. Thanh, Department of Information Technology, School of Business Information Technology, University of Economics Ho Chi Minh City, Ho Chi Minh City, Vietnam.
Email: [email protected]
Search for more papers by this authorYonggong Ren
School of Computer and Information Technology, Liaoning Normal University, Dalian, China
Search for more papers by this authorWenqiang Xu
School of Computer and Information Technology, Liaoning Normal University, Dalian, China
Search for more papers by this authorYuanxin Mao
School of Computer and Information Technology, Liaoning Normal University, Dalian, China
Search for more papers by this authorYuechu Wu
School of Computer and Information Technology, Liaoning Normal University, Dalian, China
Search for more papers by this authorCorresponding Author
Bo Fu
School of Computer and Information Technology, Liaoning Normal University, Dalian, China
Correspondence
Bo Fu, School of Computer and Information Technology, Liaoning Normal University, Dalian, China.
Email: [email protected]
Dang N. H. Thanh, Department of Information Technology, School of Business Information Technology, University of Economics Ho Chi Minh City, Ho Chi Minh City, Vietnam.
Email: [email protected]
Search for more papers by this authorCorresponding Author
Dang N. H. Thanh
Department of Information Technology, School of Business Information Technology, University of Economics Ho Chi Minh City, Ho Chi Minh City, Vietnam
Correspondence
Bo Fu, School of Computer and Information Technology, Liaoning Normal University, Dalian, China.
Email: [email protected]
Dang N. H. Thanh, Department of Information Technology, School of Business Information Technology, University of Economics Ho Chi Minh City, Ho Chi Minh City, Vietnam.
Email: [email protected]
Search for more papers by this authorAbstract
Skin is the largest organ of the human body and participates in the functional activities of the human body all the time. Therefore, human beings have a large risk of getting skin diseases. The diseased skin lesion image shows visually different characteristics from the normal skin image, and sometimes unusual skin color may indicate human viscera or autoimmune issues. However, the current recognition and classification of dermatological conditions still rely on expert visual diagnosis rather than a visual algorithm. This is because there are many kinds of lesion features of skin diseases, and the lesion accounts for a small proportion of the skin image, so it is difficult to learn the required lesion features; meanwhile, some dermatology images have too few samples to deal with the problem of small samples. In view of the above limitations, we propose a model named Lesion Area Aware Shifted windows Transformer for dermatological conditions classification rely on the powerful performance and excellent result of Swin transformer recently proposed. For brief notation, we use its abbreviation later. Our main contributions are as follows. First, we modify the Swin transformer and use it in the automatic classification of dermatological conditions. Using the self-attention mechanism of the transformer, our method can mine more long-distance correlations between diseased tissue image features. Using its shifting windows, we can fuse local features and global features, so it is possible to get better classification results with a flexible receptive field. Second, we use a skip connection to grasp and reinforce global features from the previous block and use Swin transformer to extract detailed local features, which will excavate and merge global features and local features further. Third, considering Swin transformer is a lightweight model compared with traditional transformers, our model is compact for deployment and more favorable to resource-strict medical devices.
CONFLICT OF INTEREST STATEMENT
The authors declare no conflicts of interest.
Open Research
DATA AVAILABILITY STATEMENT
The data that support the findings of this study are openly available in SD-198 and Fitzpatrick 17k datasets.
REFERENCES
- 1Khan MA, Akram T, Zhang Y-D, et al. SkinNet-ENDO: multiclass skin lesion recognition using deep neural network and entropy-normal distribution optimization algorithm with ELM. Int J Imaging Syst Technol. 2023: 1-18. doi:10.1002/ima.22863
- 2Codella NCF, Gutman D, Celebi ME, et al. Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (ISIC). In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018, pp. 168-172.
- 3Sampogna F, Finlay AY, Salek SS, et al. Measuring the impact of dermatological conditions on family and caregivers: a review of dermatology-specific instruments. J Eur Acad Dermatol Venereol. 2017; 31(9): 1429-1439.
- 4Abbasi NR, Shaw HM, Rigel DS, et al. Early diagnosis of cutaneous melanoma: revisiting the abcd criteria. JAMA. 2004; 292(22): 2771-2776.
- 5Stolz W. Abcd rule of dermatoscopy: a new practical method for early recognition of malignant melanoma. Eur J Dermatol. 1994; 4: 521-527.
- 6Pagnanelli G, Soyer HP, Argenziano G, et al. Diagnosis of pigmented skin lesions by dermoscopy: web-based training improves diagnostic performance of non-experts. Br J Dermatol. 2003; 148(4): 698-702.
- 7 IEEE Journal of Biomedical and Health Informatics. Special issue on skin image analysis in the age of deep learning. https://www.embs.org/jbhi/special-issues-page/special-issue-on-skin-image-analysis-in-the-age-of-deep-learning/.
- 8Bibi A, Khan MA, Javed MY, et al. Skin Lesion Segmentation and Classification Using Conventional and Deep Learning Based Framework. 2022.
- 9Arshad M, Khan MA, Tariq U, et al. A computer-aided diagnosis system using deep learning for multiclass skin lesion classification. Comput Intell Neurosci. 2021; 2021: 1-15.
- 10Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017; 542(7639): 115-118.
- 11Yu L, Chen H, Dou Q, Qin J, Heng P-A. Automated melanoma recognition in dermoscopy images via very deep residual networks. IEEE Trans Med Imaging. 2016; 36(4): 994-1004.
- 12Kawahara J, Daneshvar S, Argenziano G, Hamarneh G. Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE J Biomed Health Inform. 2018; 23(2): 538-546.
- 13Das Gupta M, Srinivasa S, Antony M, et al. Kl divergence based agglomerative clustering for automated vitiligo grading. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 2700-2709.
- 14Codella NC, Gutman D, Celebi ME, et al. Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE; 2018: 168-172.
10.1109/ISBI.2018.8363547 Google Scholar
- 15Hosny KM, Kassem MA. Refined residual deep convolutional network for skin lesion classification. J Digit Imaging. 2022; 35: 258-280.
- 16Yang J, Sun X, Liang J, Rosin PL. Clinical skin lesion diagnosis using representations inspired by dermatologist criteria. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1258-1266.
- 17Attique Khan M, Sharif M, Akram T, Kadry S, Hsu C-H. A two-stream deep neural network-based intelligent system for complex skin cancer types classification. Int J Intell Syst. 2022; 37(12): 10621-10649.
- 18Khan MA, Sharif M, Akram T, Damaševičius R, Maskeliūnas R. Skin lesion segmentation and multiclass classification using deep learning features and improved moth flame optimization. Diagnostics. 2021; 11(5): 811.
- 19Nichol A, Achiam J, Schulman J. On first-order meta-learning algorithms. arXiv Preprint arXiv:1803.02999. 2018.
- 20Ravi S, Larochelle H. Optimization as a Model for Few-Shot Learning. 2016.
- 21Santoro A, Bartunov S, Botvinick M, Wierstra D, Lillicrap T. Meta-learning with memory-augmented neural networks. International Conference on Machine Learning. PMLR; 2016: 1842-1850.
- 22Munkhdalai T, Yu H. Meta networks. International Conference on Machine Learning. PMLR; 2017: 2554-2563.
- 23Chopra S, Hadsell R, LeCun Y. Learning a similarity metric discriminatively, with application to face verification. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). Vol 1. IEEE; 2005: 539-546.
10.1109/CVPR.2005.202 Google Scholar
- 24Vinyals O, Blundell C, Lillicrap T, Wierstra D, et al. Matching networks for one shot learning. Adv Neural Inform Process Syst. 2016; 29: 3630-3638.
- 25Snell J, Swersky K, Zemel RS. Prototypical networks for few-shot learning. arXiv Preprint arXiv:1703.05175. 2017.
- 26Sung F, Yang Y, Zhang L, Xiang T, Torr PH, Hospedales TM. Learning to compare: relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1199-1208.
- 27Mahajan K, Sharma M, Vig L. Meta-dermdiagnosis: few-shot skin disease identification using meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 730-731.
- 28Afza F, Sharif M, Khan MA, Tariq U, Yong H-S, Cha J. Multiclass skin lesion classification using hybrid deep features selection and extreme learning machine. Sensors. 2022; 22(3): 799.
- 29Khan MA, Muhammad K, Sharif M, Akram T, de Albuquerque VHC. Multi-class skin lesion detection and classification via teledermatology. IEEE J Biomed Health Inform. 2021; 25(12): 4267-4275.
- 30Khan MA, Sharif MI, Raza M, Anjum A, Saba T, Shad SA. Skin lesion segmentation and classification: a unified framework of deep neural network features fusion and selection. Expert Syst. 2022; 39(7):e12497.
- 31Yosinski J, Clune J, Bengio Y, Lipson H. How transferable are features in deep neural networks?. arXiv Preprint arXiv:1411.1792. 2014.
- 32Long M, Cao Y, Wang J, Jordan M. Learning transferable features with deep adaptation networks. International Conference on Machine Learning. PMLR; 2015: 97-105.
- 33Ge W, Yu Y. Borrowing treasures from the wealthy: deep transfer learning through selective joint fine-tuning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1086-1095.
- 34Zamir AR, Sax A, Shen W, Guibas LJ, Malik J, Savarese S. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3712-3722.
- 35Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). Curran Associates Inc., Red Hook, NY, USA, 2017: 6000-6010.
- 36Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S. End-to-end object detection with transformers. European Conference on Computer Vision. Springer; 2020: 213-229.
10.1007/978-3-030-58452-8_13 Google Scholar
- 37Zhu X, Su W, Lu L, Li B, Wang X, Dai J. Deformable detr: Deformable transformers for end-to-end object detection. arXiv Preprint arXiv:2010.04159. 2020.
- 38Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16 × 16 words: transformers for image recognition at scale. arXiv Preprint arXiv:2010.11929. 2020.
- 39Liu Z, Lin Y, Cao Y, et al. Swin transformer: hierarchical vision transformer using shifted windows. arXiv Preprint arXiv:2103.14030. 2021.
- 40Chen P, Li W, Lu J. Data-efficient instance segmentation with a single gpu. 2021.
- 41Henkel C. Efficient large-scale image retrieval with deep feature orthogonality and hybrid-swin-transformers. 2021.
- 42Zhu F, Zhao S, Wang P, Wang H, Yan H, Liu S. Semi-supervised wide-angle portraits correction by multi-scale transformer. 2021.
- 43Liang J, Cao J, Sun G, Zhang K, Gool LV, Timofte R. Swinir: Image restoration using swin transformer. 2021.
- 44Haggenmüller S, Maron RC, Hekler A, et al. Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts. Eur J Cancer. 2021; 156: 202-216.
- 45Tschandl P, Rosendahl C, Kittler H. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci Data. 2018; 5(1): 1-9.
- 46Codella N, Rotemberg V, Tschandl P, et al. Skin lesion analysis toward melanoma detection 2018: a challenge hosted by the international skin imaging collaboration (isic). arXiv Preprint arXiv:1902.03368. 2019.
- 47Combalia M, Codella NC, Rotemberg V, et al. Bcn20000: Dermoscopic lesions in the wild. arXiv Preprint arXiv:1908.02288. 2019.
- 48Rotemberg V, Kurtansky N, Betz-Stablein B, et al. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Sci Data. 2021; 8(1): 1-8.
- 49Mendonça T, Ferreira PM, Marques JS, Marcal AR, Rozeira J. Ph 2-a dermoscopic image database for research and benchmarking. 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE; 2013: 5437-5440.
10.1109/EMBC.2013.6610779 Google Scholar
- 50Giotis I, Molders N, Land S, Biehl M, Jonkman MF, Petkov N. Med-node: a computer-assisted melanoma diagnosis system using non-dermoscopic images. Expert Syst Appl. 2015; 42(19): 6578-6585.
- 51Sun X, Yang J, Sun M, Wang K. A benchmark for automatic visual classification of clinical skin disease images. European Conference on Computer Vision. Springer; 2016: 206-222.
10.1007/978-3-319-46466-4_13 Google Scholar
- 52Pacheco AG, Lima GR, Salomão AS, et al. Pad-ufes-20: a skin lesion dataset composed of patient data and clinical images collected from smartphones. Data Brief. 2020; 32:106221.
- 53Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16 × 16 words: Transformers for Image Recognition at Scale. Proceedings of Ninth International Conference on Learning Representations. ICLR; 2021.
- 54Zhao D, Li J, Li H, Xu L. Hybrid local-global transformer for image dehazing. arXiv Preprint arXiv:2109.07100. 2021.
- 55He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778.
- 56Glorot X, Bordes A, Bengio Y. Deep sparse rectifier neural networks. J Mach Learn Res. 2011; 15: 315-323.