Transformer-Based Multi-Modal Learning for Medical Image Analysis: A Comparative Study of Vision Transformers and CNN Architectures

Medical image analysis has been revolutionized by deep learning approaches, with Convolutional Neural Networks (CNNs) serving as the backbone architecture for most state-of-the-art systems. Recently, Vision Transformers (ViTs) have emerged as powerful alternatives, demonstrating remarkable performance across various computer vision tasks. This research presents a comprehensive comparative study of Vision Transformers and CNNs for multi-modal medical image analysis across four critical tasks: classification, segmentation, detection, and reconstruction. We evaluate ۷ distinct architectures (۳ CNN-based and ۴ transformer-based models) on diverse medical imaging datasets spanning MRI, CT, X-ray, and histopathology images. Our experiments demonstrate that transformer-based architectures achieve superior performance in complex tasks requiring global contextual understanding, with an average improvement of ۴.۳% in segmentation accuracy and ۳.۸% in classification tasks compared to CNN counterparts. However, CNNs maintain advantages in efficiency, with ۲.۲× faster inference times and ۱.۸× lower memory requirements. To leverage the strengths of both approaches, we propose a novel hybrid architecture that combines local feature extraction capabilities of CNNs with the long-range dependency modeling of transformers, achieving state-of-the-art performance across all evaluated tasks while maintaining computational efficiency. Our findings provide valuable insights into the appropriate architectural choices for different medical imaging applications and resource constraints.

کلیدواژه ها:

Vision Transformer ، Convolutional Neural Network ، Medical Image Analysis ، Multi-Modal Learning ، Hybrid Architecture ، Deep Learning