Unsupervised domain adaptation (UDA) techniques address the challenge of domain shift in medical imaging by transferring knowledge from labeled source data to unlabeled target data, accounting for discrepancies in image protocols and patient demographics. However, conventional UDA methods predominantly focus on either coarse-grained feature or fine-grained feature alignment, lacking systematic integration of multi-granular feature alignment. This results in insufficient alignment of semantic information across varying feature granularities and consequently limits their performance. To tackle these issues, we propose a novel UDA framework, the Multi-Granularity Feature Alignment Network (MGFA), comprising two core modules: (1) Dual Feature Disentanglement Learning module (DFDL), which employs disentanglement learning to guide the network in learning discriminative domain-invariant features; and (2) Fine-Grained Feature Alignment module (FGFA), which leverages graph convolution to provide fine-granularity information for feature alignment. Our method has been extensively evaluated on adrenal, pulmonary and bladder cancer diagnostics, bridging the heterogeneity gap across different medical centers. The proposed MGFA outperforms state-of-the-art approaches on three cross-center medical image classification datasets, demonstrating superior generalization and diagnostic performance. Moreover, the MGFA significantly impacts AI-assisted physician diagnosis by providing robust decision support, enhancing diagnostic efficiency, and offering more reliable guidance in complex or challenging cases.