Accurate classification of colorectal polyps (CRPs) is critical for the early diagnosis and treatment of colorectal cancer (CRC). This paper presents an efficient deep learning method specifically developed to enhance the accuracy of CRPs classification, thereby assisting physicians in making informed decisions. Drawing inspiration from the sequential procedure of colonoscopy, where endoscopists first locate polyps and then proceed to detailed observations and diagnoses, we developed a novel multi-stage classification network. This network cascades several convolutional neural networks (CNNs) to mimic the gradual increase in diagnostic specificity seen in clinical settings. Furthermore, we introduced a novel attention module, the Cross-Stage Weighted Attention (CSWA), designed to amplify the effectiveness of multi-stage feature fusion by focusing on the most informative features across different stages. To train and validate our proposed network, we curated a dataset consisting of 2568 white light endoscopic images. Facing a significant class imbalance, particularly in the underrepresented categories of villous and serrated adenomas, we employed Generative Adversarial Network Augmentation (GAN-Aug) to synthesize additional images, thereby ensuring a more balanced dataset for training. An assessment by six endoscopists confirmed the high realism of polyp characteristics in the images generated by GAN-Aug. Subsequent quantitative evaluation of our CSWA-enhanced multi-stage classification network on this augmented dataset achieved an accuracy of 0.832 +/- 0.006. In convolution, our approach not only demonstrates a significant improvement over existing methods by effectively emulating the step-by-step diagnostic process of endoscopists, but also promises to greatly enhance early detection and treatment strategies for CRC, ultimately improving patient outcomes.