Colorectal cancer (CRC) represents one of the common malignancies of the gastrointestinal tract. The CRC incidence and mortality rates can be significantly reduced through early detection and resection of the precursor lesions, also known the colorectal polyps. However, such polyps can be missed during manual colonoscopy screening. With recent advances in artificial intelligence, numerous computer-aided diagnosis (CAD) methods have been proposed for colonoscopy applications. In particular, deep learning algorithms have been recently designed to incorporate sophisticated attention mechanisms into convolutional blocks and hence demonstrate a great potential for enhancing the performance of convolutional neural networks (CNNs). Nevertheless, most current deep learning techniques suffer from the high model complexity and excessive computational burden. In this paper, we introduce a deep learning approach for colorectal polyp detection and segmentation. Specifically, we propose a new shuffle efficient channel attention network (sECANet) with no dimensionality reduction. This network can be exploited to learn effective channel attention by obtaining cross-channel interactions. A total of 2112 manually-labeled images were collected from 1197 patients in a local hospital using colonoscopy screening. Additional data samples were collected from the CVC-ClinicDB, the ETIS-Larib Polyp DB and the Kvasir-SEG data set. The captured images were partitioned into 3590 training images and 330 testing images, and each image was labeled as a polyp or non-polyp image. We assessed our framework on the testing images and achieved a precision of 94.9%, a recall of 96.9%, a F1 score of 95.9%, and a F2 score of 96.5%. In conclusion, our proposed framework has a great potential of assisting endoscopists in tracking polyps during colonoscopy and therefore performing early and timely resection of such polyps before they evolve into invasive cancer types. (c) 2021 THE AUTHORS. Published by Elsevier BV on behalf of Faculty of Engineering, Alexandria University.