Objective. The combination of the motor imagery (MI) electroencephalography (EEG) signals and deep learning-based methods is an effective way to improve MI classification accuracy. However, deep learning-based methods often need too many trainable parameters. As a result, the trade-off between the network decoding performance and computational cost has always been an important challenge in the MI classification research. Approach. In the present study, we proposed a new end-to-end convolutional neural network (CNN) model called the EEG-circular dilated convolution (CDIL) network, which takes into account both the lightweight model and the classification accuracy. Specifically, the depth-separable convolution was used to reduce the number of network parameters and extract the temporal and spatial features from the EEG signals. CDIL was used to extract the time-varying deep features that were generated in the previous stage. Finally, we combined the features extracted from the two stages and used the global average pooling to further reduce the number of parameters, in order to achieve an accurate MI classification. The performance of the proposed model was verified using three publicly available datasets. Main results. The proposed model achieved an average classification accuracy of 79.63% and 94.53% for the BCIIV2a and HGD four-classification task, respectively, and 87.82% for the BCIIV2b two-classification task. In particular, by comparing the number of parameters, computation and classification accuracy with other lightweight models, it was confirmed that the proposed model achieved a better balance between the decoding performance and computational cost. Furthermore, the structural feasibility of the proposed model was confirmed by ablation experiments and feature visualization. Significance. The results indicated that the proposed CNN model presented high classification accuracy with less computing resources, and can be applied in the MI classification research.