Motor imagery (MI) EEG signals are considered a promising paradigm for BCI systems that enable humans to communicate with the outside world through the brain and have a wide range of applications to improve pa-tients' quality of life with muscle or nerve damage. Due to the low signal-to-noise ratio of the acquired EEG signals, it is challenging to decode the intent accurately and even more challenging to decode the raw EEG signals. Currently, there is no deep learning method to achieve high classification performance in decoding raw EEG signals. We propose a new end-to-end network for decoding MI EEG signals, Compact Multi-Branch One-dimensional Convolutional Neural Network (CMO-CNN), without some pre-processing such as filtering, using the original EEG signals. The 1D convolution is used as the feature extractor to extract diverse and multi-level features for fusion using different filter scales and depths of different branches. 1D Squeeze-and-Excitation blocks (SE-blocks) and shortcut connections are added to further improve the generalization and robustness of the network. 83.92% and 87.19% classification accuracies were achieved in the BCI Competition IV-2a and the BCI Competition IV-2b datasets. An 8% improvement to 63.34% was achieved in the cross-subject test, demonstrating that our proposed CMO-CNN outperforms the current state-of-the-art methods. Visual analysis of the network shows that the proposed model can accurately learn the event-related desynchronization/syn-chronization (ERD/ERS) phenomenon in the signal, and 1D convolution is actively used for feature extraction suitable for feature extraction of the original EEG signal.