Breast cancer is one of the most common causes of death among women worldwide and poses a major threat to women's health. Currently, ultrasound examination is widely used for early detection of breast cancer, and accurate segmentation of breast lesions from ultrasound images is beneficial for early diagnosis of breast cancer. In this study, a pyramid boundary attention network was developed to automatically and stably segment breast lesions. In addition, a pyramid squeeze attention module was introduced into U-Net to increase the network's receptive field and establish global attention by using convolutional kernels of different sizes, thereby improving the network's detection and segmentation capabilities for tumors of different sizes. Additionally, to address the problem of fuzzy boundaries of breast tumor in ultrasound images, a boundary branch network was designed that used the contextual local attention module to extract the feature maps of different dimensions from the encoder and integrated local spatial and high-level semantic context information. The boundary fusion module was used to extract global and local information of tumor boundaries to obtain the boundary map of tumors, thereby refining the boundary segmentation of tumors and improving the segmentation accuracy of tumors. The proposed method was validated on two public datasets, BUSI and Dataset B, with dice coefficients of 78.98% and 84.5%, respectively. Compared with other medical image segmentation methods and the latest semantic segmentation methods, the proposed method showed superior performance in segmenting breast lesion in ultrasound images.