Abstract: In the detection of type-B sleeve fillet welds, defects such as cracks and incomplete welding that have a small angle with the horizontal line produce specular reflection waves that are difficult to capture, and only two or one end-angle diffraction wave can be obtained. This makes it difficult to determine the characteristics of the defects from their images, resulting in significant quantitative deviation. A finite element model for phased array detection of type-B sleeve fillet welds with manually grooved and drilled defects was constructed. A one-dimensional linear array probe was used to capture the full matrix data of the defects, and a large number of time-domain echo signals were converted into a time-spectrum database for training a convolutional neural network model. The ability to identify grooving and drilling defects in the horizontal fusion zone was tested through sample experiments, and four neural network models—VGGNet, ResNet, EfficientNet, and GoogLeNet—were compared. Experimental results show that the accuracy of the ResNet50 model can reach 97.65%. The combination of finite element simulation and deep learning can effectively identify grooves and holes at low cost, which not only improves the accuracy of defect length quantification but also helps optimize the detection process of B-shaped sleeves, providing a new solution for other complex structures.