Abstract: In the phased array ultrasonic detection of fillet welds of type-B sleeve, for defects such as cracks and incomplete welding that have a small angle with the horizontal fusion line, the specular reflection waves are often difficult to be captured due to structural constraints, and only one or two end-angle diffraction wave can be obtained. This makes it difficult to determine the characteristics of the defects from their images, resulting in significant quantitative deviation. A finite element model for phased array detection of type-B sleeve fillet welds with manually grooved and drilled defects was constructed. A one-dimensional linear array probe was used to capture the full matrix data of the defects, and a large number of time-domain echo signals were converted into a time-spectrum database for training a convolutional neural network model. The ability to identify grooving and drilling defects in the horizontal fusion zone was tested through sample experiments, and four neural network models—VGGNet, ResNet, EfficientNet, and GoogLeNet—were compared. Experimental results show that the identification accuracy of the ResNet50 model can reach 97.65%. The combination of finite element simulation and deep learning can effectively identify grooves and holes at low cost, which not only improves the accuracy of defect length quantification but also helps optimize the detection process of type-B sleeves, providing a new solution for other complex structures.