When used to recognize side-scan sonar images of shipwreck targets, the Faster R-CNN model is time-consuming and has low efficiency and a high missed detection rate for small targets. Considering that existing datasets of side-scan sonar images of shipwreck targets are small, we propose a YOLOv3 model that can automatically recognize side-scan sonar images of shipwreck targets based on transfer learning. Based on the Darknet-53 network, we froze part of the convolutional layer of the YOLOv3 model trained on COCO dataset images, and conducted transfer learning. Multi-scale training of shallow feature fusion was done based on multi-scale feature fusion with Feature Pyramid Networks (FPN) support, and the proportion of recognized shallow features of shipwreck targets increased. Meanwhile, the parameters and sizes of target anchor boxes were reset using K-means clustering, which allowed us to improve the speed of target recognition and the precision of smaller target recognition and positioning. Lastly, the binary classification cross entropy function was used to improve the loss function of the YOLOv3 algorithm. Experimental results show that under the same recognition target, the average precision (AP) value of the YOLOv3 model based on transfer learning reached 89.49%, which is an improvement of 0.31% and 1.77%, respectively, compared with the Faster R-CNN model and the traditional YOLOv3 model. Moreover, the YOLOv3 model based on transfer learning had the highest harmonic mean (F1), reaching 90.71%, which is 3.96% and 1.63% higher, respectively, than the harmonic means of the Faster R-CNN and the traditional YOLOv3 model. Lastly, the traditional YOLOv3 model takes an average 0.17s to identify a target. In contrast, the Faster R-CNN model takes an average of 2.8s to identify a target. Hence, our transfer learning YOLOv3 model greatly improves detection efficiency, meets the needs of real-time target recognition, and ultimately has better overall performance than existing methods.