The detection of defects in wood is valuable for promoting the efficient exploitation of wood. So it is significant to further increase the accuracy of the detection of wood defects and enhance the real-time detection. In this paper, the YOLOv5 convolutional neural network is applied to wood defects detection, and the model is modified for both the YOLOv5n and YOLOv5m scales. The SimAM attention model was first incorporated into the network, and the learning rate decay strategy was replaced with CosLR, with Ghost convolution employed to minimize the model parameters. Finally, the modified network was tested for five types of wood defects, including live-knot, resin, dead-knot, knot-with-crack, and crack. It is demonstrated that the improvements resulted in a 1.5% increase in mAP0.5:0.95 for YOLOv5n-C3Ghost and a 1.6% increase in mAP0.5:0.95 for YOLOv5m-C3Ghost. In addition, there is a 51% and 63% difference in the number of model parameters, and a decrease in inference time and floating point operations respectively. The experiments indicate that our improved method not only enhances the accuracy of YOLOv5 in detecting wood defects, but also enables a reduction in the volume and computational cost of the model parameters.