Because of the high complexity and long time of CPU in convolution operation, the power consumption of GPU is high and it can not be deployed to small hardware devices. Field Programmable Logic Gate Array (FPGA) has parallelism and fast speed , Low power consumption and etc. Therefore, this paper proposes an improved RLeNet model, which optimizes model parameters and used a parallel technology to accelerate CNN hardware. The improved model parameter data volume is reduced to 3.2% of the original. Under the MNIST data set, the accuracy of the prediction result is 98.5%. Finally, on the Xilinx Nexys 4 DDR: Artix-7 development board, the prediction process of the convolutional neural network for handwriting recognition is implemented. When the system predicts with a 200MHz clock, the time to predict a picture is 31.8μs.