Railway track stiffness is an essential factor influencing the track conditions and long-term deterioration. However, the traditional ways to measure the track stiffness are based on inverse computations using multi-body simulations and/or finite element models, which are time-consuming and at low-speed operation. To overcome these challenges, we propose a convolutional neural network framework to predict the track dynamic stiffness using the accelerations captured by accelerometers mounted on the axle box in real-time. To provide a benefit of computational cost-friendly, a dilated convolutional layer has been added which allows the framework to be applied to a compact device. In our study, a nonlinear finite element model of train-track interactions has been calibrated and used to generate unbiased, full range of data sets of axle box accelerations under various track and operational factors. Subsequently, the simulated data is formatted to three different sample sizes: 250-timesteps, 500-timesteps, and 1,000-time steps. The fine-tuned CNN model is developed based on the three datasets and provides the optimal $\textrm {R}^{2}$ of 0.94, 0.94, and 0.97. The insights gained from this study can assist the track stiffness measurement in the field with a novel measurement method providing continuous, cost-friendly, fast, and implementable benefits. The quantification of dynamic track stiffness will help track engineers to locate problematic and defective tracks promptly on the vast railway networks such as mud pumping, loss of support, pulverized ballast, and so on.