Calculation of residual stress with finite element method is a basic work in failure mechanism investigation in thermal barrier coating (TBC) system because the residual stress is main driving force for crack nucleation and propagation. In this work, a complicated cosine curve with gradually increasing amplitude was used to simulate interface morphologies between layers so as to study the residual stress behavior during the cooling process in air plasma spraying TBC system by finite element method. The substrate, thermally grown oxide (TGO) and top coat (TC) are considered to be elastic and bond coat (BC) elastic-perfectly plastic. The material properties are all temperature dependent. The stress result comparison between models with and without substrate shows the effect of substrate on the residual stress distribution around layers interfaces should not be ignored as the substrate influences the value of normal residual stress as well as the stress distribution along undulating interfaces. Then the model with substrate was used to study the residual stress evolution along interfaces during cooling down from the temperature of 1000 °C to room temperature. The influences of the thickness of TGO and the amplitude and wavelength of interface on the residual stress distributions near interfaces were considered. The results show that these influences are very complicated. Meanwhile, it's found that the hybrid roughness parameter containing information for height and spacing is more suitable to describe the interface complicacy. The results facilitate understanding the failure mechanism relevant to interface morphology and TGO thickness.