The difficulty that the probabilities infinitely increase with time when the time is large enough in time-dependent perturbation theory for quantum systems with finite number of levels is treated by means of simply transforming the straightforward form of the perturbative series into the natural exponential functions, or their combinations, of new perturbative series. Three exactly solvable models are taken to check our new formulation, and excellent agreements with the exact solution are achieved.