Electron-photon temporal correlations in electron energy loss (EELS) and cathodoluminescence (CL) spectroscopies have recently been used to measure the relative quantum efficiency of materials. This combined spectroscopy, named Cathodoluminescence excitation spectroscopy (CLE), allows the identification of excitation and decay channels which are hidden in average measurements. Here, we demonstrate that CLE can also be used to measure excitation's decay time. In addition, the decay time as a function of the excitation energy is accessed, as the energy for each electron-photon pair is probed. We used two well-known insulating materials to characterize this technique, nanodiamonds with \textit{NV$^0$} defect emission and h-BN with a \textit{4.1 eV} defect emission. Both also exhibit marked transition radiations, whose extremely short decay times can be used to characterize the instrumental response function. It is found to be typically 2 ns, in agreement with the expected limit of the EELS detector temporal resolution. The measured lifetimes of \textit{NV$^0$} centers in diamond nanoparticles (20 to 40 ns) and \textit{4.1 eV} defect in h-BN flakes ($<$ 2 ns) matches those reported for those materials previously.
