Understanding the interplay between bright and dark exciton states is crucial for deciphering the luminescence properties of low-dimensional materials. The origin of the outstanding brightness of lead halide perovskites remains elusive. Here, we analyse temperature-dependent time-resolved photoluminescence to investigate the population mixing between bright and dark exciton sublevels in individual CsPbBr$_3$ nanocrystals in the intermediate confinement regime. We extract bright and dark exciton decay rates, and show quantitatively that the decay dynamics can only be reproduced with second-order phonon transitions. Furthermore, we find that any exciton sublevel ordering is compatible with the most likely population transfer mechanism. The remarkable brightness of lead halide perovskite nanocrystals rather stems from a reduced asymmetry between bright-to-dark and dark-to-bright conversion originating from the peculiar second-order phonon-assisted transitions that freeze bright-dark conversion at low temperature together with the very fast radiative recombination and favourable degeneracy of the bright exciton state.