Although decades of research have shown associations between early caregiving adversity, stress physiology and limbic brain volume (e.g., amygdala, hippocampus), the developmental trajectories of these phenotypes are not well characterized. In the current study, we used an accelerated longitudinal design to assess the development of stress physiology, amygdala, and hippocampal volume following early institutional care. Previously Institutionalized (PI; N = 93) and comparison (COMP; N = 161) youth (ages 4–20 years old) completed 1–3 waves of data collection, each spaced approximately 2 years apart, for diurnal cortisol (N = 239) and structural MRI (N = 156). We observed a developmental shift in morning cortisol in the PI group, with blunted levels in childhood and heightened levels in late adolescence. PI history was associated with reduced hippocampal volume and reduced growth rate of the amygdala, resulting in smaller volumes by adolescence. Amygdala and hippocampal volumes were also prospectively associated with future morning cortisol in both groups. These results indicate that adversity-related physiological and neural phenotypes are not stationary during development but instead exhibit dynamic and interdependent changes from early childhood to early adulthood.