Physical inactivity has been identified as an important risk factor for dementia. High levels of cardiorespiratory fitness (CRF) have been shown to reduce the risk of dementia. However, the mechanism by which exercise affects brain health is still debated. Fractal dimension (FD) is an index that quantifies the structural complexity of the brain. The purpose of this study was to investigate the effects of a 5-year exercise intervention on the structural complexity of the brain, measured through the FD, in a subset of 105 healthy older adults participating in the randomized controlled trial Generation 100 Study. The subjects were randomized into control, moderate intensity continuous training, and high intensity interval training groups. Both brain MRI and CRF were acquired at baseline and at 1-, 3- and 5-years follow-ups. Cortical thickness and volume data were extracted with FreeSurfer, and FD of the cortical lobes, cerebral and cerebellar gray and white matter were computed. CRF was measured as peak oxygen uptake (VO2peak) using ergospirometry during graded maximal exercise testing. Linear mixed models were used to investigate exercise group differences and possible CRF effects on the brain's structural complexity. Associations between change over time in CRF and FD were performed if there was a significant association between CRF and FD. There were no effects of group membership on the structural complexity. However, we found a positive association between CRF and the cerebral gray matter FD (p < 0.001) and the temporal lobe gray matter FD (p < 0.001). This effect was not present for cortical thickness, suggesting that FD is a more sensitive index of structural changes. The change over time in CRF was associated with the change in temporal lobe gray matter FD from baseline to 5-year follow-up (p < 0.05). No association of the change was found between CRF and cerebral gray matter FD. These results demonstrated that entering old age with high and preserved CRF levels protected against loss of structural complexity in areas sensitive to aging and age-related pathology.