Charge noise in the host semiconductor degrades the performance of spin-qubits and poses an obstacle to control large quantum processors. However, it is challenging to engineer the heterogeneous material stack of gate-defined quantum dots to improve charge noise systematically. Here, we address the semiconductor-dielectric interface and the buried quantum well of a $^{28}$Si/SiGe heterostructure and show the connection between charge noise, measured locally in quantum dots, and global disorder in the host semiconductor, measured with macroscopic Hall bars. In 5 nm thick $^{28}$Si quantum wells, we find that improvements in the scattering properties and uniformity of the two-dimensional electron gas over a 100 mm wafer correspond to a significant reduction in charge noise, with a minimum value of 0.29$\pm$0.02 $\mu$eV/sqrt(Hz) at 1 Hz averaged over several quantum dots. We extrapolate the measured charge noise to simulated dephasing times to cz-gate fidelities that improve nearly one order of magnitude. These results point to a clean and quiet crystalline environment for integrating long-lived and high-fidelity spin qubits into a larger system.