We directly investigated the chemical compositional origin of surface roughness variations in air-annealed ZnO single crystal samples for annealing temperatures up to 1000 °C. Atomic Force Microscopy (AFM) showed temperature-dependent changes in surface roughness and morphology, with a maximum in surface roughness of 2 nm found for samples annealed at 400 °C. The O(1s) line, measured by X-ray Photoelectron Spectroscopy (XPS) showed a maximum for Zn(OH)2 and a minimum for off-stoichiometric ZnO at 400 °C; while the Zn(2p) peaks show an increase in slope at that temperature. These results indicate that the roughness arises from Zn diffusion and loss of surface oxygen.