We present experimental results, which show a laser polarization dependent contribution to electron beam pointing jitter in laser wakefield accelerators (LWFA). We develop a theoretical model for the polarization dependence in terms of the transverse dynamics of trapped electrons, resonantly driven by bubble centroid oscillations. The latter are generated by the carrier wave phase evolution at the self-steepened laser pulse front. In the model, the polarization dependent jitter originates from shot-to-shot fluctuations of the laser carrier envelope phase. The model is verified by particle in cell simulations and suggests that for non-CEP stabilized systems the polarization dependent jitter may form an ultimate limit to beam pointing stability in LWFAs.
