In this letter, we describe quantitative magnetic imaging of superconducting vortices in RbEuFe$_4$As$_4$ in order to investigate the unique interplay between the magnetic and superconducting sublattices. Our scanning Hall microscopy data reveal a pronounced suppression of the superfluid density near the magnetic ordering temperature in good qualitative agreement with a recently-developed model describing the suppression of superconductivity by correlated magnetic fluctuations. These results indicate a pronounced exchange interaction between the superconducting and magnetic subsystems in RbEuFe\textsubscript{4}As\textsubscript{4} with important implications for future investigations of physical phenomena arising from the interplay between them.
