We experimentally investigate the nonlinear response of a multilayer graphene resonator using a superconducting microwave cavity to detect its motion. The radiation pressure force is used to drive the mechanical resonator in an optomechanically induced transparency configuration. By varying the amplitudes of drive and probe tones, the mechanical resonator can be brought into a nonlinear limit. Using the calibration of the optomechanical coupling, we quantify the mechanical Duffing nonlinearity. By increasing the drive force, we observe a decrease in the mechanical dissipation rate at large amplitudes, suggesting a negative nonlinear damping mechanism in the graphene resonator. Increasing the optomechanical backaction, we observe a nonlinear regime not described by a Duffing response that includes new instabilities of the mechanical response.
