The fashion and textiles industry, and policymakers at all levels, are showing an increased interest in the concept of circular economy as a way to decrease business risks and negative environmental impacts. However, focus is placed mainly on the material ‘stuff’ of textile fashion and its biophysical harms. The current material focus has several shortcomings, because fashion is a social-ecological system and cannot be understood merely by addressing its environmental dimensions. In this paper, we rethink the fashion system from a critical social-ecological perspective. The driver-state-response framework shows social drivers and ecological impacts as an adaptive social-ecological system, exposing how these interacting aspects need to be addressed for sustainable and resilient implementation of circular economy. We show how current responses to global sustainability challenges have so far fallen short. Our overall aim is to expand possibilities for reframing responses that better reflect the complex links between the global fashion system, culture and creativity and the dynamics of the living planet. We argue that reducing planetary pressure from the global fashion and textiles industry requires greater recognition of the system’s social drivers with more emphasis on the many cross-scale links between social and ecological dimensions. Resilient decisions aiming for sustainable circularity of the fashion industry must therefore pay attention to social activities beyond the industry value chain, not just material flows within it. Funding: Open access funding provided by Stockholm University. The work presented here was part of a consortium research project between Stockholm Resilience Centre and the Ellen MacArthur Foundation, initiated and funded by H&M Group, the Swedish public limited company. Celinda Palm, Sarah Cornell and Tiina Häyhä’s employment was part-funded through this project.