Hair cells (HCs) are mechanosensors that play crucial roles in perceiving sound, acceleration, and fluid motion. The precise architecture of the auditory epithelium and its repair after HC loss is indispensable to the function of organ of Corti (OC). In this study, we showed that Brg1 was highly expressed in auditory HCs. Specific deletion of Brg1 in postnatal HCs resulted in rapid HC degeneration and profound deafness in mice. Further experiments showed that cell-intrinsic polarity of HCs was abolished, docking of outer hair cells (OHCs) by Deiter’s cells (DCs) failed, and scar formation in the reticular lamina was deficient. We demonstrated that Brg1 ablation disrupted the Gαi/Insc/LGN and aPKC asymmetric distributions, without overt effects on the core planer cell polarity (PCP) pathway. We also demonstrated that Brg1-deficient HCs underwent apoptosis, and that leakage in the reticular lamina caused by deficient scar formation shifted the mode of OHC death from apoptosis to necrosis. Together, these data demonstrated a requirement for Brg1 activity in HC development and suggested a role for Brg1 in the proper cellular structure formation of HCs.