IRSp53 (also known as BAIAP2) is an abundant excitatory postsynaptic scaffolding protein implicated in autism spectrum disorders (ASD), schizophrenia, and attention-deficit/hyperactivity disorder (ADHD). IRSp53 is expressed in different cell types across different brain regions, although it remains unclear how IRSp53 deletion in different cell types affects brain functions and behaviors in mice. Here, we deleted IRSp53 in excitatory and inhibitory neurons in mice and compared resulting phenotypes in males and females. IRSp53 deletion in excitatory neurons driven by Emx1 leads to strong social deficits and hyperactivity without affecting anxiety-like behavior, whereas IRSp53 deletion in inhibitory neurons driven by Viaat has minimal impacts on these behaviors in male mice. In female mice, excitatory neuronal IRSp53 deletion induces hyperactivity but moderate social deficits. Excitatory neuronal IRSp53 deletion in male mice induces an increased ratio of evoked excitatory and inhibitory synaptic transmission (E/I ratio) in layer V pyramidal neurons in the prelimbic region of the medial prefrontal cortex (mPFC), whereas the same mutation does not alter the E/I ratio in female neurons. These results suggest that IRSp53 deletion in excitatory and inhibitory neurons and in male and female mice has distinct impacts on behaviors and synaptic transmission.