To investigate the association of sleep duration with type 2 diabetes mellitus (T2DM) in a rural Chinese population. A 1:1 matched nested case–control study was performed based on a cohort that had been established in rural communities in Henan Province, China. T2DM patients and healthy controls (550 pairs) were included in this study. Abnormal sleep duration significantly increased the risk of T2DM with an approximate U-shaped association (sleep duration ≤ 6 h, OR = 1.742, 95% CI = 1.007–3.011, P = 0.047; sleep duration 8–9 h, OR = 1.462, 95% CI = 1.038–2.060, P = 0.030) compared with participants with a night sleep duration of 7–8 h, after adjusting for multiple confounders. When stratified by gender, only women were sensitive to shorter sleep duration (OR = 2.483, 95% CI = 1.149–5.366, P = 0.021). Abnormal sleep duration (too short or too long) had adverse effects on homeostasis model assessment (HOMA) and blood metabolites, and the effect was more noticeable in people with longer sleep durations. In a rural Chinese population, both too short and too long sleep duration increased the risk of T2DM. Especially women with less sleep duration have a higher risk of T2DM. Abnormal sleep also affects the HOMA index and metabolites; the relationship between HOMA-IR, total cholesterol, and LDL-Cholesterol with sleep duration was U-shaped, while fasting plasma glucose, body mass index, waist circumference, and triglyceride levels increased significantly only with longer sleep duration.