According to the problem that segmentation method is difficult to get accurate and complete skeleton information, the method of capturing depth image through a depth sensor is proposed which can obtain a complete skeleton information and improve the accuracy of the skeleton key points. First, use depth images to obtain the human skeleton, second, through coordinate transformation to extract the human skeleton, third, take advantage of these features to train the hidden Markov model, last, rely on the hidden Markov model to make behavior recognition. A great number of experiments indicate that the correct recognition rate of this system reaches up to 80%.