In order to make the evaluation of the quality of translation more representative, literature lovers with different age, gender, occupation and cultural level are organized by establishing network reading groups and will be taken as appreciator and evaluators of text using machine translation. A number of evaluation indexes for translation text quality are set and a questionnaire is designed according to these indexes. The questionnaires are sent to literature lovers and data are obtained. Fuzzy matrix is established to evaluate the quality of machine translation using the fuzzy mathematics method. A comprehensive evaluation vector is obtained and the quantity evaluation of the quality of the translation text is realized. This paper provides a tool for developing and improving translation software.