Different colors can evoke different emotions for images. For example, artists usually use different color combinations to express different emotions when creating posters or pictures. Existing color transfer methods can successfully transfer visual appearance between images by changing colors. However, they often fail to change an image to meet an accurate emotion. Nowadays, there are few emotional color transfer methods between images, and those methods are imperfect in transferring the emotion from one image to another image due to the inaccuracy in the emotion calculation of the reference image. Therefore, we propose a texture-aware emotional color transfer method between images, which can adjust an image with an emotion word or a reference image. Each emotion word represents a type of emotion, such as alluring, fresh, and antique. The user simply input an emotion word (e.g., lovely), the system can automatically adjust the image to the target emotion (e.g., fresh). First, we propose a new emotion calculation method to compute the target emotion from a reference image. Unlike previous methods, we take both texture features and main colors in our emotion calculation model. Then, in order to find the proper color combinations to reflect the target emotion, three color-emotion model databases are built by color numbers of the models. Those models are obtained by exploiting the famous art theories, and we design a novel strategy to select the most suitable color-emotion model from the databases. Finally, we propose a new color transfer algorithm by utilizing color adjustment and color blending to guarantee the color gradient and naturalness. Experiments show that our method's results are more consistent with the emotion of the input image than the state-of-the-art algorithms.