To improve image search results, understanding and exploiting the subjective aspects of an image is critical. However, how to effectively extract these subjective aspects (e.g. feeling, emotion, and so on) from an image is a challenging problem. In this paper, we propose a novel approach for predicting affective aspects, one of the most interesting subjective aspects, of concepts in images by learning the semantic attributes of the concept and mining the association between the attributes and affective aspects. The main idea of the proposed approach comes from the assumption that semantic attributes of a concept will influence the user’s affect towards the concept (e.g. an animal with the semantic attributes ‘small’, ‘furry’, ‘white’ can be associated with the affective term ‘cute’). Based on this assumption, we build a multi-layer affect learning framework that consists of (1) an attribute learning layer that predicts semantic attributes of a concept and (2) an affect learning layer that exploits the outputs from the attribute learning layer for predicting the affective aspects of the concept. Through the experimental results on the Animals with Attributes dataset, we show that the proposed approach outperforms traditional approaches by up to 25% in terms of precision and successfully predicts the affect of concepts in images according to different user preferences.