Computer Aided-Diagnosis (CAD) systems have been proposed to help dermatologists diagnose melanomas. However, these systems fail to provide a medical explanation for the diagnosis. This makes the dermatologists unsure about their use, since they are not easy to understand. In this paper we propose a CAD system that extracts a clinically inspired color description of the lesion and then, uses this information to discriminate melanomas from benign lesions. The proposed system is also capable of showing the extracted color features, making the system and its decisions more comprehensible for practitioners. The development of this system is hampered by the lack of a database of detailed annotate dermoscopy images. Nonetheless, we are able to tackle this issue using an image annotation framework based on the Correspondence-LDA algorithm. This method is applied with success to the identification of relevant colors in dermoscopy images, obtaining an average Precision of 84.9% and a Recall of 85.5%. The proposed color representation is then used to classify skin lesions, resulting in a Sensitivity of 78.9% and Specificity of 76.7%, these values are promising and comparable with the state-of-the art.