Since digital images and videos are rapidly increasing in the internet with the spread of mobile computers and smartphones, research on image retrieval has gained tremendous momentum. Color, shape, and texture are major features used in image retrieval. Especially, color information has been widely used in image retrieval, because it is robust in translation, rotation, and a small change of camera view. This paper proposes a new method for histogram refinement based on local color difference. Firstly, the proposed method converts a RGB color image into a HSV color image. Secondly, it reduces the size of color space from 2563 to 32. It classifies pixels in the 32-color image into three groups according to the color difference between a central pixel and its neighbors in a 3x3 local region. Finally, it makes a color difference vector(CDV) representing three refined color histograms, then image retrieval is performed by the CDV matching. The experimental results using public image database show that the proposed method has higher retrieval accuracy than other conventional ones. They also show that the proposed method can be effectively applied to search low resolution images such as thumbnail images.