Vulnerable plaque detection to identify plaque is important in coronary heart disease diagnosis. Currently, it is conducted through manual reading of intravascular optical coherence tomography (IVOCT) images by an interventional cardiologist. However, human reading and understanding is highly subjective. An objective and automated assessment of plaque status is highly needed. This paper proposes a method for automatic image classification in IVOCT images based on different lesion types. In the proposed method, we first use detail-preserving anisotropic diffusion to remove speckle noise in IVOCT images. It removes the noise without losing details. Then, the IVOCT images are transformed to polar coordinates for feature extraction. In particular, Fisher vector and other texture features including local binary pattern and histogram of oriented gradients are studied. Finally, a support vector machine classifier is obtained to classify the IVOCT images into fivegroups: normal (Normal), fibrous plaque (FP), fibroatheroma (FA), plaque rupture (PR), and fibrocalcific plaque (FC). These five groups are obtained according to lesion characteristics. We evaluate the proposed method in a dataset of 1,000 images with five groups. Experimental results show that the proposed method achieves an average accuracy of 90% in image classification. The proposed automatic IVOCT image classification method can be used to save time and cost of cardiologist.