Cloud classification of ground-based images is a challenging task. Recent research has focused on extracting discriminative image features, which are mainly divided into two categories: 1) choosing appropriate texture features and 2) constructing structure features. However, simply using texture or structure features separately may not produce a high performance for cloud classification. In this paper, an algorithm is proposed that can capture both texture and structure information from a color sky image. The algorithm comprises three main stages. First, a preprocessing color census transform (CCT) is applied. The CCT contains two steps: converting red, green, and blue (RGB) values to opponent color space and applying census transform to each component. The CCT can capture texture and local structure information. Second, a novel automatic block assignment method is proposed that can capture global rough structure information. A histogram and image statistics are computed in every block and are concatenated to form a feature vector. Third, the feature vector is fed into a trained support vector machine (SVM) classifier to obtain the cloud type. The results show that this approach outperforms other existing cloud classification methods. In addition, several different color spaces were tested and the results show that the opponent color space is most suitable for cloud classification. Another comparison experiment on classifiers shows that the SVM classifier is more accurate than the k–nearest neighbor (k-NN) and neural networks classifiers.