In this paper, we propose a new multi-manifold metric learning (MMML) method for the task of face recognition based on image sets. Different from most existing metric learning algorithms that learn the distance metric for measuring single images, our method aims to learn distance metrics to measure the similarity between manifold pairs. In our method, each image set is modeled as a manifold and then multiple distance metrics among different manifolds are learned. With these distance metrics, the intra-class manifold variations are minimized and inter-class manifold variations are maximized simultaneously. For each person, we learn a distance metric by using such a criterion that all the learned distance metrics are person-specific and thus more discriminative. Our method is extensively evaluated on three widely studied face databases, i.e., Honda/UCSD database, CMU MoBo database and Youtube Celebrities database, and compared to the state-of-the-arts. Experimental results are presented to show the effectiveness of the proposed method.