The ability to recognize human faces is a demonstration of incredible human intelligence. Over the last four decades, attempts from diverse areas are made to replicate this outstanding visual perception of human beings in machine recognition of faces. Within the face recognition literature, researchers have centered the debate on how human beings perceive human faces and this has become an important and active research area. Psychologists concluded that holistic and local feature based approaches are dual routes to the face recognition. Although holistic based approaches have attained certain level of maturity, in general they require a preprocessing procedure to normalize the face image variations in pose, scale and illumination. This is not an easy task because it depends on the accurate detection of at least two landmarks from the face image. As a result, most approaches work on the normalized face images based on the manually identified landmarks. The recognition performance deteriorates considerably if the manual process is replaced by an automatic landmark detection algorithm. Moreover, global features are sensitive to image variations in scale, facial expression, pose and occlusion. Most of the holistic approaches are dependent on the training databases because knowledge about the face discrimination is generalized by machine learning from the face samples. A representative training database is necessary, which, however, is not available in many applications. In contrast to holistic methods, local feature based approaches are more robust to variations in pose, scale, expression and occlusion. Elastic bunch graph matching (EBGM) and active appearance models (AAM) fall into this category. In EBGM, faces are represented as graphs, with nodes positioned at facial landmarks and edges labeled with distance vectors. Each node contains a set of Gabor wavelet coefficients. The AAM can match a class of deformable objects, for example faces with different expressions. However, the performances of both EBGM and AAM depend on a good selection of facial landmarks, which are often annotated manually. This makes the approaches semi-automatic and labor consuming. One of the very fundamental problems arising when analyzing face images originates from the fact that face structures appear in different ways, depending upon the scales of observations. First, local facial structures are shown at different levels of scales, ranging from skin textures at fine scales, through eyes and mouth represented at median scales to the shape of face contours at large scales. Second, the characteristic or the description of a local structure is strongly dependent on the scale at which the structure is modeled. Third, it is unknown in advance what the proper scales are to describe different local structures of unknown face images. To cope with these problems, an image representation that explicitly incorporates the notion of scale is a crucially important tool. The scale invariant feature transform (SIFT) detects distinct local structures from images and selects appropriate scales to describe them automatically. It has shown good performance on object detection and some other machine vision applications. Recently, some initial attempts apply the SIFT algorithm in the face recognition task. However, there are still many outstanding issues and problems that need to be addressed and solved if I am to leverage the idea of SIFT and some of its good properties to solve the challenging face recognition problem. Although some local feature based approaches achieve better recognition performance than holistic approaches, their computational burden is much heavier. In this thesis, I firstly analyze the merits and deficiencies of SIFT and propose new strategies for feature extraction and image matching, which leads to a multi-scale local feature extraction and matching framework (LFEM) that overcomes some limitations of SIFT in solving the face recognition problem. In the original SIFT algorithm, a keypoint is selected only if it is larger than $26$ neighbor points or smaller than all of them. This keypoint detection method works well for rigid visual objects, which have sharp transitions between different sides of an object. In other words, there are distinct corner structures with high contrast in such objects. However, human faces are non-rigid, round and smooth. There are few obvious blobs and corners with high contrast, as the intensity changes in face images are gradual and slow in the most areas. On the other hand, the shape of the structures could be complex and some structures are close to each other or overlap. As a result, many local structures in the smooth area such as forehead, cheeks and chin cannot be detected due to the strict condition of comparing the $26$ neighbors. Hence I propose a new approach to keypoint detection which captures the information of many facial structures in the smooth area such as forehead, cheeks and chin. A partial descriptor is designed to represent the keypoints whose support areas exceed the face image. My proposed detection approach and my partial descriptor strategy produce a rich number of keypoints. As a significant keypoint should be distinct from others in terms of either its location or the image structures of its neighborhood, I further propose to remove keypoints based on their distinctiveness. A two-stage image matching scheme and a strategy of keypoint search for the nearest subjects are developed to cater for the identification task. It circumvents the problem that the most similar local structures to the probe image disperse to many different gallery images. A training procedure is developed to achieve higher recognition accuracies if multiple training samples per subject are available. It contains template selection, template synthesis and unstable keypoint removal to meet different requirements of face recognition applications. Furthermore, I propose a fully automatic face recognition framework based on both the local and global features (FAFF). As mentioned above, holistic approaches are very popular in recent years due to their good performance and low computational complexity. However, the holistic approaches require a preprocessing procedure to normalize the face image variations in pose and scale, which is usually done based on the localizations of semantic local features such as eyes, corners of mouth, nostrils and so on. However, in many real applications the appearances of these semantic features are not distinct or missing due to changes in expressions, occlusions, illuminations or image noise. Hence most of the research papers work on the manually pre-normalized faces. The recognition performance deteriorates considerably if the manual process is replaced by an automatic landmark detection algorithm. In contrast to holistic approaches, the local feature based approach LFEM proposed by me is more robust to image variations in pose and scale. Unlike the holistic approaches, the face normalization is an integrated part of the LFEM approach. And it achieves significantly better recognition performance than many popular holistic approaches, which is validated in the experiments. However, its computational time is much longer. For instance, to fulfill the face recognition task, one must search all the images in the database and compare each local feature in every image, which causes very heavy computational burden. To solve these two critical problems in holistic and local feature based approaches, I propose a fully automatic face recognition framework based on both the local and global features. To speed up the local feature based approach, I propose to integrate the local feature based approach and the holistic approach in a cascaded way. The holistic approach is used as a filter to retrieve some candidate face images from the whole gallery set. The selected face images have higher probabilities matching to the probe, than the remains in the gallery. They form a new gallery set with reduced size, on which I perform the local feature based approach for face recognition. The reduction in the size of the gallery set speeds up the recognition process of LFEM. To solve the alignment problem in holistic approaches, I design a face alignment scheme based on multi-scale local features instead of semantic facial features. In face images, even if the appearances of facial components change drastically, there are many other non-semantic features holding distinct information, which can be utilized in the alignment process. Hence, I propose to align face images based on non-semantic multi-scale local features. This fully automatic framework not only speeds up the local feature based approach, but also improves the recognition accuracy comparing with the holistic and local approaches as shown in the experiments.