The recovery of 3D information of a face from two (or more) images has been extensively studied and many interesting applications have been implemented. This paper proposes a model-based 3D face pose and animation tracker with auto-registration using a binocular EKF kernel for real time application like human computer interface. Compared with previous trackers, this tracker is able to automatically register face model, which eliminates the traditional manual registration process. Another major contribution is to use augmented measurement as a regularization strategy for suppressing variables coupling and over-fitting problem. This work also gives a compact factorized mathematic representation of the tracking kernel, which is easier to implement. Both synthetic and real experimental results show significant advantages of the proposed tracker from the aspects of processing speed and reliability compared with the conventional monocular tracker and Levenberg–Marquardt tracker.