The visual environment for observers is usually complex, and it is impossible for the human visual system (HVS) to process all signal components and figure out their relationships immediately. Selective attention in the HVS allocates most processing resources to the salient regions rather than the entire visual view equally. There are two different types of visual attention mechanism: bottom-up and top-down. Visual attention mechanism will cause the salient regions automatically ‘pop out’ in visual scenes. In this thesis, we explore the visual attention modeling and its applications in visual signal processing. Firstly, we propose a saliency detection model for images based on human visual sensitivity and amplitude spectrum. The amplitude spectrum is adopted to represent color, intensity, and orientation distributions for image patches. The saliency value of each image patch is calculated by not only the differences between amplitude spectrum of this patch and other patches in the whole image, but also the visual impacts of these differences determined by human visual sensitivity. Due to the integration of the characteristics of the HVS and better feature representation, the proposed saliency detection model can achieve better performance than existing ones.