Detection of salient motion in a video involves determining which motion is attended to by the human visual system in the presence of background motion that consists of complex visuals that are constantly changing. Salient motion is marked by its predictability compared to the more complex unpredictable motion of the background such as fluttering of leaves, ripples in water, dispersion of smoke, and others. We introduce a novel approach to detect salient motion based on the concept of “observability” from the output pixels, when the video sequence is represented as a linear dynamical system. The group of output pixels with maximum saliency is further used to model the holistic dynamics of the salient region. The pixel saliency map is bolstered by two region-based saliency maps, which are computed based on the similarity of dynamics of the different spatiotemporal patches in the video with the salient region dynamics, in a global as well as a local sense. The resulting algorithm is tested on a set of challenging sequences and compared to state-of-the-art methods to showcase its superior performance on grounds of its computational efficiency and ability to detect salient motion.