Bottom up and top down models of visual attention have been proposed to deal with the guidance of human visual attention in viewing real-world scenes. However task demands put upon us can change our behaviour dramatically. While there is evidence in support of both bottom up and top down processes guiding our attention, it is still very much a contested issue regarding the degree of power each has under certain tasks. Two experiments are reported in order to tease apart what information we use in a memory-encoding task or a search task. Critical objects in each scene were chosen and altered in terms of visual saliency and semantic congruency with the rest of the scene. Measurement of eye movements indicated that by altering task demands, the times to first fixate the critical object and the time spent on the object during the first fixation were altered significantly quicker and shorter in the search task. Semantic congruence had an effect on saccade amplitude size, but only in the memory task. Semantic congruence of the critical object had a greater effect within a memory task than first thought in terms of first fixation duration and the time to first fixate on the critical object. Both of these reduced in length with incongruence. Also salience was found to have a greater impact in active search tasks than previously believed in terms of trial dwell time and the time to first fixate on the critical object, where higher salience reduced both of these times. Overall, semantically incongruent objects were fixated sooner and for longer regardless of the task demands. The results support findings that task demands strongly effect how we look at scenes, and indicate that semantic congruence and visual salience may have greater effects in both tasks than previously thought.