The fusion of information gathered from multiple sources is essential to build a comprehensive situation picture for autonomous ground vehicles. In this paper, an approach which performs scene classification and data fusion for 3D-LIDAR scanner (Velodyne HDL-64E) and video camera is described. A geometry segmentation algorithm is proposed for detection of obstacles and ground area from data collected by the Velodyne. In the meantime, the corresponding image collected by video camera is classified patch by patch into more detailed categories. The final situation picture is obtained by fusing the classification results of Velodyne data and that of images using the fuzzy logic inference framework. The proposed approach was evaluated with datasets collected by our autonomous ground vehicle testbed in the rural area. The fused results are more reliable and more completable than those provided by individual sensors.