Tracking of 3D human body movement from multiple camera video streams is an important problem in the domain of computer vision. In this paper we perform body pose tracking in 3D space using 3D data reconstructed at every frame. We present an efficient GPU-based method for 3D reconstruction of the real world dynamic scenes. Besides volumetric reconstruction, we propose to compute view-independent 3D optical flow (i.e., scene flow) in combination with volumetric reconstruction, and have attained efficient scene flow estimation using GPU acceleration. Body pose estimation starts from a deterministic prediction based on scene flow, and then uses a multi-layer search algorithm involving stochastic search and local optimization. We design and parallelize the PSO-based (particle swarm optimization) stochastic search algorithm and 3D DT (distance transform) computation of the pose estimation method on GPU. To the end, our system can reach efficient and robust body pose tracking.