We address the problem of body pose tracking in a scenario of multiple camera setup with the aim of recovering body motion robustly and accurately. The tracking is performed on three-dimensional (3D) space using 3D data, including colored volume and 3D optical flow, which are reconstructed at each time step. We introduce strategies to compute multiple camera-based 3D optical flow and have attained efficient and robust 3D motion estimation. Body pose estimation starts with a prediction using 3D optical flow and then is changed to a lower-dimensional global optimization problem. Our method utilizes a voxel subject-specific body model, exploits multiple 3D image cues, and incorporates physical constraints into a stochastic particle-based search initialized from the deterministic prediction and stochastic sampling. It leads to a robust 3D pose tracker. Experiments on publicly available sequences show the robustness and accuracy of our approach.