The ability to measure surface temperature and represent it on a metrically accurate 3D model has proven applications in many areas, such as medical imaging, building energy auditing, and search and rescue. A system is proposed that enables this task to be performed with a handheld sensor, and for the first time with results able to be visualized and analyzed in real time. A device comprising a thermal-infrared camera and range sensor is calibrated geometrically and used for data capture. The device is localized using a combination of iterative closest point and video-based pose estimation from the thermal-infrared video footage, which is shown to reduce the occurrence of failure modes. Furthermore, the problem of misregistration, which can introduce severe distortions in assigned surface temperatures is avoided through the use of a risk-averse neighborhood weighting mechanism. Results demonstrate that the system is more stable and accurate than previous approaches, and can be used to accurately model complex objects and environments for practical tasks.