In evolutionary computation, statistical tests are commonly used to improve the comparative evaluation process of the performance of different algorithms. In this paper, three state-of-the-art Differential Evolution (DE) based algorithms, namely Dynamic Memetic Differential Evolution (MOS), Self-adaptive DE hybridized with modified multi-trajectory search (MMTS) algorithm (SaDE-MMTS) and Self-adaptive Differential Evolution Algorithm using Population Size Reduction and three Strategies Algorithm (jDElscop) as well as a novel algorithm called ensemble of parameters and mutation strategies in Differential Evolution with Self-adaption and MMTS (Sa-EPSDE-MMTS), are tested on the most recent LSO benchmark problems and comparatively evaluated using nonparametric statistical analysis. Instead of using the “Value-to-Reach” as the comparison criterion, comprehensive comparison over multiple evolution points are investigated on each test problem in order to quantitatively compare convergence performance of different algorithms. Our investigations demonstrate that even though all these algorithms yield the same final solutions on a large set of problems, they possess statistically significant variations during the convergence. Hence, we propose that evolutionary algorithms can be compared statistically along the evolution paths.