A major challenge in dynamic reconfiguration of a running system is to understand in advance the impact on the system's Quality of Service (QoS). For some systems, any unexpected change to QoS is unacceptable. In others, the possibility of dissatisfaction increases due to the impaired performance of the running system or unpredictable errors in the resulting system. In general it is difficult to choose a reasonable reconfiguration approach to satisfy a particular domain application. Our investigation on this issue for dynamic approaches is four-fold. First, we define a set of QoS characteristics to identify the evaluation criteria. Second, we design a set of abstract reconfiguration strategies bringing existing and new approaches into a unified evaluation context. Third, we design a reconfiguration benchmark to expose a rich set of QoS problems. Finally, we test the reconfiguration strategies against the benchmark and evaluate the test results. The analysis of acquired results helps to understand dynamic reconfiguration approaches in terms of their impact on the QoS of running systems and possible enhancements for newer QoS capability.