In optical phase conjugation (OPC) systems, the third-order dispersion (TOD) of optical fibers and the nonlinear resonance at well-defined signal sideband frequencies called sideband instability (SI) mainly limit the transmission performance. We present for the first time a complete theoretical analysis of sideband instability (SI) that occurs when two kinds of fibers with different characteristics are concatenated to form a dispersion-managed fiber link. We find that the magnitude of the SI gain reduces with the increase in strength of dispersion management. Next, we focus on the fiber link using the combination of standard single-mode fiber (SMF) and reverse dispersion fiber (RDF), which is widely used for simultaneously compensating second-order dispersion (SOD) and third-order dispersion (TOD). By computer simulation, it is shown that, in wavelength-division-multiplexed (WDM) systems, SI still induces significant degradation in channels located at frequencies where SI induced from other channels arises.By re-allocating the channel frequency to avoid the SI frequency, the transmission performance is improved significantly. Then we propose for the first time, a scheme for simultaneous suppression of both TOD and SI in OPC systems using a higher-order dispersion-managed link consisting of SMFs and RDFs. Computer simulation results demonstrate the possibility of 200-Gbit/s transmission over 10,000 km in the higher-order dispersion-managed OPC system, where the dispersion map is optimized by our system design strategies.