Thesis (M.Sc.)--Chulalongkorn University, 2008

The frustrated antiferromagnetic XY model on the triangular lattice is a simple model describing the critical phenomena of magnetic systems. By using the Monte Carlo simulation method associated with the Metropolis algorithm, we investigate phase transitions of the system at critical temperatures and near critical temperatures. To achieve a scaling form of important parameters, the finite size effect must be taken into account. Our results show clearly that the broken symmetries of the system occur at two separate critical temperatures, the Kosterlitz-Thouless temperature and the Ising-like temperature (TI). However these temperatures lie close to each other about 0.6% compared with the lower temperature. At TI, the reflection symmetry breaking transition occurs and it is known as the Ising-like transition. While at the lower temperature, the rotational symmetry breaking transition occurs and it is known as the Kosterlitz-Thouless transition. Moreover, we found the asymptotic behavior of the correlation length of the Ising-like transition with the critical exponent, ν = 0.74. Taken into account the finite size effect, this value approaches 0.83 which differs from the value, ν = 1, for the Ising universality class. It is likely that the Ising-like transition may not belong to the Ising universality class. Also the other parameters such as the energy of the system, the order parameters, the specific heat, the magnetic susceptibility and the correlation functions have been calculated and their properties near the critical temperatures and at the critical temperatures have been studied.