We point out an interesting connection between Williamson matrices and relative difference sets in nonabelian groups. As a consequence, we are able to show that there are relative (4t,2,4t,2t)-difference sets in the dicyclic groups Q_{8t}=\la a,b|a^{4t}=b^4=1, a^{2t}=b^2, b^{-1}ab=a^{-1}\ra for all t of the form t=2^a\cdot 10^b \cdot 26^c \cdot m with a,b,c\ge 0, m\equiv 1\ (\mod 2), whenever 2m-1 or 4m-1 is a prime power or there is a Williamson matrix over \Z_m. This gives further support to an important conjecture of Ito IT5 which asserts that there are relative (4t,2,4t,2t)-difference sets in Q_{8t} for every positive integer t. We also give simpler alternative constructions for relative (4t,2,4t,2t) -difference sets in Q_{8t} for all t such that 2t-1 or 4t-1 is a prime power. Relative difference sets in Q_{8t} with these parameters had previously been obtained by Ito IT1. Finally, we verify Ito‘s conjecture for all t\le 46.