On eigenvalues of Seidel matrices and Haemers' conjecture