Curvature Perturbations in Anisotropic Inflation with Symmetry Breaking