On groups with a finite number of normalizers