Characterizing some finite groups by average order