Men brought up with women ‘are less sexy’