I think (distaste for cross dressers) comes from the still deeply set belief that men are somehow superior to women. If a woman wants to look more like a man, that might be odd, but it's almost aspirational. A man wishing to appear and behave more like a woman is seen as a downgrade, as the man choosing to be less than he is.