I think the disparaging comments about his appearance have little to do with how people generally feel towards others- most decent adults judge a person by their character rather than their looks- but it is almost impossible not to be critical about every aspect of IS, because we are so revolted by what he has done. Crucially, we are trying to understand exactly how he could have appeared at all attractive to Helen- a clever and cultured woman of smart appearance, who enjoyed a bit of glamour in her life. Given that he does not appear to be the sharpest tool in the shed, and his personality seems petulant and crotchety under any kind of pressure, one might expect him to compensate for that by putting his best foot forward physically, at least. Instead, we see the unprepossessing and shambolic figure of a man than nobody could really consider to be in the same league as Helen.
It just leaves me feeling infuriated by his every utterance about her, and the attitude he has, which just drips with contempt and disrespect. HOW BLOODY DARE HE?! He should have felt like the luckiest man alive, but it wasn't enough for his staggering sense of entitlement. Even looking at him makes me angry.