I used to work as a SW and if ever news would circulate about a bad client, more often than not there would be workers who had seen that same client and had no problem with them. My local sex worker organisation shared a warning about a ugly mug who had been accused of r*pe and attempted murder. There were workers who had been seeing this guy for years as a regular, completely unaware of the danger.
Some observations based on my own experience. All MOO.
Opportunistic type client may test your boundaries before deciding whether you would make an easy victim. They might ask, or try to do something they know they are not allowed to do, and see how far they can take it.
If you work alone, have no security. At least where I live, outcalls to private residences are considered to be one of the most high risk types of bookings. Especially late at night. The guy is in his comfort zone and you are unfamiliar with your surroundings, the layout of the house and potential escape routes.
He might also see a lack of vigilance as an invite or justification for his behaviour.
Maybe he only has disgust or disregard for certain types of sex workers... if you don't fit that type they see you as more human.