Dennis' one-acre property, which sits just five blocks from the beach in picturesque coastal town San Clemente, has guesthouses on site that he rents out.
According to one source close to the family, Cavallari's alcoholism and unpredictable behavior angered the tenants who repeatedly complained to Dennis.
Neighbors also reported hearing regular commotion at the property including visits by the police.
One neighbor of Cavallari recalls seeing him wandering around the back yard shouting, 'the CIA is coming to get me'.
The neighbor, who asked not to be named said: 'Michael would come over early in the morning, around 7am, and ask me if I wanted to drink a beer with him, which was a little odd.
'When I refused he'd get mad. Then I remember a few times we'd see him ranting in the back yard, shouting that the CIA was coming to get him. He seemed paranoid, completely out of it.
'It's sad really, he obviously had an lot of issues going on.'
Other neighbors on the street recall an incident earlier this year in the early hours of the morning during which Michael and his girlfriend were heard screaming at each other during a blazing row.
Eventually, just a few months ago, Dennis, 63, was forced to kick his son out of the home.
The source said: 'Michael is a very difficult person to get along with; he's not all there, crazy.
'He had a room in his dad's house but he was always a huge disturbance to his father's tenants.
'He would play loud music, club bass, until three in the morning, it made one of the tenants very angry, they almost came to blows.