If Matt was abused, then possibly some of the other children were also. The guy has no boundaries and I don't think family ties would matter. Besides, he could rationalize that they weren't his biological children but they knew he "loved" them - like any other pedophile.
As I wrote the other day, the worst thing Dottie said on the stand was that Jerry started Second Mile when they found out they couldn't adopt anymore foster children. *warning bell* She had to know he was abusing their son and she wanted him to look elsewhere.
That fits with Matt coming into the hotel room and seeing him there with a boy but doing nothing to help the kid. He had been taught to look the other way, or taught that it was somehow normal.