I've been thinking about this today...
I had a co-worker at my last job who wanted to take her 10 year old grandson from her daughter. My co-worker had money, the daughter did not.
However, my co-worker's "evidence" of her being an unfit mother was not really abusive, IMO. My co-worker was upset that her daughter did not do things like take the grandson to the dentist, or allow him to participate in activities outside of school. The daughter was probably somewhat depressed, and slept a lot, and her house was apparently pretty messy. Not filthy, but things like stacks of unopened mail, 15 empty milk jugs waiting to go to recycling, etc.
My coworker's attorney said the judge would ask things like "was their evidence of abuse?" and she would need to have something more than stacks of mail to prove abuse. She pressed on, however, with nothing more incriminating than the grandson was not allowed to join the running/track group after school because his mom would have to go get him during one of her "programs". In the end, she paid over $10,000, and the grandson was in foster care for three months (apparently difficult for him, as it would be for any child) while they investigated. The daughter cleaned her house, although they did not say she had to. And in the end the judge let the daughter have the grandson back, and the daughter won't speak to the mom at all. It's been over a year.
I think it is tempting in hindsight to say, well yeah, she should have gotten custody and none of this would have happened. But I don't think she had the money OR the evidence of abuse needed to make it a reality. I know you guys have said this already, I am just adding my friend's story for another true life example.