Not so far-fetched, Mystirose. I've entertained that theory myself. The problem with it, though, as with many of our theories, is that if TN took the kids, Jr. would've said that to Crystal. Now, if TN stayed at the house with the kids and Jr. was asleep the whole time, then it's more plausible. If the request to Tommy to go to the MH was a way to set him up, as I'm inclined to believe, TN might've been in the house with the lights off when he was knocking.
I think those kids were passed around so much that they (esp) JR, wouldn't remember where they were on that day or night they were home; with TN, GGM, Misty, or one of her relatives. Remember he could have been drugged and things perhaps were a little confused by all that went on. JMO