I think they both (FD and MT) went to 80 Mt. Spring together - and then they went to 4 Jefferson Crossing to drop off MT so she could attend to her daughter.
Then, he went back to 80 Mt. Spring to finish whatever he was doing, and he returned to his residence to clean up and pick up MT so they could drop off 30 bags of garbage in Hartford.
I think he kept his phone with him because he was expecting a call from the nanny that JD wasn't home and he needed to get the kids. He didn't want to miss that call.
Speculation only.
jmo