I also had the impression he was saying he had the conversation with her on Sunday before he left. The entire conversation is about the events on Mother's Day.
He says, well it was Mother's Day and we had the two daughters coming home from a trip and I had that job in Denver. I wanted to get started on Sunday so I told my wife, Listen, the girls are gonna be home later and want to spend Mothers Day with you so do you mind if I go, and the girls were running late, so they texted her on Mothers Day.... etc.
He only refers to Sunday, so that's why I thought he told her in the morning as opposed to another day.