Swallow it, crush it, flush it, hide it under the carpet .. so many options. To throw it out of the window seems panicked and as you say, bizarre. But yes, it could absolutely point to his unhinged thinking. Just a blind panic to get rid of evidence.
In terms of a weapon, he would not be allowed to bring his gun home from work. So he’d have to hold a private firearms license. If he did so, he’d have to have a gun cabinet with the weapon/s securely looked away. With two young children in the house, he’d be stupid not to adhere to this. I can’t see it being a gun.
A mobile phone is possible IMO. In the hope it would smash. Again, pretty dumb and futile. But he might hope it would destroy traceable contents. That could be anything from GPS tracking to web history.