If that panel was levered outwards, the inner corner of the handle would have marred the wood and left those gouges in it.
Allso, does anyone know the height of the bathroom and if it has enough room for a bat to be swung overhead? If you're going to take a door apart, you do it with overhead...