In my experience, all center-fire rifles and handguns sound loud.
Outdoors, in the still of night, the sound will travel, even echo or almost linger for a time. To me, it sounds similar to a lightening crack. Inside, the walls and insulation would absorb some of the sound. In either case, you may hear it within several hundred yards, but it's not likely anyone, even with a great deal of experience could tell what kind, caliber, action-type, etc. of firearm it was from inside the house next door.
The neighbor (man) said it wasn't a handgun, .22 or .410.
A .410 is a small shot-shell that can be shot from pistol or long-barrel shotgun.
A .22 is rim-fire, one of the smallest rifle/pistol calibers.
Both are small, with very little powder, compared to larger gauge/caliber.
The neighbor sounded like he knows something about guns, enough to have an opinion. And if he discounts .22 or .410 I could take his word on that, but handgun are loud too.
Some common-use handguns and rifles both fall in the 160-170 decibels (dB) range and other common-use handguns and rifles both fall in the 140-160 dB range. Hearing damage occurs over 140, so, IMO, without a dB meter, you are probably guessing and odds are 50/50 of getting it right.