Just a comment about the "average Joe." Isn't this why WVA was formed? The Civil War was also economic. Western Virginia (WVA) was the poor country cousin of VA, an economic reason to forego ties to the South.
West Virginia was indeed formed out of those Virginian counties that refused to secede. And, yes, they refused to do so at least in part because their economies weren't particularly dependent on slave labor. But also because they had long felt neglected by the richer areas east of the Appalachians. Such a secession has been tried (before and after) in other states. The difference in the 1860s was that the West Virginia counties had the support of the federal government.
I think it's too simplistic to say either "the Civil War was fought solely to free or keep the slaves," or "the Civil War wasn't fought over slavery." Slavery as an institution (as well as opposition to it) was so ingrained in American life that it pretty much influenced everything and everyone's thinking. But as always happens, economics also had much to do with the conflict.