Fair... but lets not pretend that these non slave owners didnt understand what "the cause" was about either. The Confederate Constitution was nearly the same as our own, save a Line item veto by the executive and the line "No bill of attainder, ex post facto law, or law denying or impairing the right of property in negro slaves shall be passed."
Also in an 1861 speech Alexander Stephens, Vice President of the Confederacy proclaimed that slavery was not only the cause for succession but also the "Cornerstone" of the Confederate Nation. I get your argument that if someone invaded my home, I would take up arms against them too, but lets not act like these soldiers weren't complicit in the continuation of the institution of slavery.