quite a staggering number for a nation of only slightly more than 31,000,000.
And we did almost say "F*** 'em, we're better off without them". One of Lincoln's greatest political struggles, as the war ground on with mounting casualty figures and back-to-back Northern defeats, was keeping the North committed to the war. Even as late as 1864, after the tide had really turned on the battlefield, Grant withheld the true casualty figures from the Cold Harbor battle because of fear of the impact on the presidential election to be held in November of that year. As the war dragged on, increasing numbers of Northerners were inclined to just let the South go.
As to your question: Like most historical questions, the answers are going to be multi-faceted. Along with all the socio-economic factors others have listed, I don't think you can discount the real sense in the North that secession was tantamount to treason against the nation. Lincoln certainly viewed it that way, although his policy prior to Ft. Sumter was to take a light hand toward the South, in hopes that better minds and cooler heads would prevail. It should be remembered that there was distinct anti-secession sentiment in the South as well; primarily for economic reasons, but there was a distinct minority of southerners who also viewed secession as treason.
The fact that the first tier of seven states to secede did so through a series of faits accomplis prior to the mandated requirement for popular conventions to decide the matter, speaks volumes.
The shelling of Ft. Sumter, as well as the surrender/capture of several other Federal military installations in Florida, and along the Gulf Coast during the same time period, marked the end of discussion.
it should be remembered that secession was threatened prior to the Civil War. Andrew Jackson faced the same threats of secession from South Carolina over the tariff of Abominations, and responded with the threat of force; although, in that case, South Carolina backed down, and forestalled the eventual for about 30 years.