This is actually a really interesting point/question, and it's one that doesn't have an easy answer. It's true that, overall, most slaves (in pre-civil-war US) were owned by a small number of plantations. But in some southern states, nearly half the White families owned at least one slave. In those families, obviously the women and children didn't themselves "own" any slaves, but they nevertheless controlled that person's labor. Similarly, wealthy plantation owners frequently rented their slaves to poorer whites. So even a relatively unprivileged class of White people benefited from slave labor directly despite themselves getting a raw deal relative to the land owners.
But ultimately, the south fought a civil war in order to preserve the institution of slavery because it was the fundamental organization of society on top of which the economy was built, and many non-elite white men nevertheless allied themselves with this effort.
As for projecting the past, I don't know. It's certainly plausible that White people today are still reaping the benefits that were established by slavery and racism in a broad, systemic sense. As far as I know, my relatives didn't own slaves, but my relatives did do things like get mortgages that Black people couldn't do. That doesn't seem fair does it? It's a little bit like inheriting stolen property. But if you go back far enough, what isn't stolen property?
You say this Civil War bit as if the proletariat has ever had significant say in whether they are commissioned into wars. Even if they individually supported war and slavery, they had no power to start a war over it. It was the wealthy elite.
...in some southern states, nearly half the White families owned at least one slave.
Which states? This is easier to imagine of e.g. South Carolina than of Tennessee.
In general, exaggerating differences in the interests of different portions of the working class is not to the advantage of the working class. American blacks have suffered more from our racist authoritarian capitalist system than poor whites have, but they have both suffered. Many residents of southern states did not enthusiastically join the war effort; this was why they had to have a "Confederate Home Guard" to brutalize conscripts.
I'm not trying to solve all the problems of the working class (which is not a monolith except when viewed through certain narrow theoretical lenses). I am just clarifying that race-based slavery was fundamental to southern society and economy, and it is ahistorical and misleading to suggest that it only or even primarily benefited a tiny minority of wealthy plantation owners.
The antebellum economy was largely agricultural. A family working ten acres of corn with hand- and mule-power did not in any sense benefit from the fact that other landowners had slaves. It would be more accurate to say that they were in competition with enslaved labor. One might as well ask American factory workers how much they've benefited from cheap overseas factory labor.
Upthread you claimed that such small farmers also used slaves. That may have happened occasionally, but the basic requirements of agriculture in a temperate climate (there is a limited time period during which particular tasks can possibly take place) would have made it rare.
But ultimately, the south fought a civil war in order to preserve the institution of slavery because it was the fundamental organization of society on top of which the economy was built, and many non-elite white men nevertheless allied themselves with this effort.
As for projecting the past, I don't know. It's certainly plausible that White people today are still reaping the benefits that were established by slavery and racism in a broad, systemic sense. As far as I know, my relatives didn't own slaves, but my relatives did do things like get mortgages that Black people couldn't do. That doesn't seem fair does it? It's a little bit like inheriting stolen property. But if you go back far enough, what isn't stolen property?