Not to speak of all the injuries and fatalities happening in those factories day in day out. On top of all the other negative health effects from dust and such.

But somehow the rich dudes in their villas, and especially their money, need protection.

Luddites were very careful not to hurt any people (I think in one case they were cornered by defensive action, but that was clearly in self-defense), and they even only targeted specific machines they had problems with, leaving all other machines intact. But just the same I don't agree with their methods since any violence is easily co-opted to paint them as bad and ultimately hurts the goals they were trying to achieve.

Violence certainly wasn't the first thing they tried here

You people need to understand that the Unabomber was also a Luddite. Propaganda works in both ways.