Visa and Mastercard run debit networks for majority of banks and credit unions. They get fees there as well.

Even lower income citizens use debit cards more than cash nowadays.

You would need to use different networks like Discover and American Express to effectively boycott them