The issue is that people opposing it don't have a coherent view of what they want. They don't want a free market solution; they don't like the socialist position where the government picks the people; they don't like the lottery either. They want their preferred type of people to get the visa in a manner not too different from quotas/dei/affirmative action etc., which is impossible to engineer. It ultimately ends up creating a compromise brokered by lobbyists and politicians, kind of like how we do everything else in the country.