Experts do disagree with one another often enough... unless they're all in the pay of a single interest group or our filter bubbles prevent us from seeing the experts on one side of the disagreement. You might not think either of those is happening, but if you trust the experts first how would you know if the former was happening? And wouldn't removing people make the latter failure scenario more likely?