The mistake you make is to think of two groups, when there are actually three. Active, passive and "outiders", who got their stock outside of the stock market. Maybe they are early investors or maybe they get stock as part of their compensation package.

If passive investors make worse trades with outsiders than the active investors do, they will underperfom.