Yep, first load isn't very quick, as there are a lot of things that need to be loaded which are never going to be very small. However, first load only happens once. After that the assets should be cached in your browser and loads should be much faster.
There is definitely something to be said for faster first loads, but unlike many other sites on the web, ours is of course optimized for consistent/repeated use, so in the scheme of things first load is negligible compared to making sure it runs fast while actually using it 100s of subsequent times, which (I hope) it does.
Definitely wouldn't let that discourage you from learning React. If you want a smaller bundle size, you can use Preact[1], which is nearly a drop-in replacement for React's runtime but much smaller.
This has finally convinced me to not waste my time learning React, at least for now.