Double? I doubt that. Typically, I ride ~20km/h on gravel and ~23km/h on well-paved roads at commute speeds (i.e., no sweating). Maybe the gravel roads are terrible in SV, but around here, it hardly makes a difference in speed as gravel roads are well-maintained.
The trail right along the shore is hard dirt, not quite asphalt, but passable. So that one is okay-ish.
But the connecting trail from SJ downtown along the creek towards the bay at some point changes from asphalt to literally a trough full of gravel. You'll lose a lot of speed there, no matter what kind of tires you're using.