If it is true that non-HTML resources are a factor based on Google Toolbar reporting, that's scary. On high confidence sites -- more than 1000 datapoints, in Google Webmaster terms -- average speed seems stable and correlated with other performance measures. This is not true in my experience with medium confidence sites, 100 to 1000 Toolbar datapoints.

I'm hopeful that Googlebot is the primary signal.