> Is it possible to do this at a smaller scale once a day at the end of the day with "all new content scraped from the Internet nightly"?
If the model has the ability to browse the web (the tech behind ChatGPT by design does, but it is disabled in ChatGPT proper, which is, in some respects, a conservative public demo) and incorporate data in responses, this can “cover” a bit for staleness in the base model.
Even if you can't retrain the model fully daily, with enough resources [0] you can have multiple training sessions running concurrently and just swap in new backend models behind the interface as they arr ready, which, combined with browsing ability. acheives something very similar. This obviously, barring an enormous advantage in underlying tech or access to relevant training data for one player, works most in favor of whoever can subsidize the biggest hardware commitment.
(Given the different training stages, there may be cost efficiency advantages to, say, running less-frequent iterations of the lowest-level training stage but more frequent iterations of the higher-level ones.)
[0] probably an utterly ludicrous investment for all but a handful of firms, but for a ~$trillion firm where this hits very close to their core business? Not so ludicrous.
If the model has the ability to browse the web (the tech behind ChatGPT by design does, but it is disabled in ChatGPT proper, which is, in some respects, a conservative public demo) and incorporate data in responses, this can “cover” a bit for staleness in the base model.
Even if you can't retrain the model fully daily, with enough resources [0] you can have multiple training sessions running concurrently and just swap in new backend models behind the interface as they arr ready, which, combined with browsing ability. acheives something very similar. This obviously, barring an enormous advantage in underlying tech or access to relevant training data for one player, works most in favor of whoever can subsidize the biggest hardware commitment.
(Given the different training stages, there may be cost efficiency advantages to, say, running less-frequent iterations of the lowest-level training stage but more frequent iterations of the higher-level ones.)
[0] probably an utterly ludicrous investment for all but a handful of firms, but for a ~$trillion firm where this hits very close to their core business? Not so ludicrous.