Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

30PB / 100Gbps comes down to about a month, 4 links would give you a week, so that seems pretty quite acceptable for a training run, especially since you can overlap the initial loading of the array with the first training, i.e train as data becomes available.

It goes without saying any data pre-processing needs to be done before writing, at the storage site, or on the training GPUs.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: