Taking subsets of datasets and cross validation

woolford180

I've noticed that once a dataset is loaded there is no feature to select a smaller subset of the loaded dataset. What if I am trying to train a neural network on a computer with not enough resources to load the full dataset into system memory.

Are you planning on adding functionality that allows users to split the dataset into chunks for training on smaller sets of data and cross-validation purposes?

Also, implementing functionality that allows the data to be sent to remote servers in chunks? That way it is easier to train on servers with limited ram?

robin_lobel

There's a parameter for that: Usage (in the Dataset tab, that's the first parameter of the Formatting panel). You can have a look at this tutorial showing how to use it: https://torchstudio.ai/transferlearning/

Some more clarifications: when you click Load, it doesn't mean the dataset is loaded in memory - it just mean the dataset is now ready to serve samples. When you browse the samples in the Formatting panel, they are just streamed live one by one from the dataset on demand.

However when you launch a training, all the dataset samples (or 20% of those if you reduced Usage to 20%) are indeed streamed and stored in the training server memory. There will be an option in a couple updates from now to store those samples in the training server harddrive instead of the training server memory - a bit slower, but the training server memory requirements are then much much lower.