Deep learing on big dataset

1

S3 Browser

If you need to upload dataset from Windows machine you can use https://s3browser.com/ it's nice GUI for S3 buckets.

#Web Service Automation #Automation #Cloud Storage 8 social mentions

2

Poli Systems S3

Pro tip use s3 bucket from polisystems.ch . I use their service when doing training of Real ESERGAN models and works fine and faster than loading dataset from google drive (you might get lot of timeouts and api limits when transfering lot of data from gdrive). For 2 TB you would pay like 11.19€. I also did script that allows you to connect s3 bucket to colab Make sure you create credentials file first [default] Aws_access_key_id = YOURACCESSKEY Aws_secret_access_key = YOURSECRETKEY Then upload it to colab drive and run: !wget https://github.com/kahing/goofys/releases/latest/download/goofys !mkdir ~/.aws !cp credentials ~/.aws/ !mkdir /mnt/s3 !chmod +x goofys !./goofys --profile default --endpoint https://s3.polisystems.ch/ --debug_fuse --debug_s3 default /mnt/s3.

#Cloud Storage #VPS #Backup And Disaster Recovery 1 social mentions