Google Colab Disk space vs Google Drive disk space - google-drive-api

I'm using Google Colab to do some machine learning project. I've mounted my drive, activated the GPU, purchased extra storage space from google drive, and have over 100Gb of free space on google drive, but the "drive" monitor in my Colab notebook says that the drive is filling up. Screen shots of my Colab notbook and Google Drive storage below.
Why does the notebook show that the drive is filling up? I've refreshed my google drive several times to make sure that the storage space that's showing is accurate.

The Google Drive storage and Google Colab disk space are different.
Google drive storage is the space given in the google cloud. whereas the colab disk space is the amount of storage in the machine alloted to you at that time. You can increase the storage by changing the runtime.
A machine with GPU has more memory and diskspace than a runtime with cpu only.
Similarly if you want more, you can change the runtime to a TPU machine.

Google drive storage and google colab disk are different. Google drive storage is object storage system while google colab disk is a File system that you use during the colab runtime(mostly a SSD, or a HDD). Both are different and have different usage (read here for more).
The problem you are facing is the shortage of SSD disk during the runtime. Google colab has no option to increase this. You need to shift to cloud notebooks to increase the size of the runtime SSD.

Related

How can I speed up unzipping / reading files with a mounted Google Drive in Colab?

I'm using Colab with a mounted Google Drive to unpack zips and consolidate the csvs that come out of them. But this, for example:
for z in zip_list:
zipfile.ZipFile(z, 'r').extractall()
zipfile.ZipFile(z, 'r').close()
os.remove(z)
runs about 60x slower in Colab/Drive compared to when I run it on my local computer. Why is this so much slower and how can I fix it?
A typical strategy is to copy the .zip file from Drive to the local disk first.
Unzipping involves lots of small operations like file creation, which are much faster on a local disk than Drive, which is remote.

Google drive disconnect from COLAB

I am working on neural networks in Keras and I use Colab to train my network. Unfortunately, any time I stop the training, one of the following problems occurs:
Colab unmounts my gdrive folder. So I must remount it to restart the training.
My gdrive folder on Colab partially empties (I loose my dataset). In this case I also need to restart the session in order to remount gdrive.
Does anyone know the reasons ?
By stopping the training, do you mean stopping the kernel?
If you stop or restart the kernel, the drive will be unmounted.
If you want your training to continue, save your models into checkpoints.

Google drive synch from "Drive to PC"

I have created a new file and google drive through the browser.
But I am unable to see the new file on the connected google drive physic PC drive.
Is this not achievable with google drive?
Only one side of synch only work(PC to Google Drive)? Two way syncing is not possible (PC to Google Drive and Google drive to PC)?
AFAIK, this is a one way sync (PC to Drive not the other way around). When installing the Google Drive app and do a backup and sync, Google Drive acts as a Cloud backup for any changes to the selected folders you've wanted to backup.
As stated in this guide:
The Backup and Sync tool is split up into two main sections:
Google Drive: This performs the same function as the original Google Drive app. You choose what folders to sync from your Google Drive cloud storage, and they’ll appear in a Google Drive folder on your PC. Anything you put into that folder will also sync to Google Drive.
My Computer: This part is new, and allows you to sync files between your computer and Drive without putting them in the dedicated Google Drive folder. Just pick the folders from your computer you want to sync, and they’ll sync to your cloud storage (though they’ll appear in a separate section of the Google Drive interface, rather than with all your other Drive files.)
Hope this helps.

Google Drive upload in Service Account

We have a requirement where we should provide capability to upload files up to 100 GB size. Current flow which we have is to put the file from client location/local system to the application server. Then application server pushes the file to a service account in Google Drive server. I would like to know if there is a way to push the file from local system directly to service account in Google Drive. This would help us to not have to store such big files in application server. Anyone has any suggestion on this ?
Service accounts are not meant to be used like this. They have limited quota: Google Drive Service Accounts Additional Storage

File size limits on Samba

Our shared drive in the office is synced to dropbox, so I want to stop people putting large (> 400MB) files on it.
The shared drive is on a PC running Ubuntu, and we use Samba to share the drive with the office PCs (all running Windows 7).
Is it possible to put a limit on file size?
Sometimes this works:
[global]
max disk size = 100
(=100MB)