i used
from google.colab import drive
drive.mount('/content/drive')
But only my disk is loading
And the folder in which I have access is not loaded.
I want to upload a file from someone else's google drive to my colab and then use it.
How can I do that?
I have already tried posting the link but get
Mountpoint must be in a directory that exists
I am working on google colaboratory and i have mounted my gdrive to it so i can access the csv data i am processing with pandas for example.
# Mount the path to the location of your data
from google.colab import drive
drive.mount('/gdrive')
Everything works great.
But now when i share my colab to other colleagues they have no access to the csv files ofcourse.
So how can i share these csv residing in my drive as well ? Or are there other alternatives to hosting the data into gdrive so i and my colleagues can access them through a panda read_csv ?
There are 2 easy ways.
1. gdown
If you have just 1-2 csv files that can be shared publicly. You can get the file IDs. Then, use gdown to download it (no authentication).
Get ID from the shared link
https://drive.google.com/file/d/1KgSEm4A7hCov2Ta6-CZldqT-9JgtiVRI/view?usp=sharing
!gdown --id 1KgSEm4A7hCov2Ta6-CZldqT-9JgtiVRI
Your friend will get dataset.zip in the current directory.
2. kora.drive
If you have many csv files/folders, just put them all in a single folder and share that folder to your friend. I made a library to help download the whole folder. It will check user permission to read that folder, by asking you to authenticate.
Get Folder ID from the folder link
https://drive.google.com/drive/folders/1HvIeNhqtFVllXFWzH5NawDdnIfgGDwCK
!pip install kora -q
from kora import drive
drive.download_folder("1HvIeNhqtFVllXFWzH5NawDdnIfgGDwCK")
Your friend will get a new directory Dataset inside current directory (e.g. /content/) with all files/subfolders copied from your google drive.
I use the 'mount drive' to use the dataset folder.
However, the google drive belongs to only me so others who want to run my code on colab have to download the dataset and upload to their google drive, then mount drive to run the code.
Is there any solution for others to run my the code without downloading the dataset?
I made a library to do just that
!pip install kora
from kora import drive
drive.download_folder(folder_id)
There seem to be lots of ways to access a file on Google Drive from Colab but no simple way to save a file from Google Colab back to Google Drive.
For example, to access a Google Drive file from Colab, you can mount the Google Drive using
from google.colab import drive
drive.mount('/content/drive')
However, to save an output file you've generated in Colab on Google Drive the methods seem very complicated as in:
Upload File From Colab to Google Drive Folder
Once Google Drive is mounted, you can even view the drive files in the Table of Contents from Colab. Is there no simple way to save or copy a file created in Colab and visible in the Colab directory back to Google Drive?
Note: I don't want to save it to a local machine using something like
from google.colab import files
files.download('example.txt')
as the file is very large
After you have mounted the drive, you can just copy it there.
# mount it
from google.colab import drive
drive.mount('/content/drive')
# copy it there
!cp example.txt /content/drive/MyDrive
Other answers suggest how to copy a specific file, I would like to mention you can also copy the entire directory, which is useful when copying logs from callbacks from Colab to Drive:
from google.colab import drive
drive.mount('/content/drive')
In my case, the folder names were:
%cp -av "/content/logs/scalars/20201228-215414" "/content/drive/MyDrive/Colab Notebooks/logs/scalars/manual_add"
You can use shutil to copy/move files between colab and google drive
import shutil
shutil.copy("/content/file.doc", "/content/gdrive/file.doc")
When you are saving files, simply specify the Google Drive path for saving the file.
When using large files, Colab sometimes syncs the VM and Drive asynchronously. To force the sync, simply run:
from google.colab import drive
drive.flush_and_unmount()
in my case I use the common approach with the !cp command.
But sometimes, it didn't work in Colab because we didn't enter the right file path.
basic code: !cp source_filepath destination_filepath
implementation code:
!cp /content/myfolder/myitem.txt /content/gdrive/MyDrive/mydrivefolder/
in addition, to correctly enter the path, you can copy the path location from the table of contents on the left side by clicking the dot menu -> copy path.
Once you see the file in the Table of Contents of Colab on the left, simply drag that file into the "/content/drive/My Drive/" directory located on the same panel. Once the file is inside your "My Drive", you will be able to see it inside your Google Drive.
After you mount your drive...
from google.colab import drive
drive.mount('/content/drive')
...just prepend the full path, including the mounted path (/content/drive) to the file you want to write.
someList = []
with open('/content/drive/My Drive/data/file.txt', 'w', encoding='utf8') as output:
for line in someList:
output.write(line + '\n')
In this case we save it in a folder called data located in the root of your Google Drive.
You may often run into quota limits using the gdown library.
Access denied with the following error:
Too many users have viewed or downloaded this file recently. Please
try accessing the file again later. If the file you are trying to
access is particularly large or is shared with many people, it may
take up to 24 hours to be able to view or download the file. If you
still can't access a file after 24 hours, contact your domain
administrator.
You may still be able to access the file from the browser:
https://drive.google.com/uc?id=FILE_ID
No doubt gdown is faster but i copy my files using the command below and avoid quota limits
!cp /content/drive/MyDrive/Dataset/test1.zip /content/dataset
Is it possible to copy a file from Google Drive to Google Cloud Storage? I imagine it would be very fast since both are on a similar storage system.
I haven't seen any information on either ways to do this seamlessly, without having to download the file (probably to an out-of-google system) and then re-uploading it. About one year ago, a similar question was asked: Transfer files from dropbox/drive to Google cloud storage.
Is there a way to do this directly, with the file staying within-Google the entire time?
#frunkad commented a link to a nice workaround, but for completeness I will recite this here, as it is currently the top result in search.
You can open a Colab (Juypiter Notebook on Google servers), mount your gDrive and use the gCloud CLI to copy files.
open https://colab.research.google.com/ and connect to a machine
Mount your Drive by executing the code below, clicking on the link and pasting the authentification key
from google.colab import drive
drive.mount(‘/content/drive’)
Connect to your Google Cloud Storage. (Project id can be found here)
from google.colab import auth
auth.authenticate_user()
project_id = 'your-project-id'
!gcloud config set project {project_id}
!gsutil ls
Copy files using gsutil. Use -m tag for multi-threading to increase speed. (There is a Subfolder called "My Drive" that you have to address in your mounted drive)
bucket_name = 'your_bucket_name'
!gsutil -m cp -r /content/drive/My\ Drive/Your-Data/* gs://{bucket_name}/
Original Author Philip Lies
Link to his colab: https://colab.research.google.com/drive/1Xc8E8mKC4MBvQ6Sw6akd_X5Z1cmHSNca
There's no direct way to do this. You could write an App Engine or Compute Engine app that reads from Drive using its API and writes to GCS using its API.