Mounting Google Drive on Colab - google-drive-api

I'm trying to mount my drive on Colab using the following code:
from google.colab import drive
drive._mount('/content/drive')
And I'm getting the following error:
298 # Terminate the DriveFS binary before killing bash.
ValueError: mount failed: invalid oauth code
Tried deleting cookies, restarting the session, but nothing seems to work. I am using a few different google accounts, and only one has a Colab Pro subscription.. but this wasn't an issue till today.
Thanks.

for the moment the only solution that is working right now is this from this similar question but two months ago:
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
%cd /content
!mkdir drive
%cd drive
!mkdir MyDrive
%cd ..
%cd ..
!google-drive-ocamlfuse /content/drive/MyDrive
Let's hope the normal way drive.mount will be fixed soon !

Here is the solution you can try:
As of new (messed up) update you can not mount using drive._mount anymore (or at least for now). So what you can do is either copy your folder/directory from your other google drive to your current google drive (the account you are log in to colab with) OR you can simply log out from colab and log in with your google account that has the drive you want to mount to
Remember
use drive.mount and not drive._mount after following the above guideline

Related

Google-chrome on AWS Lambda

It is possible to run Google-chrome not Chromium with puppeteer in AWS Lambda with container?
Script stuck when I create new page in browser:
const page = await browser.newPage();
Logs from AWS lambda:
mkdir: cannot create directory ‘/.local’: Read-only file system
touch: cannot touch ‘/.local/share/applications/mimeapps.list’: No such file or directory
/usr/bin/google-chrome-stable: line 45: /dev/fd/62: No such file or directory
/usr/bin/google-chrome-stable: line 46: /dev/fd/62: No such file or directory
[0213/000419.523205:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[0213/000419.528197:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[0213/000419.648505:WARNING:audio_manager_linux.cc(60)] Falling back to ALSA for audio output. PulseAudio is not available or could not be initialized.
DevTools listening on ws://127.0.0.1:46195/devtools/browser/1d348770-1c99-48a5-934c-fae5254fc766
[0213/000419.769218:WARNING:bluez_dbus_manager.cc(248)] Floss manager not present, cannot set Floss enable/disable.
prctl(PR_SET_NO_NEW_PRIVS) failed
prctl(PR_SET_NO_NEW_PRIVS) failed
I do not use puppeteer but that doesn't matter much.
FROM public.ecr.aws/lambda/provided:al2
RUN yum install unzip atk at-spi2-atk gtk3 cups-libs pango libdrm \
libXcomposite libXcursor libXdamage libXext libXtst libXt \
libXrandr libXScrnSaver alsa-lib \
xorg-x11-server-Xvfb wget shadow-utils -y
COPY install-chrome.sh /tmp/
RUN /usr/bin/bash /tmp/install-chrome.sh
ENV DBUS_SESSION_BUS_ADDRESS="/dev/null"
I am not 100% DBUS_SESSION_BUS_ADDRESS is necessary. I am also not 100% sure whether explicitly naming all these packages are necessary, I stole everything from a dozen different places, likely the chrome rpm will pull in what it needs, but I never used any RHEL based system so I am totally clueless. I know this works. Optimizations are welcome.
Here's the script:
#!/usr/bin/bash
# Download and install chrome
wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
# Without -y it doesn't run because it needs to add dependencies.
yum install -y google-chrome-stable_current_x86_64.rpm
rm google-chrome-stable_current_x86_64.rpm
CHROMEVERSION=`wget -qO- https://chromedriver.storage.googleapis.com/LATEST_RELEASE`
wget --no-verbose -O /tmp/chromedriver_linux64.zip https://chromedriver.storage.googleapis.com/$CHROMEVERSION/chromedriver_linux64.zip
unzip /tmp/chromedriver_linux64.zip -d /opt
rm /tmp/chromedriver_linux64.zip
mv /opt/chromedriver /opt/chromedriver-$CHROMEVERSION
chmod 755 /opt/chromedriver-$CHROMEVERSION
ln -fs /opt/chromedriver-$CHROMEVERSION /usr/local/bin/chromedriver
# Create a user. /usr/sbin is not on $PATH.
/usr/sbin/groupadd --system chrome
/usr/sbin/useradd --system --create-home --gid chrome --groups audio,video chrome
You can verify it is working by starting it locally with docker run --mount type=tmpfs,destination=/tmp --read-only this simulates well the environment of AWS Lambda. Then you need to run su chrome -c 'xvfb-run chromedriver --allowed-ips=127.0.0.1'. I am using https://github.com/instaclick/php-webdriver/ which is a very thin PHP client for W3C and Selenium 2 webdriver. I used this to test:
<?php
namespace WebDriver;
require 'vendor/autoload.php';
#mkdir('/tmp/chrome');
chmod('/tmp/chrome', 0777);
$wd_host = 'http://localhost:9515';
$web_driver = new WebDriver($wd_host);
$session = $web_driver->session('chrome', [['goog:chromeOptions' => ['args' => [
'--no-sandbox',
'--user-data-dir=/tmp/chrome'
]]]]);
$session->open('https://google.com');

Zipping and/or transferring very large files (200+ GB) on Google Drive using Colab

I've been using a script to mount two google drives on one google drive runtime to move files around, but it complains that I'm out of space when I'm moving the large files. I tried running it one file at a time and I'm still stuck on the large files...
Edit:
I use
from google.colab import drive
drive.mount('/drive1')
and
!apt-get install -y -qq software-properties-common module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
!mkdir -p /drive2
!google-drive-ocamlfuse /drive2
to mount the two drives, then I tried a simple copy:
!cp -vru /drive2/folder /drive1/folder
but it kept failing due to the drive space. I also tried splitting the files using zip (without copying it to the google colab file system first), so all the data stays in the google drive
! zip -r -db -dc -Z bzip2 -T -m -s 5g /drive1/zipfilename /drive1/folder
but I still kept running out of space.

Unable to locate package google-drive-ocamlfuse , suddenly stopped working

I have been using colab lately and till now I used google-drive-ocamlfuse to link my project to my drive but not it is unable to load package.
!apt-get update -qq 2>&1 > /dev/null
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!add-apt-repository ppa:alessandro-strada/google-drive-ocamlfuse-beta
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
got referenced form here
https://medium.com/#burakteke/tutorial-on-using-google-colab-for-kaggle-competition-620393c22821
till yesterday it was working like a charm but today it suddenly stopped working.
This should work fine as per https://github.com/astrada/google-drive-ocamlfuse/issues/493#issuecomment-422380636
Change the build between amd64, arm64, armhf, i386 from https://launchpad.net/~alessandro-strada/+archive/ubuntu/ppa/+packages
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!wget https://launchpad.net/~alessandro-strada/+archive/ubuntu/google-drive-ocamlfuse-beta/+build/15331130/+files/google-drive-ocamlfuse_0.7.0-0ubuntu1_amd64.deb
!dpkg -i google-drive-ocamlfuse_0.7.0-0ubuntu1_amd64.deb
!apt-get install -f
!apt-get -y install -qq fuse
from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
An even simpler fix is to use the built-in Drive FUSE client:
from google.colab import drive
drive.mount('/content/gdrive')
More details:
https://colab.research.google.com/notebooks/io.ipynb#scrollTo=u22w3BFiOveA

Access data from Google Team Drive in Google Colaboratory

is there a way to access my data from Google Team Drive? I used the provided code, but can only access my normal drive and not Team Drive:
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
I appreciate every help!
You may use the REST API to manage your Team Drive. Check this documentation. Here's a list of methods for this resource:
create
Creates a new Team Drive.
delete
Permanently deletes a Team Drive for which the user is an organizer. The Team Drive cannot contain any untrashed items.
get
Gets a Team Drive's metadata by ID.
list
Lists the user's Team Drives.
update
Updates a Team Drive's metadata

Unable to install Couchbase sync gateway

I am trying to install Couchbase sync gateway using the steps from the following URL for MacOS
https://developer.couchbase.com/documentation/mobile/current/installation/sync-gateway/index.html
This issue is, i downloaded "couchbase-sync-gateway-enterprise_1.4.1-3_x86_64.tar.gz" and its in "Downloads" folder in MacBook.
When I execute this command -> sudo tar -zxvf couchbase-sync-gateway-enterprise_1.4.1-3_x86_64.tar.gz --directory /opt
[MyMacbook:downloads administrator$ sudo tar -zxvf couchbase-sync-gateway-enterprise_1.4.1-3_x86_64.tar.gz --directory /opt
tar: could not chdir to '/opt']
Throws error as "tar: could not chdir to '/opt'"
I don't understand how to resolve this. I couldn't get help anywhere.
Please help installing Couchbase sync gateway successfully on MacBook.
It dosen't actually matter if /opt does not exist on your Mac. You can either create one before firing the command with the following command:
sudo mkdir /opt
Otherwise, you can replace "opt" with locations like : /Users/< your_user > or /usr/share or /var