I have been using colab lately and till now I used google-drive-ocamlfuse to link my project to my drive but not it is unable to load package.
!apt-get update -qq 2>&1 > /dev/null
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!add-apt-repository ppa:alessandro-strada/google-drive-ocamlfuse-beta
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
got referenced form here
https://medium.com/#burakteke/tutorial-on-using-google-colab-for-kaggle-competition-620393c22821
till yesterday it was working like a charm but today it suddenly stopped working.
This should work fine as per https://github.com/astrada/google-drive-ocamlfuse/issues/493#issuecomment-422380636
Change the build between amd64, arm64, armhf, i386 from https://launchpad.net/~alessandro-strada/+archive/ubuntu/ppa/+packages
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!wget https://launchpad.net/~alessandro-strada/+archive/ubuntu/google-drive-ocamlfuse-beta/+build/15331130/+files/google-drive-ocamlfuse_0.7.0-0ubuntu1_amd64.deb
!dpkg -i google-drive-ocamlfuse_0.7.0-0ubuntu1_amd64.deb
!apt-get install -f
!apt-get -y install -qq fuse
from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
An even simpler fix is to use the built-in Drive FUSE client:
from google.colab import drive
drive.mount('/content/gdrive')
More details:
https://colab.research.google.com/notebooks/io.ipynb#scrollTo=u22w3BFiOveA
Related
I'm trying to mount my drive on Colab using the following code:
from google.colab import drive
drive._mount('/content/drive')
And I'm getting the following error:
298 # Terminate the DriveFS binary before killing bash.
ValueError: mount failed: invalid oauth code
Tried deleting cookies, restarting the session, but nothing seems to work. I am using a few different google accounts, and only one has a Colab Pro subscription.. but this wasn't an issue till today.
Thanks.
for the moment the only solution that is working right now is this from this similar question but two months ago:
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
%cd /content
!mkdir drive
%cd drive
!mkdir MyDrive
%cd ..
%cd ..
!google-drive-ocamlfuse /content/drive/MyDrive
Let's hope the normal way drive.mount will be fixed soon !
Here is the solution you can try:
As of new (messed up) update you can not mount using drive._mount anymore (or at least for now). So what you can do is either copy your folder/directory from your other google drive to your current google drive (the account you are log in to colab with) OR you can simply log out from colab and log in with your google account that has the drive you want to mount to
Remember
use drive.mount and not drive._mount after following the above guideline
I've been using a script to mount two google drives on one google drive runtime to move files around, but it complains that I'm out of space when I'm moving the large files. I tried running it one file at a time and I'm still stuck on the large files...
Edit:
I use
from google.colab import drive
drive.mount('/drive1')
and
!apt-get install -y -qq software-properties-common module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
!mkdir -p /drive2
!google-drive-ocamlfuse /drive2
to mount the two drives, then I tried a simple copy:
!cp -vru /drive2/folder /drive1/folder
but it kept failing due to the drive space. I also tried splitting the files using zip (without copying it to the google colab file system first), so all the data stays in the google drive
! zip -r -db -dc -Z bzip2 -T -m -s 5g /drive1/zipfilename /drive1/folder
but I still kept running out of space.
Is possible to run puppeteer on my Debian environment ? I don't have the desktop environment installed on it.
When I run my example.js script I have a Error: Failed to launch chrome! libX11-xcb.so.1: cannot open shared object file: No such file or directory
Thx
Ok I found the solution:
Install the Debian required dependencies to use chromium:
apt-get install gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
And launch the browser without the sandbox:
example.js:
const puppeteer = require('puppeteer');
(async() => {
const browser = await puppeteer.launch({args: ['--no-sandbox']});
const page = await browser.newPage();
await page.goto('https://example.com');
await page.screenshot({path: 'example.png'});
browser.close();
})();
Maybe some useful documentation here:
https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#running-puppeteer-in-docker
The use of --no-sandbox is described as "strongly discouraged", but there are hints about running Puppeteer in a Docker container which should solve the missing shared libraries issues.
The Dockerfile suggested in the above mentioned link is:
FROM node:12-slim
# Install latest chrome dev package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
# Note: this installs the necessary libs to make the bundled version of Chromium that Puppeteer
# installs, work.
RUN apt-get update \
&& apt-get install -y wget gnupg \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
# If running Docker >= 1.13.0 use docker run's --init arg to reap zombie processes, otherwise
# uncomment the following lines to have `dumb-init` as PID 1
# ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_x86_64 /usr/local/bin/dumb-init
# RUN chmod +x /usr/local/bin/dumb-init
# ENTRYPOINT ["dumb-init", "--"]
# Uncomment to skip the chromium download when installing puppeteer. If you do,
# you'll need to launch puppeteer with:
# browser.launch({executablePath: 'google-chrome-stable'})
# ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
# Install puppeteer so it's available in the container.
RUN npm i puppeteer \
# Add user so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space
&& groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser \
&& chown -R pptruser:pptruser /node_modules
# Run everything after as non-privileged user.
USER pptruser
CMD ["google-chrome-stable"]
Then the suggested Docker build and run commands are:
docker build -t puppeteer-chrome-linux .
docker run -i --init --rm --cap-add=SYS_ADMIN \
--name puppeteer-chrome puppeteer-chrome-linux \
node -e "`cat yourscript.js`"
I tested the process described here above myself , and after that received a no usable sandbox error (therefore the missing shared object error appeared to be solved by it).
Concerning the sandbox related error, suggested resolution from the same source than mentioned here above, is enabling user namespace cloning.
sudo sysctl -w kernel.unprivileged_userns_clone=1
is there a way to access my data from Google Team Drive? I used the provided code, but can only access my normal drive and not Team Drive:
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse
from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()
import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
I appreciate every help!
You may use the REST API to manage your Team Drive. Check this documentation. Here's a list of methods for this resource:
create
Creates a new Team Drive.
delete
Permanently deletes a Team Drive for which the user is an organizer. The Team Drive cannot contain any untrashed items.
get
Gets a Team Drive's metadata by ID.
list
Lists the user's Team Drives.
update
Updates a Team Drive's metadata
I want to get chrome to work headless with selenium on an ubuntu server so that I can run tests against it.
here's the commands I ran to set up chrome headless using xvfb.
Am I missing anything?
apt-get update && apt-get upgrade
apt-get install -y libxss1 libappindicator1 libindicator7 unzip
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
sudo apt-get update
sudo apt-get install google-chrome-stable
sudo apt-get install -y xvfb xfonts-100dpi xfonts-75dpi xfonts-cyrillic xorg dbus-x11
wget -N http://chromedriver.storage.googleapis.com/2.24/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
chmod 755 chromedriver
sudo mv -f chromedriver /usr/local/share/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver
sudo apt-get install -y python-pip libosmesa6
sudo apt-get update && sudo apt-get install -y xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic xvfb x11-apps imagemagick firefox google-chrome-stable
pip install pyvirtualdisplay selenium
sudo apt-get install firefox -y
useradd -m jenkins-user
mkdir /home/jenkins-user/log
chown jenkins-user:root /home/jenkins-user/log
sudo apt-get install -y default-jdk
apt-get install -y git
iptables -N SELENIUM
iptables -A INPUT -p tcp --dport 4444 -j SELENIUM
iptables -A SELENIUM -p tcp --dport 4444 -s 127.0.0.1 -j ACCEPT
iptables -A SELENIUM -p tcp --dport 4444 -j DROP
apt-get install jq
Here is a snippet from where I create my driver instance.
This is working on my mac, but have not tested this on ubuntu yet. I'll be doing that very soon.
Note the option: --disable-gpu. This is what made it to work on my local machine.
System.setProperty("webdriver.chrome.driver", chromeAbsolutePath + "");
ChromeOptions options = new ChromeOptions();
options.addArguments("--disable-extensions");
options.addArguments("--headless");
options.addArguments("--disable-gpu");
DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability("acceptInsecureCerts", false);
capabilities.setCapability("acceptSslCerts", false);
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
driver = new ChromeDriver(capabilities);
System.out.println(BROWSER + " on local machine initiated");