I need to download whole folder from google drive using wget.
How can I do that?
"You can't" more and less...
pip install gdown
gdown --no-check-certificate --folder <folder URL>
I've been using it for a while in macOS and Ubuntu.
You can download zipped files like this:
Suppose this is the shared link from your google drive:
https://drive.google.com/file/d/345efwerfer23rffedser/view?usp=sharing
copy this part of the shared_link: 345efwerfer23rffedser
to bellow command
!wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=345efwerfer23rffedser' -O 'files.zip'
So I've been trying to get wget to download a Google Drive file that I uploaded. Unfortunately, Google Drive incorrectly flags the file as a virus, so wget can't get the direct download link.
Things I've tried:
using the gdrive.pl fie that someone made, but I'm on Windows, and /tmp/cookies.txt does not exist.
doing wget --no-check-certificate https://docs.google.com/uc?export=download&id=FILEID -O FILENAME, but it says 400 Bad Request
using https://docs.google.com/uc?export=download&id=ID, but it fails because of the download infected file warning.
Does anyone have any suggestions to solve this?
Here is what I was able to do, based on a starting point I found at https://medium.com/#acpanjan/download-google-drive-files-using-wget-3c2c025a8b99 :
Edit I noticed you said Windows, so this command with sed won't work natively in Windows - I'll put steps without sed for Windows below
You of course start by sharing the file and getting the file ID from the share link on google drive. Then:
wget --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=SHARE_LINK_ID" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p' > /tmp/confirm && wget --load-cookies /tmp/cookies.txt --no-check-certificate "https://docs.google.com/uc?export=download&confirm="$(cat /tmp/confirm)"&id=SHARE_LINK_ID" -O YOUR_FILENAME && rm /tmp/cookies.txt /tmp/confirm
Replace SHARE_LINK_ID with your ID from your shared file link. Replace YOUR_FILENAME with your desired output file name.
This attempts to download the file and gets the html of the warning message about potential viruses in the file. It uses cookies as you need to use the same session ID for the subsequent download with the confirmation code.
It then gets the generated confirm code from that response and writes it to a temporary file.
I then does another wget adding the confirmation code to the query string to download the file, using the saved cookie to allow the confirmation code to work for the saved session.
Most likely this could be worked into a script, passing an argument of the share link ID to make it more useful.
For Windows (without sed)
wget --save-cookies %TMP%/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=SHARE_LINK_ID" -O %TMP%/confirm.txt
Downloads the confirmation html.
notepad %TMP%/confirm.txt
Opens %TMP%/confirm.txt in Notepad to get the confirm code string (CTRL+F to look for "confirm=" and get the code right after that). Replace it in the below command line (along with putting in the filename you want and the share link ID from google drive)
wget --load-cookies %TMP%/cookies.txt --no-check-certificate "https://docs.google.com/uc?export=download&confirm=CONFIRM_CODE&id=SHARE_LINK_ID" -O YOUR_FILENAME
Delete the temp files:
del %TMP%/cookies.txt %TMP%/confirm.txt
Try this. Don't forget to replace two FILEID and one FILENAME fields with your desired file's file id and the output file's name respectively.
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=FILEID" -O FILENAME && rm -rf /tmp/cookies.txt
source: https://medium.com/geekculture/wget-large-files-from-google-drive-336ba2e1c991
I'm learning data analysis in Zeppelin, I'm a mechanical engineer so this is outside my expertise.
I am trying to download two csv files using a file that contains the urls, test2.txt. When I run it I get no output, but no error message either. I've included a link to a screenshot showing my code and the results.
When I go into Ambari Sandbox I cannot find any files created. I'm assuming the directory the file is in is where the csv files will be downloaded too. I've tried using -P as well with no luck. I've checked in man wget but it did not help.
So I have several questions:
How do I show the output from running wget?
Where is the default directory that wget stores files?
Do I need additional data in the file other than just the URLs?
Screenshot: Code and Output for %sh
Thanks for any and all help.
%sh
wget -i /tmp/test2.txt
%sh
# list the current working directory
pwd # output: home/zeppelin
# make a new folder, created in "tmp" because it is temporary
mkdir -p /home/zeppelin/tmp/Folder_Name
# change directory to new folder
cd /home/zeppelin/tmp/Folder_Name
# transfer the file from the sandbox to the current working directory
hadoop fs -get /tmp/test2.txt /home/zeppelin/tmp/Folder_Name/
# download the URL
wget -i test2.txt
I downloaded a website with HTTrack using the following command:
/usr/local/bin/httrack https://www.website.com -O /Users/mainuser/Desktop/website -n -j
I than located the index.html file in website folder and run it. Chrome returns the message: file not found. That's funny, because normally the websites I parse with httrack work just fine on my file system. What cold be the reason for this behaviour?
Try the following, using the v (verbose) flag to enable HTTrack to give you as much debugging information as possible:
/usr/local/bin/httrack "https://www.website.com" -O "/Users/mainuser/Desktop/website" -n -j -v
I have a problem with running tesseract-ocr engine on linux. I've downloaded RUS language data and put it to tessdata directory (/usr/local/share/tessdata). When I'm trying to run tesseract with command tesseract blob.jpg out -l rus , it displays an error:
Error opening data file /usr/local/share/tessdata/eng.traineddata
Please make sure the TESSDATA_PREFIX environment variable is set to the parent directory of your "tessdata" directory.
Failed loading language eng
Tesseract couldn't load any languages!
Could not initialize tesseract.
According to compiling guide, I used export TESSDATA_PREFIX='/usr/local/share/'
to point my tessdata directory.
Maybe I should edit any config files? Tesseract try to load 'eng' data files instead of 'rus'.
Screenshot:
http://i.stack.imgur.com/I0Guc.png
You can grab eng.traineddata Github:
wget https://github.com/tesseract-ocr/tessdata/raw/main/eng.traineddata
Check https://github.com/tesseract-ocr/tessdata for a full list of trained language data.
When you grab the file(s), move them to the /usr/local/share/tessdata folder. Warning: some Linux distributions (such as openSUSE and Ubuntu) may be expecting it in /usr/share/tessdata instead.
# If you got the data from Google, unzip it first!
gunzip eng.traineddata.gz
# Move the data
sudo mv -v eng.traineddata /usr/local/share/tessdata/
The simpliest way is to install the needed package:
sudo apt-get install tesseract-ocr-eng #for english
sudo apt-get install tesseract-ocr-tam #for tamil
sudo apt-get install tesseract-ocr-deu #for deutsch (German)
As you can notice, it opens the road to others languages (i.e. tesseract-ocr-fra).
I had this error too on the Windows machine.
My solution.
1) Download your language files from
https://github.com/tesseract-ocr/tessdata/tree/3.04.00
For example, for eng, I downloaded all files with eng prefix.
2) Put them into tessdata directory inside of some folder. Add this folder into System Path variables as TESSDATA_PREFIX.
Result will be
System env var: TESSDATA_PREFIX=D:/Java/OCR
And OCR folder has tessdata with languages files.
This is a screenshot of the directory:
No previous solution worked for me.
I've installed both by apt-get and manually downloading the tessdata, moved around /usr and so on and no one worked even if i exported the variable thousand times.
Finally, on a last try before start to cry i've tried to pass the path directly to the instance of Tesseract().
In Python: tr = Tesseract("/usr/local/share/tesseract-ocr/") and now it works. To clarify, im using tesserwrap module.
For Windows Users:
In Environment Variables, add a new variable in system variable with name "TESSDATA_PREFIX" and value is "C:\Program Files (x86)\Tesseract-OCR\tessdata"
tesseract --tessdata-dir <tessdata-folder> <image-path> stdout --oem 2 -l <lng>
In my case, the mistakes that I've made or attempts that wasn't a success.
I cloned the github repo and copied files from there to
/usr/local/share/tessdata/
/usr/share/tesseract-ocr/tessdata/
/usr/share/tessdata/
Used TESSDATA_PREFIX with above paths
sudo apt-get install tesseract-ocr-eng
First 2 attempts did not worked because, the files from git clone did not worked for the reasons that I do not know. I am not sure why #3 attempt worked for me.
Finally,
I downloaded the eng.traindata file using wget
Copied it to some directory
Used --tessdata-dir with directory name
Take away for me is to learn the tool well & make use of it, rather than relying on package manager installation & directories
For me the problem was in how I downloaded the train data files. Make sure you get the raw link.
Initially I was using:
wget https://github.com/tesseract-ocr/tessdata_best/blob/master/eng.traineddata
When I changed it to:
wget https://github.com/tesseract-ocr/tessdata_best/raw/master/eng.traineddata
It worked
For Ubuntu just run the below command and the Environment variable error will disappear.
command:
export TESSDATA_PREFIX=Path_of_your_tessdata_folder
Command Example:
export TESSDATA_PREFIX=/home/amar/Desktop/OCR/tesseract-4.1.1/tessdata
This command will set the tessdata folder's path to the environment variable with name TESSDATA_PREFIX and the above error will be resolved.
You can call tesseract API function from C code:
#include <tesseract/baseapi.h>
#include <tesseract/ocrclass.h>; // ETEXT_DESC
using namespace tesseract;
class TessAPI : public TessBaseAPI {
public:
void PrintRects(int len);
};
...
TessAPI *api = new TessAPI();
int res = api->Init(NULL, "rus");
api->SetAccuracyVSpeed(AVS_MOST_ACCURATE);
api->SetImage(data, w0, h0, bpp, stride);
api->SetRectangle(x0,y0,w0,h0);
char *text;
ETEXT_DESC monitor;
api->RecognizeForChopTest(&monitor);
text = api->GetUTF8Text();
printf("text: %s\n", text);
printf("m.count: %s\n", monitor.count);
printf("m.progress: %s\n", monitor.progress);
api->RecognizeForChopTest(&monitor);
text = api->GetUTF8Text();
printf("text: %s\n", text);
...
api->End();
And build this code:
g++ -g -I. -I/usr/local/include -o _test test.cpp -ltesseract_api -lfreeimageplus
(i need FreeImage for picture loading)
I'm using windows OS, I tried all solutions above and none of them work.
Finally, I install Tesseract-OCR on D drive(Where I run my python script from) instead of C drive and it works.
So, if you are using windows, run your python script in the same drive as your Tesseract-OCR.
In Google Colab I resolved the issue in this way:
!sudo apt-get install tesseract-ocr-*
Because if you use this command !sudo apt install tesseract-ocr then it imports 2 languages but when you intend to work on non-English languages then the former command works.
Afterwards, use this command !pip install pytesseract
You can also check languages in this way !tesseract --list-langs
I'm using Visual Studio 2017 Community Edition.
I solved this problem by making a directory called tessdata in the Debug directory of my project. Then I put the eng.traineddata file into said directory.
C# developer working on Windows here. What works for me is simply download the file eng.traineddata from the following URL:
https://github.com/tesseract-ocr/tessdata/blob/master/eng.traineddata
and copy it to the following directory in my Console Application project:
[Project Directory]\bin\Debug\tessdata
I did manually create the tessdata folder above.
tessdata_dir_config = r'--tessdata-dir "/usr/local/Cellar/tesseract/4.1.1/share/tessdata"'
pytesseract.image_to_string(imgCrop,lang='eng',config=tessdata_dir_config)
Add this to your code :
instance.setDatapath("C:\\somepath\\tessdata");
instance.setLanguage("eng");
How I solved the problem in my Manjaro Xfce:
Message “TesseractError: (1, 'Error opening data file /home/julio/snap/tesseract/common/eng.traineddata Please make sure the TESSDATA_PREFIX environment variable is set to your "tessdata" directory. Failed loading language 'eng' Tesseract couldn't load any languages! Could not initialize tesseract.')”
Then, in my Manjaro, I typed: sudo pacman -S tesseract
Then the system installed both the “tesseract” and also a package name “leptonica”
After this step, I thought everything was ok, and tried to run my simple script. However, the error message changed to something like this (it changed the previous “/home” location to other “/usr”-like location):
“"Please make sure the TESSDATA_PREFIX environment variable is set to your "tessdata" directory. Failed loading language 'eng' Tesseract couldn't load any languages! Could not initialize tesseract.')"”
Then I realized that there had appeared this message when I installed “tesseract” with pacman: “You must install one of tesseract-data-* packages or whole tesseract-data group”
So, I tried the command: “sudo pacman -S tesseract-data”, and the system presented lots of language options to me. So I’ve chosen some languages, installed as follows, and the module started to work like a charm:
sudo pacman -S tesseract-data-eng
sudo pacman -S tesseract-data-por
sudo pacman -S tesseract-data-fra
sudo pacman -S tesseract-data-spa
I tried some portuguese special characters (like "ão"), that only worked when I used the argument "lang='por'" in the pytesseract.image_to_string(img,lang='por')
As of 2021, My solution for Ubuntu is to download the zip files from https://github.com/tesseract-ocr/tessdata_best/releases/tag/4.1.0, extract and copy the neccessary .traineddata files into /usr/local/share/tessdata. This is the default folder for tesseract 4.1.1 to search for trained data.
I had the same problem with DEU language on macOS. I could solve it by installing all additional languages like so:
brew install tesseract-lang
as suggested on https://formulae.brew.sh/formula/tesseract
**IF you have windows OS then please add your TesseractOCR to system variable.
Eg..
Find the path where Tesseract is installed in your c drive (in my case r"C:\Program Files\Tesseract-OCR\tesseract.exe")**
2)make sure you have the required files ie tessdata, tessdata if not then download it from https://github.com/tesseract-ocr/tessdata https://github.com/tesseract-ocr/langdata (At least those languages which you want to convert)
past it into the main directory in my case C:\Program Files\Tesseract-OCR
4)Add the path of the directory to your system environment variable
for that
search environment variable in start bar
go to environment variable
click path in your system environment variable (NOT IN USER ENVIRONMENT VARIABLE)
past the path of tesseractocr
thats all...