I'm learning data analysis in Zeppelin, I'm a mechanical engineer so this is outside my expertise.
I am trying to download two csv files using a file that contains the urls, test2.txt. When I run it I get no output, but no error message either. I've included a link to a screenshot showing my code and the results.
When I go into Ambari Sandbox I cannot find any files created. I'm assuming the directory the file is in is where the csv files will be downloaded too. I've tried using -P as well with no luck. I've checked in man wget but it did not help.
So I have several questions:
How do I show the output from running wget?
Where is the default directory that wget stores files?
Do I need additional data in the file other than just the URLs?
Screenshot: Code and Output for %sh
Thanks for any and all help.
%sh
wget -i /tmp/test2.txt
%sh
# list the current working directory
pwd # output: home/zeppelin
# make a new folder, created in "tmp" because it is temporary
mkdir -p /home/zeppelin/tmp/Folder_Name
# change directory to new folder
cd /home/zeppelin/tmp/Folder_Name
# transfer the file from the sandbox to the current working directory
hadoop fs -get /tmp/test2.txt /home/zeppelin/tmp/Folder_Name/
# download the URL
wget -i test2.txt
I have a firebreath plugin with installer.cmake script for Mac. Instead of creating a dmg file it creates a package based on pmdoc folder.
COMMAND ${CMD_CP} -r ${CMAKE_CURRENT_SOURCE_DIR}/Mac/MyPlugin.pmdoc ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}/MyPlugin.pmdoc
COMMAND /Applications/PackageMaker.app/Contents/MacOS/PackageMaker --doc ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}/MyPlugin.pmdoc --version ${FBSTRING_PLUGIN_VERSION} --out ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}/MyPlugin.pkg
Problem is I want to use FB variables in one of the pmdoc file, for example set title to ${FBSTRING_PluginName} ${FBSTRING_PLUGIN_VERSION} Obviously, copy command just copies the file, but how can I replace variables with their values?
Use cmake's configure_file. This will take an input file and an output file; the output file will have all variables replaced. Lots of examples of this in the firebreath codebase.
I have a Perl script that I wrote to package release scripts.
The RTC bits in the script are as follows.
List the workspaces:
lscm list workspaces -r "$reposURI" -u $reposUser -P $reposPwd
List the componets:
lscm compare ws "$ws1" ws "$ws2" -r "$reposURI" -u $reposUser -P $reposPwd -I c
Compare the 2 workspaces' specified component to the changed files:
lscm compare ws "$ws1" ws "$ws2" -r "$reposURI" -u $reposUser -P $reposPwd -I cf
Great! I have the liust of files changed (trust me, this took a LOT of working out). Now, next step is simply extract the files listed from the changed workspace:
According to the documentation there is a "Lscm extract", but it seems not on the version I have. I cannot upgrade as this is a corporate environment where software installs are controlled centrally, and they are sticking with the current RTC version (3).
So, is there an alternative way?
I don't know if a lscm extract: it doesn't seem to exist in the RTC documentation.
The help page only mention a lscm changeset extract (used in RTC3.x).
lscm extract is only referenced one, in the article "Using the Jazz SCM command line to support software configuration audit", and I would say it is an error.
You can load only the file you care about: scm load <workspace> <path-in-workspace>. That will get the version onto the disk, but it will pollute your disk with RTC metadata (ie, the .jazz5 dir in the root of your sandbox). I suggest running in a temporary directory and then deleting that directory once you have the file content that you want.
That's kind of kludgy. Ideally you'd be able to move onto a modern version of RTC and use the 'extract' subcommand that you mention.
I have a problem with running tesseract-ocr engine on linux. I've downloaded RUS language data and put it to tessdata directory (/usr/local/share/tessdata). When I'm trying to run tesseract with command tesseract blob.jpg out -l rus , it displays an error:
Error opening data file /usr/local/share/tessdata/eng.traineddata
Please make sure the TESSDATA_PREFIX environment variable is set to the parent directory of your "tessdata" directory.
Failed loading language eng
Tesseract couldn't load any languages!
Could not initialize tesseract.
According to compiling guide, I used export TESSDATA_PREFIX='/usr/local/share/'
to point my tessdata directory.
Maybe I should edit any config files? Tesseract try to load 'eng' data files instead of 'rus'.
Screenshot:
http://i.stack.imgur.com/I0Guc.png
You can grab eng.traineddata Github:
wget https://github.com/tesseract-ocr/tessdata/raw/main/eng.traineddata
Check https://github.com/tesseract-ocr/tessdata for a full list of trained language data.
When you grab the file(s), move them to the /usr/local/share/tessdata folder. Warning: some Linux distributions (such as openSUSE and Ubuntu) may be expecting it in /usr/share/tessdata instead.
# If you got the data from Google, unzip it first!
gunzip eng.traineddata.gz
# Move the data
sudo mv -v eng.traineddata /usr/local/share/tessdata/
The simpliest way is to install the needed package:
sudo apt-get install tesseract-ocr-eng #for english
sudo apt-get install tesseract-ocr-tam #for tamil
sudo apt-get install tesseract-ocr-deu #for deutsch (German)
As you can notice, it opens the road to others languages (i.e. tesseract-ocr-fra).
I had this error too on the Windows machine.
My solution.
1) Download your language files from
https://github.com/tesseract-ocr/tessdata/tree/3.04.00
For example, for eng, I downloaded all files with eng prefix.
2) Put them into tessdata directory inside of some folder. Add this folder into System Path variables as TESSDATA_PREFIX.
Result will be
System env var: TESSDATA_PREFIX=D:/Java/OCR
And OCR folder has tessdata with languages files.
This is a screenshot of the directory:
No previous solution worked for me.
I've installed both by apt-get and manually downloading the tessdata, moved around /usr and so on and no one worked even if i exported the variable thousand times.
Finally, on a last try before start to cry i've tried to pass the path directly to the instance of Tesseract().
In Python: tr = Tesseract("/usr/local/share/tesseract-ocr/") and now it works. To clarify, im using tesserwrap module.
For Windows Users:
In Environment Variables, add a new variable in system variable with name "TESSDATA_PREFIX" and value is "C:\Program Files (x86)\Tesseract-OCR\tessdata"
tesseract --tessdata-dir <tessdata-folder> <image-path> stdout --oem 2 -l <lng>
In my case, the mistakes that I've made or attempts that wasn't a success.
I cloned the github repo and copied files from there to
/usr/local/share/tessdata/
/usr/share/tesseract-ocr/tessdata/
/usr/share/tessdata/
Used TESSDATA_PREFIX with above paths
sudo apt-get install tesseract-ocr-eng
First 2 attempts did not worked because, the files from git clone did not worked for the reasons that I do not know. I am not sure why #3 attempt worked for me.
Finally,
I downloaded the eng.traindata file using wget
Copied it to some directory
Used --tessdata-dir with directory name
Take away for me is to learn the tool well & make use of it, rather than relying on package manager installation & directories
For me the problem was in how I downloaded the train data files. Make sure you get the raw link.
Initially I was using:
wget https://github.com/tesseract-ocr/tessdata_best/blob/master/eng.traineddata
When I changed it to:
wget https://github.com/tesseract-ocr/tessdata_best/raw/master/eng.traineddata
It worked
For Ubuntu just run the below command and the Environment variable error will disappear.
command:
export TESSDATA_PREFIX=Path_of_your_tessdata_folder
Command Example:
export TESSDATA_PREFIX=/home/amar/Desktop/OCR/tesseract-4.1.1/tessdata
This command will set the tessdata folder's path to the environment variable with name TESSDATA_PREFIX and the above error will be resolved.
You can call tesseract API function from C code:
#include <tesseract/baseapi.h>
#include <tesseract/ocrclass.h>; // ETEXT_DESC
using namespace tesseract;
class TessAPI : public TessBaseAPI {
public:
void PrintRects(int len);
};
...
TessAPI *api = new TessAPI();
int res = api->Init(NULL, "rus");
api->SetAccuracyVSpeed(AVS_MOST_ACCURATE);
api->SetImage(data, w0, h0, bpp, stride);
api->SetRectangle(x0,y0,w0,h0);
char *text;
ETEXT_DESC monitor;
api->RecognizeForChopTest(&monitor);
text = api->GetUTF8Text();
printf("text: %s\n", text);
printf("m.count: %s\n", monitor.count);
printf("m.progress: %s\n", monitor.progress);
api->RecognizeForChopTest(&monitor);
text = api->GetUTF8Text();
printf("text: %s\n", text);
...
api->End();
And build this code:
g++ -g -I. -I/usr/local/include -o _test test.cpp -ltesseract_api -lfreeimageplus
(i need FreeImage for picture loading)
I'm using windows OS, I tried all solutions above and none of them work.
Finally, I install Tesseract-OCR on D drive(Where I run my python script from) instead of C drive and it works.
So, if you are using windows, run your python script in the same drive as your Tesseract-OCR.
In Google Colab I resolved the issue in this way:
!sudo apt-get install tesseract-ocr-*
Because if you use this command !sudo apt install tesseract-ocr then it imports 2 languages but when you intend to work on non-English languages then the former command works.
Afterwards, use this command !pip install pytesseract
You can also check languages in this way !tesseract --list-langs
I'm using Visual Studio 2017 Community Edition.
I solved this problem by making a directory called tessdata in the Debug directory of my project. Then I put the eng.traineddata file into said directory.
C# developer working on Windows here. What works for me is simply download the file eng.traineddata from the following URL:
https://github.com/tesseract-ocr/tessdata/blob/master/eng.traineddata
and copy it to the following directory in my Console Application project:
[Project Directory]\bin\Debug\tessdata
I did manually create the tessdata folder above.
tessdata_dir_config = r'--tessdata-dir "/usr/local/Cellar/tesseract/4.1.1/share/tessdata"'
pytesseract.image_to_string(imgCrop,lang='eng',config=tessdata_dir_config)
Add this to your code :
instance.setDatapath("C:\\somepath\\tessdata");
instance.setLanguage("eng");
How I solved the problem in my Manjaro Xfce:
Message “TesseractError: (1, 'Error opening data file /home/julio/snap/tesseract/common/eng.traineddata Please make sure the TESSDATA_PREFIX environment variable is set to your "tessdata" directory. Failed loading language 'eng' Tesseract couldn't load any languages! Could not initialize tesseract.')”
Then, in my Manjaro, I typed: sudo pacman -S tesseract
Then the system installed both the “tesseract” and also a package name “leptonica”
After this step, I thought everything was ok, and tried to run my simple script. However, the error message changed to something like this (it changed the previous “/home” location to other “/usr”-like location):
“"Please make sure the TESSDATA_PREFIX environment variable is set to your "tessdata" directory. Failed loading language 'eng' Tesseract couldn't load any languages! Could not initialize tesseract.')"”
Then I realized that there had appeared this message when I installed “tesseract” with pacman: “You must install one of tesseract-data-* packages or whole tesseract-data group”
So, I tried the command: “sudo pacman -S tesseract-data”, and the system presented lots of language options to me. So I’ve chosen some languages, installed as follows, and the module started to work like a charm:
sudo pacman -S tesseract-data-eng
sudo pacman -S tesseract-data-por
sudo pacman -S tesseract-data-fra
sudo pacman -S tesseract-data-spa
I tried some portuguese special characters (like "ão"), that only worked when I used the argument "lang='por'" in the pytesseract.image_to_string(img,lang='por')
As of 2021, My solution for Ubuntu is to download the zip files from https://github.com/tesseract-ocr/tessdata_best/releases/tag/4.1.0, extract and copy the neccessary .traineddata files into /usr/local/share/tessdata. This is the default folder for tesseract 4.1.1 to search for trained data.
I had the same problem with DEU language on macOS. I could solve it by installing all additional languages like so:
brew install tesseract-lang
as suggested on https://formulae.brew.sh/formula/tesseract
**IF you have windows OS then please add your TesseractOCR to system variable.
Eg..
Find the path where Tesseract is installed in your c drive (in my case r"C:\Program Files\Tesseract-OCR\tesseract.exe")**
2)make sure you have the required files ie tessdata, tessdata if not then download it from https://github.com/tesseract-ocr/tessdata https://github.com/tesseract-ocr/langdata (At least those languages which you want to convert)
past it into the main directory in my case C:\Program Files\Tesseract-OCR
4)Add the path of the directory to your system environment variable
for that
search environment variable in start bar
go to environment variable
click path in your system environment variable (NOT IN USER ENVIRONMENT VARIABLE)
past the path of tesseractocr
thats all...
I found this question which had an answer to the question of performing batch conversions with Pandoc, but it doesn't answer the question of how to make it recursive. I stipulate up front that I'm not a programmer, so I'm seeking some help on this here.
The Pandoc documentation is slim on details regarding passing batches of files to the executable, and based on the script it looks like Pandoc itself is not capable of parsing more than a single file at a time. The script below works just fine in Mac OS X, but only processes the files in the local directory and outputs the results in the same place.
find . -name \*.md -type f -exec pandoc -o {}.txt {} \;
I used the following code to get something of the result I was hoping for:
find . -name \*.html -type f -exec pandoc -o {}.markdown {} \;
This simple script, run using Pandoc installed on Mac OS X 10.7.4 converts all matching files in the directory I run it in to markdown and saves them in the same directory. For example, if I had a file named apps.html, it would convert that file to apps.html.markdown in the same directory as the source files.
While I'm pleased that it makes the conversion, and it's fast, I need it to process all files located in one directory and put the markdown versions in a set of mirrored directories for editing. Ultimately, these directories are in Github repositories. One branch is for editing while another branch is for production/publishing. In addition, this simple script is retaining the original extension and appending the new extension to it. If I convert back again, it will add the HTML extension after the markdown extension, and the file size would just grow and grow.
Technically, all I need to do is be able to parse one branches directory and sync it with the production one, then when all changed, removed, and new content is verified correct, I can run commits to publish the changes. It looks like the Find command can handle all of this, but I just have no clue as to how to properly configure it, even after reading the Mac OS X and Ubuntu man pages.
Any kind words of wisdom would be deeply appreciated.
TC
Create the following Makefile:
TXTDIR=sources
HTMLS=$(wildcard *.html)
MDS=$(patsubst %.html,$(TXTDIR)/%.markdown, $(HTMLS))
.PHONY : all
all : $(MDS)
$(TXTDIR) :
mkdir $(TXTDIR)
$(TXTDIR)/%.markdown : %.html $(TXTDIR)
pandoc -f html -t markdown -s $< -o $#
(Note: The indented lines must begin with a TAB -- this may not come through in the above, since markdown usually strips out tabs.)
Then you just need to type 'make', and it will run pandoc on every file with a .html extension in the working directory, producing a markdown version in 'sources'. An advantage of this method over using 'find' is that it will only run pandoc on a file that has changed since it was last run.
Just for the record: here is how I achieved the conversion of a bunch of HTML files to their Markdown equivalents:
for file in $(ls *.html); do pandoc -f html -t markdown "${file}" -o "${file%html}md"; done
When you have a look at the script code from the -o argument, you'll see it uses string manipulation to remove the existing html with the md file ending.