I want to upload bulk csv file using phonix but I can not understood below command. Can you explain me in details ?
HADOOP_CLASSPATH=$(hbase mapredcp):/path/to/hbase/conf hadoop jar phoenix-<version>-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool --table EXAMPLE --input /data/example.csv
I take this command from following website.
https://phoenix.apache.org/bulk_dataload.html
I am not sure if you are still looking for an answer. But here it is. You are first setting the HADOOP_CLASSPATH and then call the executable "hadoop" with jar options to look for phoenix client jar and the class to run with parameters.
The following can help you to understand hadoop command usage (try typing hadoop on your ssh shell)
Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
CLASSNAME run the class named CLASSNAME
or
where COMMAND is one of:
fs run a generic filesystem user client
version print the version
jar <jar> run a jar file
note: please use "yarn jar" to launch
YARN applications, not this command.
checknative [-a|-h] check native hadoop and compression libraries availability
distcp <srcurl> <desturl> copy file or directories recursively
envvars display computed Hadoop environment variables
archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
classpath prints the class path needed to get the
credential interact with credential providers
Hadoop jar and the required libraries
daemonlog get/set the log level for each daemon
trace view and modify Hadoop tracing settings
Most commands print help when invoked w/o parameters.
Related
I have installed the ZABBIX from the sources, I need to be recompiling to add the --enable-java option.
Then I executed ./configure and make,but I don't known which executable file should be replaced in the installation directory，
I search the zabbix_server file in the unzip directory,
[root#zbx-flm zabbix-3.2.1]# find . -name zabbix_server -type f
./misc/init.d/freebsd/zabbix_server
./misc/init.d/fedora/core/zabbix_server
./misc/init.d/fedora/core5/zabbix_server
./misc/init.d/tru64/zabbix_server
./misc/init.d/suse/9.1/zabbix_server
./misc/init.d/suse/9.3/zabbix_server
./misc/init.d/suse/9.2/zabbix_server
These are just the startup files, not the binary files。So what should I do?
If these are unmodified sources and the compilation was successful, the server binary will be in the src/zabbix_server subdirectory.
Please note that the --enable-java flag does not modify the Zabbix server binary. Instead, Java gateway is built in src/zabbix_java. That is not a single binary anymore, though.
I am bit new to hadoop. I have recently install a stable version of Apache Hadoop 2.7.2 on ubuntu 14.04 OS.
I am trying to execute some basic Hadoop command such as follow
hadoop version
The command gives me the correct output as follows that is correct.
However, when I try to execute the hadoop fs -ls, it give me error.
I have searched the previous question related to this problem on stackoverflow such as StackoverflowQuestion. But, I am not finding /user directory in my hadoop installation. Could you please help me how can I resolve this issue?
The content of my .bashrc file is as follows:
The content of hdfs-site.xml file is as follows:
First of all the command "hadoop fs -ls" is a command to the HDFS file system and not a Linux command.
Second, the command as you typed it is incomplete. The correct syntax is "hadoop fs -ls [-d] [-h] [-R] " where the [-d], [-h] and [-R] components of the command are optional. That said, you MUST specify a path for the "args" component. The "args" component of the command expects an HDFS path (e.g. substituting / for "args" will list the entire tree ** ON HDFS ** starting at the HDFS root directory /). You will need to create a directory called "user" on HDFS under the root directory using "hadoop fs -mkdir /user". The the command "hadoop fs -ls /user" will work and will show an empty user directory.
Third, there is no way to tell HDFS to use a value for the "args" by supplying it a value of a local filesystem (Linux) path ...which is what you are attempting or understanding it to be. Any value for "args" must resolve to an HDFS filesystem path and not a Linux filesystem path.
Fourth, for newcomers to Hadoop, it is very important to have a clear distinction between the native host operating system filesystem (in this case Linux filesystem) and the Hadoop filesystem (in this case HDFS).
one thing to note when doing a hadoop commands in v2.7.2 is that, hadoop works on top of Linux OS, hence when we want to access into Hadoop Distributed File System, we would use something like this command; hdfs dfs -ls / instead of hadoop fs -ls.
Also, in your hdfs-site.xml configurations, you seemed to miss out adding this properties.
<property>
<name>dfs.datanode.dir</name>
<value>file://path/to/datanode</value>
</property>
Please take note of your $HADOOP_HOME as well.
Steps To Replicate
On Windows 8.
In shell (with SSH connection active):
rhc snapshot save [appname]
Error
No system SSH available. Please use the --ssh option to specify the path to your SSH executable, or install SSH.
Suggested Solution
From this post:
Usage: rhc snapshot-save <application> [--filepath FILE] [--ssh path_to_ssh_executable]
Pass '--help' to see the full list of options
Question
The path to keys on PC is:
C:\Users\[name]\.ssh
How do I define this in the rhc snaphot command?
Solution
rhc snapshot save [appname] --filepath FILE --ssh "C:\Users\[name]\.ssh"
This will show the message:
Pulling down a snapshot of application '[appname]' to FILE ...
... then after a while
Pulling down a snapshot of application '[appname]' to FILE ... DONE
Update
That saved the backup in a file called "FILE" without an extension, so I'm guessing in the future I should define the filename as something like "my_app_backup.tar.gz" ie:
rhc snapshot save [appname] --filepath "my_app_backup.tar.gz" --ssh "C:\Users\[name]\.ssh"
It will save in the repo directory, so make sure you move it out of this directory before you git add, commit, push etc, otherwise you will upload your backup too.
I recently installed hadoop on my local ubuntu. I have started data-node by invoking bin/start-all.sh script. However when I try to run the word count program
bin/hadoop jar hadoop-examples-1.2.1.jar wordcount /home/USER/Desktop/books /home/USER/Desktop/books-output
I always get a connect exception. The folder 'books' is on my deskop(local filesystem). Any suggestions on how to overcome this?
I have followed every steps in this tutorial. I am not sure how to get rid of that error. All help will be appreciated.
copy your books file into your hdfs
and for the input path argument use hdfs path of your copied book file.
for more detail go through below link.
http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_1_--_Running_WordCount#Basic_Hadoop_Admin_Commands
There is a bit of confusion here, when you run the hadoop ... command then the default filesystem which it uses is the hadoop distributed filesystem hence the files must be located on the hdfs for hadoop to access it.
To copy files from the local filesystem to the hadoop filesystem you have to use the following command
hdfs dfs -copyFromLocal /path/in/local/file/system /destination/on/hdfs
One more thing if you want to run the program from your IDE directly then sometimes you get this issue which can be solved by adding the
core-site.xml and hdfs-site.xml files in the conf variable something like
conf.addResource(new Path("/usr/local/hadoop/etc/hadoop/core-site.xml"));
conf.addResource(new Path("/usr/local/hadoop/etc/hadoop/hdfs-site.xml"));
change the path above to the hdfs-site.xml and core-site.xml to your local path.
So the above arguments can also be provided from the command line by adding them to the classPath with -cp tag.
I tried to do this in Hudson but:
I have a script in Perl on my server (windows): lets say: d\util\demo.pl I want to have it running in Hudson. so I go to Hudson, create new job, go to: Build Execute Windows batch command and add: perl.exe d\util\demo.pl
I got this error: 'perl.exe' is not recognized as an internal or external command, operable program or batch file.
please help!
It can't find the perl.exe in the path of the agent that is running the task. Verify that perl is properly installed AND that the path where perl.exe was in stalled to is in you system path on EVERY agent that will run this task.
Can you run that command from any folder of the server?
If yes, than the Hudson server runs definitely under a different user account. Make sure that the user account Hudson is running under has all necessary environment variables set.
If not, than add the full qualified path to the perl.exe (e.g. C:\program files\perl\bin\perl.exe d:\util\demo.pl). If this doesn't help, you have to also set all environment variables (see "if yes").