Apache Sqoop : scoop-import giving Undefined error. - mysql

I am working with Apache Hadoop and Apache Sqoop. And I'm trying to import the mysql tables into the hdfs.
Here is the command which i am executing:
sqoop-import --connect jdbc:mysql://localhost/billing_engine -username root -password root > --table cpDetail;
I have setup the Sqoop home environment variable as follows:
export SQOOP_HOME=/Users/bng/Documents/sqoop-1.4.6.bin__hadoop-2.0.4-alpha
export PATH=$PATH:$SQOOP_HOME/bin
But executing the above command, gives me the following error:
readlink: illegal option -- f
usage: readlink [-n] [file ...]
usage: dirname path
/Users/bng/Documents/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/bin/sqoop-import: line 26: /Users/bng/sqoop: Undefined error: 0
Here is the screenshot showing my name node:
Please suggest, where am i wrong?

This is the right command which i needed to use:
sqoop import --connect jdbc:mysql://localhost/billing_engine?useSSL=false --username bil --password bil --table cpdetail -m 1 --target-dir /sqoopFromMysql
The details of the command is as follows;
Sqoop import : the command telling to use sqoop's import command
--connect : indicating the connection to be used
jdbc:mysql://localhost/billing_engine?useSSL=false : connecting to mysql using the jdbc. The host for db is localhost and database name is billing_engine. useSSL=false specifies that we are not connecting over SSL layer.
--username bil --password bil : specifies the username and password for the database.
--table cpdetail : specifies the particular table
-m 1 : specifies the maps to be used
--target-dir /sqoopFromMysql : specifies the target directory in HDFS, where the data will be imported.

Related

Sqoop Import error while loading data from MySQL to HDFS

I am trying to load data from MySQL to HDFS by using below query:
$ sqoop import --connect jdbc:mysql://127.0.0.1:3306/userdb \
--username root --password pwd \
--driver com.mysql.jdbc.Driver --table import-all-tables \
--m 1 --target-dir /user/cloudera/foldername
I get a "communications link failure error".
MYSQL Source: Installed in my local computer
HDFS Destination: Cloudera in virtual machine
Please help me how to load data from my local computer to cloudera HDFS .

Create / import hive table using Sqoop

When I use below import command it allows me to create table and import data from mysql to Hive and I can see the table "widgets" in Hive.
sqoop import --connect jdbc:mysql://localhost:3306/hadoopguide --table widgets --username <username> --password <password> --split-by id -m 1 --hive-import;
But whenever I use below "create-hive-table" command, I get an error.
Command:
sqoop create-hive-table --connect jdbc:mysql://localhost:3306/hadoopguide --table widgets --username <username> --password <password> --fields-terminated-by ',';
Error:
17/03/14 21:30:21 INFO hive.HiveImport: FAILED: Execution Error,
return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient 17/03/14 21:30:21
ERROR tool.CreateHiveTableTool: Encountered IOException running create
table job: java.io.IOException: Hive exited with status 1 at
org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385)
at
org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239)
at
org.apache.sqoop.tool.CreateHiveTableTool.run(CreateHiveTableTool.java:58)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145) at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at
org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) at
org.apache.sqoop.Sqoop.main(Sqoop.java:238)
Can anyone please assist me why I am getting this error.
Any input from your side will be great :-) .
Is you Hadoop Services running.
$ start-all.sh
hive> show database;

Sqoop command for import table

I'm using dse 5.0.3 with have sqoop version 1.4.5.15.1 while im importing my data from mysql to cql this error i get
./dse sqoop cql-import --table npa_nxx --connect jdbc:mysql://localhost/npa_nxx_demo --username root --password 123
ERROR 13:20:53,886 Imported Failed: Parameter 'directory' is not a directory.
Please help me to solve it.!!
It's difficult to be exact about the problem but I'd suggest that you aren't including any cassandra parameters to the cql-import command is attempting to import to an hdfs directory that also isn't declared. Try including a cassandra-keyspace and cassandra-table on the command. Like:
./dse sqoop cql-import --table npa_nxx --connect jdbc:mysql://localhost/npa_nxx_demo --username root --password 123 --cassandra-keyspace npa_nxx --cassandra-table npa_nxx_data
This assumes that the cassandra keyspace and table are correctly setup.
Since this import looks like it is from the dse sqoop demo I'd suggest following the README.txt more closely as it has the correct options for this import in it.

sqoop export with procedure option is failing

I am trying to export the data from emr to postgre using sqoop. when I run below command, it is throwing the following exception
sqoop export --connect jdbc:postgresql://test.xxxxx.us-east-1.rds.amazonaws.com/RulesPOC --username user --password password --call merge_AccountRepo -m 1 --input-fields-terminated-by '|' --export-dir s3://<bucket_name>/output/ --driver org.postgresql.Driver --columns "num,max"
Error:
16/07/14 21:25:07 INFO tool.CodeGenTool: Beginning code generation
16/07/14 21:25:07 ERROR tool.ExportTool: Encountered IOException running export job: java.io.IOException: No columns to generate for ClassWriter
[
Can someone please suggest what is wrong?

Sqoop: Could not load mysql driver exception

I Installed Sqoop in my local machine. Following are the config information.
Bash.bashrc:
export HADOOP_HOME=/home/hduser/hadoop
export HBASE_HOME=/home/hduser/hbase
export HIVE_HOME=/home/hduser/hive
export HCAT_HOME=/home/hduser/hive/hcatalog
export SQOOP_HOME=/home/hduser/sqoop
export PATH=$PATH:$HIVE_HOME/bin
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HBASE_HOME/bin
export PATH=$PATH:$SQOOP_HOME/bin
export PATH=$PATH:$HCAT_HOME/bin
Hadoop:
Version: Hadoop 1.0.3
Hive:
Version: hive 0.11.0
Mysql Connector driver
version: mysql-connector-java-5.1.29
"The driver is copied to the lib folder of sqoop"
Sqoop :
version: sqoop 1.4.4
After making all the installation I create a table in mysql named practice_1, But when I run the load command to load data from mysql to hdfs the command throws an exception:
ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
Coud anyone please guide me what can be the possible problem.
You need database driver in 'SQOOP' classpath check this
It has wonderful explanation about the 'SQOOP'
SQOOP has other options like
Ex: --driver com.microsoft.jdbc.sqlserver.SQLServerDriver -libjars=".*jar"
from here
You can use Sqoop with any other JDBC-compliant database. First, download the appropriate JDBC driver for the type of database you want to import, and install the .jar file in the $SQOOP_HOME/lib directory on your client machine. (This will be /usr/lib/sqoop/lib if you installed from an RPM or Debian package.) Each driver .jar file also has a specific driver class which defines the entry-point to the driver. For example, MySQL's Connector/J library has a driver class of com.mysql.jdbc.Driver. Refer to your database vendor-specific documentation to determine the main driver class. This class must be provided as an argument to Sqoop with --driver.
You may be interested in understanding the difference between connector and driver here is the article
Another solution which avoids using a shared library is adding the driver jar to the classpath of sqoop by using HADOOP_CLASSPATH. I haven't got the -libjars option to work. This solution works also on a secure cluster using kerberos.
HADOOP_CLASSPATH=/use.case/lib/postgresql-9.2-1003-jdbc4.jar
sqoop export --connect jdbc:postgresql://db:5432/user \
--driver org.postgresql.Driver \
--connection-manager org.apache.sqoop.manager.GenericJdbcManager \
--username user \
-P \
--export-dir /user/hive/warehouse/db1/table1 \
--table table2
This one works at least with sqoop 1.4.3-cdh4.4.0
You need to add the MySql connector to /usr/lib/sqoop/lib.
MySQL JDBC Driver by default is not present in Sqoop distribution in order to ensure that the default distribution is fully Apache license compliant.
Hope this helps...!!!
copy the 'mysql-connector-java-5.1.41-bin.jar' into sqoop/lib folder and execute sqoop import statements
If you have copied mysql driver to the sqoop lib folder. It will work for sure. Make sure you sqoop command is correct
/home/hduser/sqoop/bin/sqoop import --connect jdbc:mysql://localhost:3306/test --username root --password root -–table practice_1 -m 1
It's a Oozie ShareLib problem. The script below works for my:
At Shell
sudo -u hdfs hadoop fs -chown cloudera:cloudera /user/oozie/share/lib/lib_20170719053712/sqoop
hdfs dfs -put /var/lib/sqoop/mysql-connector-java.jar /user/oozie/share/lib/lib_20170719053712/sqoop
sudo -u hdfs hadoop fs -chown oozie:oozie /user/oozie/share/lib/lib_20170719053712/sqoop
oozie admin -oozie http://localhost:11000/oozie -sharelibupdate
oozie admin -oozie http://localhost:11000/oozie -shareliblist sqoop
At Hue Sqoop Client
sqoop list-tables --connect jdbc:mysql://localhost/retail_db --username root --password cloudera
More detail at:
https://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
You need to grant priveleges to the tables as below:
grant all privileges on marksheet.* to 'root'#'192.168.168.1'
identified by 'root123';
flush privileges;
Here is sample command that I have successfully executed:
sqoop import --verbose --fields-terminated-by ',' --connect
jdbc:mysql://192.168.168.1/test --username root --password root123
--table student --hive-import --create-hive-table --hive-home /home/training/hive --warehouse-dir /user/hive/warehouse
--fields-terminated-by ',' --hive-table studentmysql