I created a SQL table and I am trying to import it data by using Sqoop to hdfs. But when I run sqoop import command, I am getting the following error.
This is the command:
hduser#ubuntu:~$sqoop import --connect jdbc:mysql://192.168.94.255/Cause_of_Death --table Death2014 --username root -P --target-dir /sqoop_out -m 1
Result:
Warning: /home/hduser/sqoop-1.4.7.bin__hadoop-2.6.0/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hduser/sqoop-1.4.7.bin__hadoop-2.6.0/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hduser/sqoop-1.4.7.bin__hadoop-2.6.0/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hduser/sqoop-1.4.7.bin__hadoop-2.6.0/bin/../../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
19/12/20 02:54:21 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Enter password:
19/12/20 02:54:27 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
19/12/20 02:54:27 INFO tool.CodeGenTool: Beginning code generation
19/12/20 02:54:27 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:875)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:59)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1872)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1671)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Related
I am receiving an error when connecting mysql and I do not know the steps to connect the driver. I am using AWS EMR as a student. My intention is to run some pig commands after I connect.
[hadoop#ip-172-31-88-249 ~]$ sqoop import --connect jdbc:mysql://retail-db.cjmn8pjxazxq.us-east-1.rds.amazonaws.com/retail --table customers --username admin --password ***** --target-dir SQP/input/sqoop1 -m 1
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
22/10/26 02:30:38 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
22/10/26 02:30:39 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
22/10/26 02:30:39 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
22/10/26 02:30:39 INFO tool.CodeGenTool: Beginning code generation
22/10/26 02:30:39 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:875)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:59)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1872)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1671)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
I was not able to find a step by step process to add the driver for AWS EMR
EMR is just a handful of Linux servers that you have SSH access to.
Sqoop is just a JVM application.
You add the JDBC driver to its classpath (i.e. it's lib directory)
Am trying to run the below command in cloudera and getting link failure error. I have tried to restart mysqld service too, no use. Kindly some one help friends.
Code and error:
[cloudera#quickstart ~]$ sqoop list-databases --connect "jdbc:mysql://quickstart.cloudera:3306" --username=retail_dba --password=cloudera
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/09/22 09:45:59 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.10.0
17/09/22 09:45:59 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/09/22 09:45:59 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/09/22 09:46:16 ERROR manager.CatalogQueryManager: Failed to list databases
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Download mysql-connector-java-5.1.21.jar and copy it into the sqoop lib folder then try to run the sqoop import as follows:
sqoop list-databases \
--connect "jdbc:mysql://localhost:3306" \
--username=retail_dba \
--password=cloudera
Is there anything wrong with the below command. Its not working for me.
sqoop import-all-tables
--connect jdbc:mysql://localhost/retail_db --username=retail_dba
-- compression-codec=snappy
--as-parquetfile --hive-import -m 1
16/08/17 08:34:07 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
16/08/17 08:34:07 INFO tool.BaseSqoopTool:
Using Hive-specific delimiters for output. You can override
16/08/17 08:34:07 INFO tool.BaseSqoopTool:
delimiters with --fields- terminated-by, etc.
16/08/17 08:34:08 INFO manager.MySQLManager:
Preparing to use a MySQL streaming resultset.
16/08/17 08:34:08 INFO tool.CodeGenTool: Beginning code generation
16/08/17 08:34:08 ERROR sqoop.Sqoop:
Got exception running Sqoop:
java.lang.NullPointerException
java.lang.NullPointerException
Can you post the complete error you are getting?
Does you error contain - Class not found exception? If yes, go to $HADOOP_HOME/etc/hadoop find mapred-site.xml and change/add the value of property mapreduce.framework.name as yarn. Save the file and restart hadoop
I am getting the following error while trying to list databases from mysql database using sqoop. I am using Cloudera VM CDH4 were it does not come with MySql pre installed by default. I installed MySql as per cloudera tutorial. Now I am trying to list database from MySQl and it fails. Is there any jdbc connectivity issue?
[cloudera#localhost ~]$ sqoop list-databases --connect "jdbc:mysql://localhost.localdomain" --user root --password aaaaaaaa
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
14/12/17 11:52:07 INFO sqoop.Sqoop: Running Sqoop version: 1.4.3-cdh4.7.0
14/12/17 11:52:07 ERROR tool.BaseSqoopTool: Error parsing arguments for list-databases:
14/12/17 11:52:07 ERROR tool.BaseSqoopTool: Unrecognized argument: --user
14/12/17 11:52:07 ERROR tool.BaseSqoopTool: Unrecognized argument: root
14/12/17 11:52:07 ERROR tool.BaseSqoopTool: Unrecognized argument: --password
14/12/17 11:52:07 ERROR tool.BaseSqoopTool: Unrecognized argument: aaaaaaaa
Try --help for usage instructions.
usage: sqoop list-databases [GENERIC-ARGS] [TOOL-ARGS]
Common arguments:
--connect <jdbc-uri> Specify JDBC connect
string
--connection-manager <class-name> Specify connection manager
class name
--connection-param-file <properties-file> Specify connection
parameters file
--driver <class-name> Manually specify JDBC
driver class to use
--hadoop-home <hdir> Override
$HADOOP_MAPRED_HOME_ARG
--hadoop-mapred-home <dir> Override
$HADOOP_MAPRED_HOME_ARG
--help Print usage instructions
-P Read password from console
--password <password> Set authentication
password
--password-file <password-file> Set authentication
password file path
--skip-dist-cache Skip copying jars to
distributed cache
--username <username> Set authentication
username
--verbose Print more information
while working
Generic Hadoop command-line arguments:
(must preceed any tool-specific arguments)
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Try this instead:
sqoop list-databases --connect jdbc:mysql://localhost \
--username root \
--password aaaaaaaa
Problem was there is not options --user for Sqoop instead you have to use --username.
I have downloaded the connector and saved the jar file under the sqoop/lib folder, even then I'm facing this issue.
at#ubuntu:~$ sqoop list-databases --connect "jdbc:mysql://localhost"
--username root --password root Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of
your HCatalog installation. 14/01/08 16:20:45 WARN tool.BaseSqoopTool:
Setting your password on the command-line is insecure. Consider using
-P instead. 14/01/08 16:20:45 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 14/01/08 16:20:45 ERROR sqoop.Sqoop:
Got exception running Sqoop: java.lang.RuntimeException: Could not
load db driver class: com.mysql.jdbc.Driver
java.lang.RuntimeException: Could not load db driver class:
com.mysql.jdbc.Driver at
org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:772)
at
org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
at
org.apache.sqoop.manager.CatalogQueryManager.listDatabases(CatalogQueryManager.java:57)
at
org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145) at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at
org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) at
org.apache.sqoop.Sqoop.main(Sqoop.java:238)
The Reason For this error is sqoop is not able to connect to specified mysql driver
Try to Place mysql jars in the below libs
/opt/cloudera/parcels/CDH/lib/sqoop/lib/
/opt/cloudera/parcels/CDH-1.5.0-1../lib/sqoop/lib/
/var/lib/sqoop/
and then Restart the cluster once and run the sqoop command, then it will works.
Thanks.
Download mysql connector and place in "locationOfSqoop/sqoop/lib"
instead of placing connector folder (mysql-connector-java-5.1.41) ,place (mysql-connector-java-5.1.36-bin.jar) only jar file
"locationOfSqoop/sqoop/lib/mysql-connector-java-5.1.36-bin.jar"
First download mysql driver, you can download it from
https://dev.mysql.com/downloads/connector/j/5.1.html
then place the driver in the following location where sqoop runs (ie) in the node where sqoop runs
/var/lib/sqoop
if you didn't find the dir make one and then
place
mysql-connector-java-version-bin.jar
in
/var/lib/sqoop
than restart and complete the sqoop task
In Sqoop-1.7 mysql-connector is not present. So download the jar and copy to sqoop lib.
Download MySql connector from following link:
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
Extract mysql-connector-java-5.1.46.tar.gz
Copy mysql-connector-java-5.1.46.jar to $SQOOP_HOME/lib directory
[ If you are using Sqoop with hadoop ]: Copy mysql-connector-java-5.1.46.jar to lib directory which is present in hdfs along with other jars present in the $SQOOP_HOME/lib/