I am new to Hadoop world. Just started learning the new things about hadoop.
I am getting below error while importing data from mysql to hdfs using sqoop:
sqoop:000> sqoop import --connect jdbc:mysql://localhost/books --username root --password thanks --table authors --m 1;
Exception has occurred during processing command
Exception: org.codehaus.groovy.control.MultipleCompilationErrorsException Message: startup failed:
groovysh_parse: 1: expecting EOF, found 'import' # line 1, column 7.
sqoop import --connect jdbc:mysql://localhost/books --username root --password thanks --table authors --m 1;
^
1 error
Could you help me in fixing this error?
it seems that you are using sqoop2
you need to follow these steps!
1st step
check if you have installed sqoop correctly
sqoop:000> show version --all
you should get a response something like this
Server version: Sqoop 2.0.0-SNAPSHOT revision Unknown Compiled by
jarcec on Wed Nov 21 16:15:51 PST 2012 Client version: Sqoop
2.0.0-SNAPSHOT revision Unknown Compiled by jarcec on Wed Nov 21 16:15:51 PST 2012 Protocol version: [1]
2nd step
Check what connectors are available on your Sqoop server:
sqoop:000> show connector --all
1 connector(s) to show: Connector with
id 1:
Name: generic-jdbc-connector
Class:
org.apache.sqoop.connector.jdbc.GenericJdbcConnector
Supported job
types: [EXPORT, IMPORT]
3rd step
sqoop:000> create connection --cid 1
Creating connection for connector
with id 1
Please fill following values to create new connection object
Name: First connection
Configuration configuration
JDBC Driver Class: com.mysql.jdbc.Driver
JDBC Connection String: jdbc:mysql://mysql.server/database
Username: sqoop
Password: *****
JDBC Connection Properties: There are currently
0 values in the map: entry#
Security related configuration options Max connections: 0 New
connection was successfully created with validation status FINE and
persistent id 1
step 4
now create a job for importing data
at the end it will also ask for extractore and loaders, use 1 as a value for both.
sqoop:000> create job --xid 1 --type import
Creating job for
connection with id 1 Please fill following values to create new job
object
Name: First job
Database configuration
Table name: users
Table SQL statement: Table
column names:
Partition column name:
Boundary query:
Output configuration
Storage type:
0 : HDFS Choose: 0
Output directory: /user/jarcec/users
New job was successfully created with
validation status FINE and persistent id 1
step 5
now start the job
sqoop:000> start job --jid 1
and import your data
you need to pass --target-dir argument HDFS path where sqoop should copy MySql records.
try:
sqoop import --connect jdbc:mysql://localhost/books --username root --password thanks --table authors --target-dir /mysqlCopy --m 1;
Related
I am new to cassandra. Here I am tring to transfer whole my MYSQL database to cassandra using sqoop. But after all setup, when i execute following command.
bin/dse sqoop import-all-tables -m 1 --connect jdbc:mysql://127.0.0.1:3306/ABCDatabase --username root --password root --cassandra-thrift-host localhost --cassandra-create-schema --direct
I have received following error.
Sqoop functionality has been removed from DSE.
It said that sqoop functionality is removed from datastax. can you please if it removed then is there any other way to do that?
Thanks
You can use Spark to transfer data - it should be easy, something like:
val table = spark.read.jdbc(jdbcUrl, "table", connectionProperties)
table.write.format("org.apache.spark.sql.cassandra").options(
Map("table" -> "TBL", "keyspace" -> "KS")).save()
Examples of jdbc URLs, options, etc. are described in Databrick's documentation as they could be different for different databases.
I am working with Apache Hadoop and Apache Sqoop. And I'm trying to import the mysql tables into the hdfs.
Here is the command which i am executing:
sqoop-import --connect jdbc:mysql://localhost/billing_engine -username root -password root > --table cpDetail;
I have setup the Sqoop home environment variable as follows:
export SQOOP_HOME=/Users/bng/Documents/sqoop-1.4.6.bin__hadoop-2.0.4-alpha
export PATH=$PATH:$SQOOP_HOME/bin
But executing the above command, gives me the following error:
readlink: illegal option -- f
usage: readlink [-n] [file ...]
usage: dirname path
/Users/bng/Documents/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/bin/sqoop-import: line 26: /Users/bng/sqoop: Undefined error: 0
Here is the screenshot showing my name node:
Please suggest, where am i wrong?
This is the right command which i needed to use:
sqoop import --connect jdbc:mysql://localhost/billing_engine?useSSL=false --username bil --password bil --table cpdetail -m 1 --target-dir /sqoopFromMysql
The details of the command is as follows;
Sqoop import : the command telling to use sqoop's import command
--connect : indicating the connection to be used
jdbc:mysql://localhost/billing_engine?useSSL=false : connecting to mysql using the jdbc. The host for db is localhost and database name is billing_engine. useSSL=false specifies that we are not connecting over SSL layer.
--username bil --password bil : specifies the username and password for the database.
--table cpdetail : specifies the particular table
-m 1 : specifies the maps to be used
--target-dir /sqoopFromMysql : specifies the target directory in HDFS, where the data will be imported.
I am using hadoop-2.6.0 with kerberos security. I have installed hbase with kerberos security and could able to create table and scan it.
I could run sqoop job as well to import data from mysql into hdfs but sqoop job fails when trying to import from mysql into HBase.
Sqoop Command
sqoop import --hbase-create-table --hbase-table newtable --column-family ck --hbase-row-key id --connect jdbc:mysql://localhost/sample --username root --password root --table newtable -m 1
Exception
15/01/21 16:30:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x734c0647, quorum=localhost:2181, baseZNode=/hbase
15/01/21 16:30:24 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknownerror)
15/01/21 16:30:24 INFO zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
15/01/21 16:30:24 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x14b0ac124600016, negotiated timeout = 40000
15/01/21 16:30:25 ERROR tool.ImportTool: Error during import: Can't get authentication token
Could you please try the following :
In the connection string add the port number as :
jdbc:mysql://localhost:3306/sample
Remove --table newtable. Create the required table on Hbase first with the column family.
mention --split-by id
Finally mention a specific --fetch-size , as the sqoop client for MySQL have an error internally which attempts to set the default MIN fetch size which will run into an exception.
Could you attempt the import again and let us know ?
I Installed Sqoop in my local machine. Following are the config information.
Bash.bashrc:
export HADOOP_HOME=/home/hduser/hadoop
export HBASE_HOME=/home/hduser/hbase
export HIVE_HOME=/home/hduser/hive
export HCAT_HOME=/home/hduser/hive/hcatalog
export SQOOP_HOME=/home/hduser/sqoop
export PATH=$PATH:$HIVE_HOME/bin
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HBASE_HOME/bin
export PATH=$PATH:$SQOOP_HOME/bin
export PATH=$PATH:$HCAT_HOME/bin
Hadoop:
Version: Hadoop 1.0.3
Hive:
Version: hive 0.11.0
Mysql Connector driver
version: mysql-connector-java-5.1.29
"The driver is copied to the lib folder of sqoop"
Sqoop :
version: sqoop 1.4.4
After making all the installation I create a table in mysql named practice_1, But when I run the load command to load data from mysql to hdfs the command throws an exception:
ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver
Coud anyone please guide me what can be the possible problem.
You need database driver in 'SQOOP' classpath check this
It has wonderful explanation about the 'SQOOP'
SQOOP has other options like
Ex: --driver com.microsoft.jdbc.sqlserver.SQLServerDriver -libjars=".*jar"
from here
You can use Sqoop with any other JDBC-compliant database. First, download the appropriate JDBC driver for the type of database you want to import, and install the .jar file in the $SQOOP_HOME/lib directory on your client machine. (This will be /usr/lib/sqoop/lib if you installed from an RPM or Debian package.) Each driver .jar file also has a specific driver class which defines the entry-point to the driver. For example, MySQL's Connector/J library has a driver class of com.mysql.jdbc.Driver. Refer to your database vendor-specific documentation to determine the main driver class. This class must be provided as an argument to Sqoop with --driver.
You may be interested in understanding the difference between connector and driver here is the article
Another solution which avoids using a shared library is adding the driver jar to the classpath of sqoop by using HADOOP_CLASSPATH. I haven't got the -libjars option to work. This solution works also on a secure cluster using kerberos.
HADOOP_CLASSPATH=/use.case/lib/postgresql-9.2-1003-jdbc4.jar
sqoop export --connect jdbc:postgresql://db:5432/user \
--driver org.postgresql.Driver \
--connection-manager org.apache.sqoop.manager.GenericJdbcManager \
--username user \
-P \
--export-dir /user/hive/warehouse/db1/table1 \
--table table2
This one works at least with sqoop 1.4.3-cdh4.4.0
You need to add the MySql connector to /usr/lib/sqoop/lib.
MySQL JDBC Driver by default is not present in Sqoop distribution in order to ensure that the default distribution is fully Apache license compliant.
Hope this helps...!!!
copy the 'mysql-connector-java-5.1.41-bin.jar' into sqoop/lib folder and execute sqoop import statements
If you have copied mysql driver to the sqoop lib folder. It will work for sure. Make sure you sqoop command is correct
/home/hduser/sqoop/bin/sqoop import --connect jdbc:mysql://localhost:3306/test --username root --password root -–table practice_1 -m 1
It's a Oozie ShareLib problem. The script below works for my:
At Shell
sudo -u hdfs hadoop fs -chown cloudera:cloudera /user/oozie/share/lib/lib_20170719053712/sqoop
hdfs dfs -put /var/lib/sqoop/mysql-connector-java.jar /user/oozie/share/lib/lib_20170719053712/sqoop
sudo -u hdfs hadoop fs -chown oozie:oozie /user/oozie/share/lib/lib_20170719053712/sqoop
oozie admin -oozie http://localhost:11000/oozie -sharelibupdate
oozie admin -oozie http://localhost:11000/oozie -shareliblist sqoop
At Hue Sqoop Client
sqoop list-tables --connect jdbc:mysql://localhost/retail_db --username root --password cloudera
More detail at:
https://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/
You need to grant priveleges to the tables as below:
grant all privileges on marksheet.* to 'root'#'192.168.168.1'
identified by 'root123';
flush privileges;
Here is sample command that I have successfully executed:
sqoop import --verbose --fields-terminated-by ',' --connect
jdbc:mysql://192.168.168.1/test --username root --password root123
--table student --hive-import --create-hive-table --hive-home /home/training/hive --warehouse-dir /user/hive/warehouse
--fields-terminated-by ',' --hive-table studentmysql
I am new to Sqoop. I am trying to import data from MySQL to hbase. That's why have to use Database connector for MySQL. Path to my connector file is /usr/lib/sqoop2/lib/mysql-connector-java-5.1.6.jar at server. Database name is :testhadoop and table which i am using is employee the command i enter is as
root#server:~# sqoop import --connect jdbc:mysql//localhost/testhadoop --driver com.mysql.jdbc.Driver --username root --table mytable
After hitting Enter key, i have to enter root password. And then a long long error message come
13/09/12 17:39:16 WARN sqoop.ConnFactory: Parameter --driver is set to an
explicit driver however appropriate connection manager is not being set
(via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
13/09/12 17:39:16 INFO manager.SqlManager: Using default fetchSize of 1000
13/09/12 17:39:16 INFO tool.CodeGenTool: Beginning code generation
13/09/12 17:39:16 ERROR manager.SqlManager:
Error executing statement: java.sql.SQLException:
No suitable driver found for jdbc:mysql//localhost/testhadoop
Please tell me how to get rid of this problem.
Based on the command line it seems that you are using Sqoop 1.x whereas the JDBC driver is in path for Sqoop2. I would recommend to copy the jar file mysql-connector-java-5.1.6.jar to /usr/lib/sqoop/lib instead, so that it's available for Sqoop 1.
Also I would strongly suggest to drop the parameter --driver as it will force Sqoop to use Generic JDBC Connector instead of the specialized MySQL connector.