I'm training myself for the HDPCD exam, so I'm testing all possible import and exports using MySQL to Hive. In this example, I would like to import a table from MySQL and create from scratch the same table in hive using the parameter --create-hive-table. Although in the [documentation][1] it's included I have found a right example to do it. I have tried this but it doesn't work
sqoop import --connect jdbc:mysql://master/poc --username root --table dept --where 'id_dept > 2' --hive-import --hive-database poc --hive-table deptv2 --create-hive-table true -m 1 --split-by id_dept
Please, guys if someone of you knows how to use it let me know it. I appreciate thanks so much
I'm back cause I have just to try again just putting the parameter without any input and it worked that's it. Anyways I'll leave an example. Probably it's going to help someone.
sqoop import --connect jdbc:mysql://master/poc --username root \
--table dept --where 'id_dept > 2' --hive-import \
--hive-database poc --hive-table deptv2 --create-hive-table -m 1 --split-by id_dept
Thanks.
Related
I wanna sqoop big table in mysql.
this table has about 300,000,000 rows and operating.
I wanna sqoop as soon as possible, not burden to system.
So, my sqoop command is
sqoop import \
--connect jdbc:mysql://.... \
--username ... \
--password ... \
--query "SELECT * FROM table_name where \$CONDITIONS" \
--hive-import \
--hive-overwrite \
--hive-table ........ \
--create-hive-table \
--mapreduce-job-name .... \
--num-mappers $mapper_count \
--fetch-size $patch_count \
--split-by "account_id"
I want to move all of big table, 500,000 at a time using 100 mappers.
I tried setting num-mappers=100 and fetch-size=500000. But only num-mappers adopted. So, operating Database stressed moving 3,000,000 rows each mappers.
Please advice.
Thanks.
I have a table (regularly updated) in Hive that I want to have in one of my tool that has a MySQL database. I can't just connect my application to the Hive database, so I want to export those data directly in the MySQL database.
I've searched a bit and found out that it was possible with Sqoop, and I've been told to use Oozie since I want to regularly update the table and export it.
I've looked around for a while and tried some stuff but so far I can't succeed, and I just don't understand what I'm doing.
So far, the only code I understand but doesn't work looks like that :
export --connect jdbc:mysql://myserver
--username username
--password password
--table theMySqlTable
--hive-table cluster.hiveTable
I've seen people using temporary table and export it on a txt file to then export it, but I'm not sure I can do it.
Should Oozie have specific parameters too ? I'm not the administrator so I'm not sure if I'm able to do it...
Thank you !
Try this.
sqoop export \
--connect "jdbc:sqlserver://servername:1433;databaseName=EMP;" \
--connection-manager org.apache.sqoop.manager.SQLServerManager \
--username userid \
-P \
--table theMySqlTable\
--input-fields-terminated-by '|' \
--export-dir /hdfs path location of file/part-m-00000 \
--num-mappers 1 \
I'm trying to load MySQL table to hbase using sqoop. I'm using the below command but it is showing unexpected tIdentifier error. Please help.
sqoop import --connect jdbc:mysql://localhost/manideep --username root --password cloudera --table sample --hbase-table H_LOAN --column-family CD --hbase-row-key id -m 1
Using the following command has worked for me to achieve hbase table import in Sgoop:
sqoop import --connect "jdbc:mysql://localhost/manideep" --username <username> --password ****** --table <mysql-table-name> --columns "column1,colmn2,column3" --hbase-table <hbase-table-name> --column-family <hbase-table-column-family> --hbase-row-key <row-key-column-name> -m 1
If the solution does not work, please post the exact issue (exception details) you have faced.
I'm trying to import data from MySQL table to Hive using Sqoop. From what I understood there are 2 ways of doing that.
Import data into HDFS and then create External Table in Hive and load data into that table.
Use create-hive-table while running Sqoop query to create a new table in Hive and directly load data into that. I am trying to do this but can't do it for some reason
This is my code
sqoop import \
--connect jdbc:mysql://localhost/EMPLOYEE \
--username root \
--password root \
--table emp \
--m 1 \
--hive-database sqoopimport \
--hive-table sqoopimport.employee \
--create-hive-table \
--fields-terminated-by ',';
I tried using --hive-import as well but got error.
When I ran the above query, job was successful but there was no table created in hive as well as data was stored in \user\HDFS\emp\ location where \HDFS\emp was created during the job.
PS: Also I could not find any reason for using --m 1 with Sqoop. It's just there in all queries.
I got the import working with following query. There is no need to write create-hive-table we can just write new table name with hive-table and that table will be created. Also if there is any issue then go to hive-metastore location and run rm *.lck then try import again.
sqoop import \
--connect jdbc:mysql://localhost/EMPLOYEE \
--username root \
--password root \
--table emp4 \
--hive-import \
--hive-table sqoopimport.emp4 \
--fields-terminated-by "," ;
In my system I have database in Mysql. I want to import that to hadoop file system. I found something about Sqoop, but i'm not getting command to do that.
sqoop import --connect jdbc:mysql://mysql-server-name/db_name --username user --password password --table table_name --target-dir target_directory_name -m1
Hope it helps..
You need to install mysql jdbc/java connector and run sqoop command.
sudo yum install mysql-connector-java
ln -s /usr/share/java/mysql-connector-java.jar /var/lib/sqoop/mysql-connector-java.jar
You can run sqoop command from
Save data into mysql from hive hadoop through sqoop?
1) Install & configure MySQL first. Create database in MySQL
2) sqoop import --connect jdbc:mysql://localhost/databasename --username $USER_NAME --password $PASSWORD$ --table tablename --m 1 command will import data.
e.g.
sqoop import --connect jdbc:mysql://localhost/testDb --username root --password hadoop123 --table student --m 1
In above command, values of various parameters database:‘testDb’ , username: ‘root’, password: ‘hadoop123’, and table student.
Have a look at this article 1 and article 2 for better understanding in step-by-step manner