I have a problem using sqoop to export hive bigint data to mysql.
The type of the column in mysql and hive is bigint.
I get the following error:
Caused by: java.lang.NumberFormatException: For input string: "3465195470"
...
At java.lang.Integer.parseInt (Integer.java:583)
It seems that an error occurs when converting a string stored in hdfs to a numeric type.
Both hive and mysql columns are bigint types, how do i solve the problem?
add sqoop command
export -connect "jdbc:mysql://{url}/{db}?{option}"
--username {username}
--password {password}
--table {username}
--columns "column1,column2,column3"
--export-dir /apps/hive/warehouse/tmp.db/{table}
--update-mode allowinsert
--update-key column1
--input-fields-terminated-by "\001"
--input-null-string "\\N"
--input-null-non-string "\\N"
--null-string "\\N"
--null-non-string "\\N"
It could be an issue due to missing column or wrong column position.
Also there is no need of --null-string and -null-non-string. These are used in sqoop import commands.
Related
I'm trying to export one of the tables from hive to MySQL using sqoop export. The hive table data contains the special characters.
My hive "special_char" table data:
1 じゃあまた
2 どうぞ
My Sqoop Command:
sqoop export --verbose --connect jdbc:mysql://xx.xx.xx.xxx/Sampledb --username abc --password xyz --table special_char --direct --driver com.mysql.jdbc.Driver --export-dir /apps/hive/warehouse/sampledb.db/special_char --fields-terminated-by ' '
After using the above sqoop export command, the data is stored in the form of question marks (???) instead of actual message with special characters.
MySql "special_char" table:
id message
1 ?????
2 ???
Can anyone please help me out,in storing the special characters instead of question marks (???).
Specify proper encoding and charset in the JDBC URL as below:
jdbc:mysql://xx.xx.xx.xxx/Sampledb?useUnicode=true&characterEncoding=UTF-8
sqoop export --verbose --connect jdbc:mysql://xx.xx.xx.xxx/Sampledb?useUnicode=true&characterEncoding=UTF-8 --username abc --password xyz --table special_char --direct --driver com.mysql.jdbc.Driver --export-dir /apps/hive/warehouse/sampledb.db/special_char --fields-terminated-by ' '
Please verify charset encoding for Japanese characters and use proper one.
Reference: https://community.hortonworks.com/content/supportkb/198290/native-sqoop-export-from-hdfs-fails-for-unicode-ch.html
I have 1000 tables with more than 100000 records in each table in mysql. The tables have 300-500 columns.
Some of tables have columns with special characters like .(dot) and space in the column names.
Now I want to do sqoop import and create a hive table in HDFS in a single shot query like below
sqoop import --connect ${domain}:${port}/$(database) --username ${username} --password ${password}\
--table $(table) -m 1 --hive-import --hive-database ${hivedatabase} --hive-table $(table) --create-hive-table\
--target-dir /user/hive/warehouse/${hivedatabase}.db/$(table)
After this the hive table is created but when I query the table it shows error as
This error output is a sample output.
Error while compiling statement: FAILED: RuntimeException java.lang.RuntimeException: cannot find field emp from [0:emp.id, 1:emp.name, 2:emp.salary, 3:emp.dno]
How can we replace the .(dot) with _(underscore) while doing sqoop import itself. I would like to do this dynamically.
Use sqoop import \ with --query option rather than --table and in query use replace function .
ie
sqoop import --connect ${domain}:${port}/$(database) --username ${username} --password ${password}\
-- query 'Select col1 ,replace(col2 ,'.','_') as col from table.
Or (not recommended) write a shell script which can do find and replace "." to "_" (Grep command)at /user/hive/warehouse/${hivedatabase}.db/$(table)
how does Sqoop mapped import csv file to my sql table's column ? I just ran below import and export sqoop command and it work properly but not sure how Sqoop mapped the imported result into my sql table column's ? I have CSV file created manually which I want to export to my sql so need a way to specify csv file & column mapping ..
sqoop import \
--connect jdbc:mysql://mysqlserver:3306/mydb \
--username myuser \
--password mypassword \
--query 'SELECT MARS_ID , MARKET_ID , USERROLE_ID , LEADER_MARS_ID , CREATED_TIME , CREATED_USER , LST_UPDTD_TIME , LST_UPDTD_USER FROM USERS_TEST u WHERE $CONDITIONS' \
-m 1 \
--target-dir /idn/home/data/user
Deleted record from my sql database and run the below export command which inserted data back into table .
sqoop export \
--connect jdbc:mysql://mysqlserver:3306/mydb \
--table USERS_TEST \
--export-dir /idn/home/data/user \
--username myuser \
--password mypassword \
You can utilize --input-fields-terminated-by and --columns parameters to control the structure of the data to be exported back to RDBMS through Sqoop.
I would recommend you to refer the sqoop user guide for more information.
I am trying to import data to HDFS from a RDBMS table. I am then using create-hive-table to copy schema to hive and then load data to that hive table.
command used to import to HDFS
sqoop import --connect jdbc:mysql://localhost/sqoop --username sqoop --password sqoop --table customers --warehouse-dir testingsqoop -m 1 --fields-terminated-by ',' --enclosed-by "\'" --lines-terminated-by "\n"
command used to create hive table:
sqoop create-hive-table --connect jdbc:mysql://localhost/sqoop --username sqoop --password sqoop --table customers --hive-table customers --fields-terminated-by "," --enclosed-by "\'" --lines-terminated-by "\n"
And finally the query used to load data to hive
load data inpath '/user/cloudera/testingsqoop/customers/*' into table customers;
As I am enclosing the fields with a single quote ', hive while creating the table is not considering the --enclosed-by flag, hence the columns in the hive table are still having quotes '.
NULL 'Richard' 'Hernandez' 'XXXXXXXXX' 'XXXXXXXXX' '6303 Heather Plaza' 'Brownsville' 'TX' '78521'
However if I don't use --enclosed-by it works fine, but i want to have it.
1) Could you please help regarding this?
2) Also is there any way i can specify multiple chars for field terminator?
Thanks!
Try below,
--fields-terminated-by \01
--hive-drop-import-delims
--null-string \N
--null-non-string \N
in your sqoop import data query and hive table definition queries.
Most likely, your syntax is causing the exception. Try using:
--enclosed-by "'"
instead of \'.
Yes, you can import with multiple characters set as the field delimiters.
I have my data store into hive table.
i want to transfer hive tables selected data to mysql table using sqoop.
Please guide me how to do this?
check out the sqoop guide here
You need to use sqoop export, here is the example
sqoop export --connect "jdbc:mysql://quickstart.cloudera:3306/retail_rpt_db" \
--username retail_dba \
--password cloudera \
--table departments \
--export-dir /user/hive/warehouse/retail_ods.db/departments \
--input-fields-terminated-by '|' \
--input-lines-terminated-by '\n' \
--num-mappers 2
sqoop export to export data to mysql from Hadoop.
--connect JDBC url
--username mysql username
--password password for mysql user
--table mysql table name
--export-dir valid hadoop directory
--input-fields-terminated-by column delimiter in Hadoop
--input-lines-terminated-by row delimiter in Hadoop
--num-mappers number of mappers to process the data