I am using sqoop to import data from Mysql into Hbase.
It works fine but there is one issue.
As i read from Sqoop documentation , sqoop converts mysql data into String and then store it in Hbase.
However this would be problem for me as i will have to export data back from Hbase to Mysql and at that time , how will sqoop deduce data type information for the Hbase data ?
Someone please help for solution to this problem.
What you can do is - during export, just export it to a temporary table in MySql side. At that point the datatype will be different. Then write a query to insert them to the original MySql table from the temp table and during this time filter out unexpected data or convert datatype.
I faced pretty similar issue with timestamp datatype as in Hive I was storing it as bigint. During export I first inserted them as it is to make Sqoop export works. If its successful then I run a query that actually load those data from temp table to original table while converting the data at the same time. Hope it helps.
Related
i've a MSSQL database and trying to migrate to MySQL database.. the problem is when I using MySQL WorkBench, some table records in my MSSQL database is not migrated (there is an error and MySQL Workbench not responding).. is there any tools to export MSSQL table records into SQL file that compatible to be executed in MySQL?
Both T-SQL and MySQL and support VALUES clause. So, right click on the database and select Generate scripts from Tasks:
Then you can choose objects:
and then make sure you have selected to get the data, too:
You can even get the schema and change it a little bit to match the MySQL syntax.
For small amount of data this is pretty cool. If you are exporting large tables it will be better to use another tool. For example, using bcp you can export your data in CSV format and then import it in the MySQL database.
I have imported table from SQL server to Hive, whereas float field from SQL server is getting converted to double into hive. And the few of the values are loaded with exponential format.
Ex.
Value from Table= 10488666.6666667
Value loaded into Hive= 1.0488666666666666E7
Also there is one more issue observed for same field where extra precisions are getting added,
Ex. Value from table= 3688666.66666667
Value from Hive table= 3688666.6666666665
Can someone suggest me how to get rid of this and load the values as it is from RDBMS to Hive ,since it will be ease to query in hive.
Don't rely on what you are seeing on JDBC client, Hive shell.
See actual data in HDFS of your hive table. To check this go to hive shell and fire
show create table tablename
Check location key and go to hdfs and check actual data. It should be same.
I did face such issue in the past, an easy way out would be use to "String" in hive table. The system does put additional effort to convert on the fly, however, it should be negligible.
I have a huge data in Excel. I need to import it into MySQL. I've saved my Excel file as CSV. There are several columns in my table which are having contact numbers. I've sat format for such fields to 'Number' with decimal places zero. When I saved it as CSV, its showing such values of such fields with Exponential Notification E+.
eg. Mobile_Number is 9840741284 but its showing 9.84E+09 in csv.
I am using MySQL Workbench 6.3 CE to view it. If I keep Mobile_Number column's datatype as int then it stores data as "2147483647" in mysql. All the records which are having Exponential sign are showing the same value in mysql which is 2147483647. That's why I changed the data type to varchar(255) from int. Now it showing result 9.84E+09 format. I am trying to import the data using load data infile command.
How to resolve it? Is there any other way to import data from excel to mysql as it is?
I've changed the data type of Mobile_Numbers to bigint rather than only int in mysql. And its working properly. Thanks a lot Adyson for great help.
I am trying to import a CSV with columns [ID, timestamp, account, liters]->[#, datetime, #, #] using the MYSQL workbench 6.3. MySQL creates the table fine using the CSV (imports 1 record) and then I reimport the same CSV for data rows. MySQL identifies the columns and matches them with table well. Then it takes a long time and reports that all rows have been imported but table shows no rows.
I have searched forums and seem people had problems with timestamps but the solutions involved using the commandline. Can someone please guide if I should format my csv differently? It's a big pain. Thanks
OK..so this was problem with date format. When I specified the datetime field, I couldn't see a field popping up at the bottom to specify the format as per https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior (screen resolution issues). Anyway, I types in the matching format there and CSV is importing now..Thanks
Currently, I’m executing the following steps(Hadoop 1.1.2, Hive 0.11 and Sqoop-1.4.3.bin__hadoop-1.0.0) :
Import data from MySQL to Hive using Sqoop
Execute a query in Hive and store its output in a Hive table
Export the output to MySQL using Sqoop
I was wondering if it would be possible to combine steps 2 & 3 – the output of the Hive query written directly to the MySQL database.
I read about the external tables but couldn’t find an example where the LOCATION clause points to something like jdbc:myql://localhost:3306//. Is it really possible?
This thread talks about the JDBC Storage Handler but couldn't find a Hive example for the same(I guess its unimplemented!)
The link you provided, it seems the Bug is Unresolved.
But from your problem what I understand is that, you want to do a select query in hive and the output of this query needs to written in MySql. Correct me if I am wrong ?
If this is case you can use Sqoop export for this.
Please check this answer of mine: https://stackoverflow.com/a/17753176/1970125
Hope this will help.