I would like to know how I can move date from Hive to MySQL?
I have seen example on how to move hive data to Amazon DynamoDB but not for a RDBMS like MySQL. Here is the example that I saw with DynamoDB:
CREATE EXTERNAL TABLE tbl1 ( name string, location string )
STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
TBLPROPERTIES ("dynamodb.table.name" = "table",
"dynamodb.column.mapping" = "name:name,location:location") ;
I would like to do the same but with MySQL instead. I wonder if I need to code my own StorageHandler? I also to do not want to use sqoop. I want to be able to do my query directly in my HiveQL script.
You'd currently need a JDBC StorageHandler, which one has not been created just yet, but you could certain build your own.
There is currently an issue report for this which you can follow here:
https://issues.apache.org/jira/browse/HIVE-1555
Have you tried using Sqoop?. Its a good tool to do such kind of stuff.
There are many options. You can download the files in hive as csv file and then try bulk insert into mysql tables. You can use Sqoop. Or you can use some of the popular ETL tools like
Pentaho and many others.
Related
I have a .sql file from Oracle which contains create table/index statements and a lot of insert statements(around 1M insert).
I can manually modify the create table/index part(not too much), but for the insert part there are some Oracle functions like to_date.
I know MySql has a similar function STR_TO_DATE but the usage of the parameter is different.
I can connect to MySQL, but the .sql file is the only thing I got from Oracle.
Is there any way I can import this Oracle .sql file into MySQL?
Thanks.
Although the above job can be done by manually editing the script appropriately however there are products available which can be of use. Refer to the link for more information on one such product.
P.S. I am not affiliated in any way to the product
Since you mention about insert script basically i think you will be inserting data for this you can use any ETL tool, like open source tool like Pentaho data integrator, pretty simple to do, just search table to table transformation from different database connection on youtube to learn you should be able to connect to both mysql and oracle database else this wont help, but all the table structures you should create manually in the source database for data - you can just load it using ETL, no need to edit for every single line of insert if its more than 100 may be its very painful thing to do.
Was wondering if anyone had any insight or recommended tools for exporting the records from a PostgreSQL database and importing them into a MySQL database. I believe the table structure is 100% identical.
Thoughts? Thanks!
The command
pg_dump --data-only --column-inserts <database_name>
will generate SQL-standard-compliant INSERT statements with all column names listed and one VALUES clause per INSERT. This is the most portable way of moving data from PostgreSQL to any other SQL database.
Check out SquirrelSQL, it can pump data from one database brand into another via the DBCopy plugin. When the table structures are really identical it works quite well.
There is a ruby app called Taps that will do it. I've used it before with great success:
http://adam.heroku.com/past/2009/2/11/taps_for_easy_database_transfers/
I have a table in a postgreSQL database that I would like to convert into a MySQL table. What would be the easiest way to do this? Any tools to do this? Remember, I am not converting the whole database to MySQL... I am just taking a table from that postgreSQL and convert it to MySQL table
You can use pg_dump to create a SQL dump from a single table and use this dump as input for mysql. Take a look at the option --data-only when you already have the tablestructure in your MySQL database and --column-inserts to create straight forward INSERT statements.
An ETL tool like SSIS, Pentaho, Talend or several others will do the trick. Most support a wide variety of data sources and destinations.
I'm wondering if there is a utility that exists to create a data dictionary for a MySQL database.
I'm considering just writing a php script that fetches the meta data about the database and displays it in a logical format for users to understand but I'd rather avoid that if there is some pre-built utility out there that can simply do this for me.
Have you looked into HeidiSQL or phpMyAdmin?
Also, MySQL Admin.
Edit#1 fixed typo, added more info
Take a look at https://stackoverflow.com/a/26703098/4208132
There is a db_doc.lua plugin for MySQL Workbench CE
[EDITED]
It seems that the LUA plugin support was discontinued.
So I wrote a plugin in Python to generate data dictionaries.
It is available at: https://github.com/rsn86/MWB-DBDocPy
Looks like MySQL Admin is now MySQL Workbench and you need the Enterprise version to get their reporting tool called DBDoc. It explains a little about customizing DBDoc reporting templates at http://dev.mysql.com/doc/workbench/en/dbdoc-templates.html
The easiest thing to do is Download Toad for MySQL, which is free, and create your own query against the mysql information_schema internal database. You can add columns you want to the query below. Then select all results and export as csv using TOAD.
use information_schema;
desc columns;
select c.table_name, c.column_name, c.data_type from columns c
where c.table_schema = "mydatabaseinstance";
I have a MySQL database with over 40,000 records I want to import into a new PostgreSQL database; I want to be able to map the values from the old table and column names into new table and column names... how do I do this?
For instance, I want to take this:
Table name: Horribly_Named_Table
=> Horribly_Named_Column: value1
(MySQL)
... and translate it to this:
Table name: better_named_table
=> better_named_column: value1
(PostgreSQL)
I've never done a move like this before, so any help is appreciated!
I recommend using a simple transformation within Pentaho Data Integration: setup is very simple and there is a wizard for loading database base data from one database to another:
See a similar answer here:
Migrate from Oracle to MySQL
If you are only referring to the difference in UPPER/lowercase names, then you don't really need to do something.
Just make sure you are not quoting the table names and they will not be case sensitive.
This_Table_Name is the same as this_table_name and that is the same as THIS_TABLE_NAME.
But "this_table_name" is something different then "This_Table_Name"
mysqldump has a compatibility mode, check "ansi" and "postgresql".