I am looking for a way to query the metadata of my HIVE data with a HiveQL command.
I configured a MySQL metastore, but it is necessary to query the metadata via HIVE command because then I want to access the data with ODBC connection to the HIVE system.
To see them from Hive, you must use the commands to display the DDL. You'll probably need to parse it.
Database metadata:
describe database extended <db_name>
To see table and columns metadata:
describe formatted <db_name>.<table_name>
Another option is to directly connect to the matastore database, but you'll be outside Hive.
You can do it currently using Hive JDBC StorageHandler: https://github.com/qubole/Hive-JDBC-Storage-Handler
Example of table creation from their page:
DROP TABLE HiveTable;
CREATE EXTERNAL TABLE HiveTable(
id INT,
id_double DOUBLE,
names STRING,
test INT
)
STORED BY 'org.apache.hadoop.hive.jdbc.storagehandler.JdbcStorageHandler'
TBLPROPERTIES (
"mapred.jdbc.driver.class"="com.mysql.jdbc.Driver",
"mapred.jdbc.url"="jdbc:mysql://localhost:3306/rstore",
"mapred.jdbc.username"="root",
"mapred.jdbc.input.table.name"="JDBCTable",
"mapred.jdbc.output.table.name"="JDBCTable",
"mapred.jdbc.password"="",
"mapred.jdbc.hive.lazy.split"= "false"
);
I tested, it works fine with MySQL. And FilterPushDown also works.
There are already a bunch of tables in the SYS db and the INFORMATION_SCHEMA db which maps to the tables in the RDBMS.
Check this file:
https://github.com/apache/hive/blob/1e3e07c87e71dc16f05ad269b250d65ad7c02232/metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-2.hive.sql
If the table isn't there you can create hive tables pointing to mysql, the same way it is done in this file.
eg.
https://github.com/apache/hive/blob/1e3e07c87e71dc16f05ad269b250d65ad7c02232/metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-2.hive.sql#L1520-L1546
Related
I have table which has a few data.
name score
1 AAA 100
2 BBB 98
3 CCC 85
Now I want to make the insert sentence such as
insert into pepolescore(name,score) VALUE("CCC",85)
automatically.
Is there any command to do this or any function ? by mysql commandline or phpmyadmin.
MySQL queries can address another schema on the same MySQL Server instance by using qualified table names. See https://dev.mysql.com/doc/refman/8.0/en/identifier-qualifiers.html
But this does not work if the tables are on separate MySQL Servers. A given SQL query can only address schemas on the same server.
Here are a few workarounds:
Use mysqldump to export data from one table and then use mysql to import it to the other table on the other instance. You need to be careful not to let mysqldump output the DROP TABLE command, so read about the options here: https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html
MySQL supports a table engine called FEDERATED, where a table can function as a sort of proxy to a table on another MySQL Server. Then you can use INSERT ... SELECT syntax as if the tables were co-located on the same MySQL Server. The Federated engine has limitations, so read https://dev.mysql.com/doc/refman/8.0/en/federated-storage-engine.html and its subsections to learn more.
Use a community tool such as pt-archiver to copy data from one MySQL instance to the other. Read the manual to learn more: https://docs.percona.com/percona-toolkit/pt-archiver.html
Write your own custom code in a client application. Create two connections, one for each MySQL Server. Fetch query results from the first server, and store the resulting rows in variables in your application. Then use these rows as the tuples to insert using the second connection to the other MySQL Server. This involves writing more code, but you get a lot of flexibility.
how to create sql statement ?
say i have create a data base in MySQL and i want to port it another database.
so i want convert my database to sql statement.
Use mysqldump. It can do that and much more.
From the documentation:
The mysqldump client [produces] a set of SQL statements that can be run to reproduce the original schema objects, table data, or both.
I would like to know how I can move date from Hive to MySQL?
I have seen example on how to move hive data to Amazon DynamoDB but not for a RDBMS like MySQL. Here is the example that I saw with DynamoDB:
CREATE EXTERNAL TABLE tbl1 ( name string, location string )
STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
TBLPROPERTIES ("dynamodb.table.name" = "table",
"dynamodb.column.mapping" = "name:name,location:location") ;
I would like to do the same but with MySQL instead. I wonder if I need to code my own StorageHandler? I also to do not want to use sqoop. I want to be able to do my query directly in my HiveQL script.
You'd currently need a JDBC StorageHandler, which one has not been created just yet, but you could certain build your own.
There is currently an issue report for this which you can follow here:
https://issues.apache.org/jira/browse/HIVE-1555
Have you tried using Sqoop?. Its a good tool to do such kind of stuff.
There are many options. You can download the files in hive as csv file and then try bulk insert into mysql tables. You can use Sqoop. Or you can use some of the popular ETL tools like
Pentaho and many others.
We will migrate the database from mysql to postgresql in our product(through java). So we need to change the mysql query to postgresql query in java application. How to create the table i.e., databasename.tablename in postgresql.
For mysql, we can directly create the table e.g create table information.employee.
Here database name is "information" and table name is "employee" . Is it possible to achieve same query in postgresql.
I searched google it says cross database reference is not possible. Please help me.
I saw pg_class table it contains the table names in the specific database, like wise databse and tables relationships are stored in any other table.
This is normally done using schemas rather than databases, which is more or less like how MySQL organizes it anyway.
Instead of
create database xyz
use
create schema xyz
When you create tables, create them:
create table xyz.myTable
you will need to update your search path to see them on the psql command line tool, or if you want to query them without using the schema explicitly. The default schema is public, so when you create a table without a schema name, it ends up in public. If you modify your search_path as below, the default schema becomes the first in the list: xyz.
set search_path=xyz,public,pg_catalog;
and you must not have spaces in that statement. You can do it globally for a user/role too:
alter role webuser set search_path=xyz,public,pg_catalog;
Also, don't forget that postgresql string matches are case sensitive by default (this one catches people out a lot).
If you want to have different physical locations for the files for each schema, you can do that with tablespaces. If you have a look at the postgresql documentation page, they have info on how to do it, it's pretty easy.
database in MySQL == schema in PostgreSQL. So you will most probably want to migrate all your mysql dbs into one postgres db. Then you will be able to do "cross-database" queries.
See my answer to this question: Relationship between catalog, schema, user, and database instance
I would like to select data from a second MySQL database in order to migrate data from one server to another.
I'm looking for syntax like
SELECT * FROM username:password#serverip.databaseName.tableName
Is this possible? I would be able to do this in Microsoft SQL Server using linked servers, so I'm assuming it's possible in MySQL as well.
You can create a table using FEDERATED storage engine:
CREATE TABLE tableName (id INT NOT NULL, …)
ENGINE=FEDERATED
CONNECTION='mysql://username:password#serverip/databaseName/tableName'
SELECT *
FROM tableName
Basically, it will serve as a view over the remote tableName.
There are generally two approaches you can take, although neither of them sound like what you're after:
Use replication and set up a master/slave relationship between the two databases.
Simply dump the data (using the command line mysqldump tool) from the 1st database and import it into the 2nd.
However, both of these will ultimately migrate all of the data (i.e.: not a subset), although you can specify specific table(s) via mysqldump. Additionally, if you use the mysqldump approach and you're not using InnoDB you'll need to ensure that the source database isn't in use (i.e.: has integrity) when the dump is created.
You can't do this directly, but as someone else alluded to in a comment, you can use mysqldump to export the contents of a table as a SQL script.
At that point you could run the script on the new server to create the table, or if more manipulation of the data is required, import that data into a table with a different name on the new server, then write a query to copy the data from there.