This question already has an answer here:
Is it possible to load parquet table directly from file?
(1 answer)
Closed 7 years ago.
We want to do real time replication from mysql to hdfs with the files being stored as the parquet format in the hdfs cluster.
As for as we know ,we can do this using either
1)tungsten replicator or
2)Mysql server supports live replication to hdfs.
But our problem is that none of them support conversion to parquet while loading data to hdfs.
so just wanted to know whether is there any way to do real time replication with the file being stored as parquet in hdfs cluster.
Second question is that when you load csv file in hive tables using "LOAD DATA INPATH" and if the table has been define as Parquet file format ,will hive convert the file to parquet format or we need to write a utility to convert the file to parquet format and then load.
Second question : The CREATE TABLE statement should specify the Parquet storage format with syntax.
it all boils down to the version of Hive . some version do not support parquet file
Related
I am fetching JSON data from different API's. I want to store them in HDFS and then use them in MongoDB.
Do I need to convert them to avro, sequence file, parquet, etc., or can I simply store them as plain JSON and load them to the database later?
I know that if i convert them to another format they will get distributed better and compressed, but how will I be able then to upload an avro file to MongoDB? MongoDB only accepts JSON. Should I do another step to read them from avro and convert them to JSON?
How large is the data you're fetching? If it's less than 128MB (with or without compression) per file, it really shouldn't be in HDFS.
To answer the question, format doesn't really matter. You can use SparkSQL to read any Hadoop format (or JSON) to load into Mongo (and vice versa).
Or you can write the data first to Kafka, then use a process such as Kafka Connect to write to both HDFS and Mongo at the same time.
I have a exported sql schema file. (similar to what we have here https://livesql.oracle.com/apex/livesql/file/content_O5AEB2HE08PYEPTGCFLZU9YCV.html)
This file is having all Create table and insert values commands.
I want to export all the database to a CSV or JSON format.
Is there a way to achieve the same?
You appear to be asking how to convert the data from the .sql script directly to a raw data format. That would require a SQL parser capable of reading the .sql script format. This is implemented in MySQL using a combination of the mysql client and the MySQL Server SQL parser. It would be an awful lot of work to duplicate this.
Honestly, the easiest solution is to use the mysql client to import the .sql script's tables and data into a MySQL instance. Then you could dump the data in CSV format, or whatever other format you want.
You can run queries using the mysql client in batch mode to dump results to CSV format (really tab-delimited), or you could write a simple client in Python or whatever your favorite language is.
Another alternative is to use mysqldump --tab to dump CSV files. I'll encourage you read the documentation about that.
I have a dataset which is exported from MongoDB and compressed as snappy file and want to load the data into MySQL.
I have searched a lot but couldn't find a solution. So is there a way to load the snappy data from MongoDB into MySQL?
This question already has answers here:
Migrating data into Firebase from MySQL
(4 answers)
Closed 6 years ago.
I read about exporting it to json, but how do I go about taking the table structures and exporting them to json and the follow-up question is how do i do so for the data in the tables as well!?
It's not possible to import mysql table structure in Firebase console. You'll need to convert your mysql data into JSON format then upload it. Also, if you're uploading more than 250MB JSON file then use this Firebase-Import utility tool. Hope this helps.
This question already has answers here:
Automate transfer of csv file to MySQL
(3 answers)
Closed 8 years ago.
I would like to know what would be the best way to automate the loading of an excel file into a mysql database.
The file would most likely be .csv, although, if there is a solution for text files, i can live with that. The data in to file would have to replace what is already in the database table.
I am searching for a solution meanwhile, and have found several for doing approximately this manually, as in, loading a file once, but i need this to happen every few minutes, if it is possible.
There is a native MySQL feature that allows importing a CSV file easily: LOAD DATA INFILE. All you need to do is declare your field- and line-separator correctly, if the default settings do not match your input file.
Please note that a CSV file is not an Excel file. It is a file format that Excel happens to be able to read.
If you really want to import Excel files (a .xlsx file, taht is), then you need some external library to first parse the Excel file, as MySQL is not able to read it natively.