Automate transfer of csv file to MySQL - mysql

I have a csv file that I will be regularly updating through a batch script that calls cygwin+ bash script. I would like to automate the upload of the csv file into a MySQL database such that a table in my database would be updated with the csv file at regular intervals. The database is currently running on a Windows Server 2003 machine and administered with phpMyAdmin.
I have looked online and found some ways that I could achieve part of that, but I am confused as to where the code presented in those sources should be placed and how they would be called. For instance, Import CSV file directly into MySQL seems to show how to upload a csv file to a MySQL database from the SQL command line once, but not repeatedly, the latter being what I need.
I would prefer the solution to involved bash scripting (as opposed to batch and php) if possible (i.e. I would prefer a solution that I could integrate with the bash scripts that update the csv file).
Thank you

You can execute a MySQL script from the command line by doing something like:
mysql -uUsername -pPassword database_name < infile.sql
You could invoke that from the command line and in the infile.sql you could have code like:
LOAD DATA INFILE 'filename.csv' TO table_name
FIELDS TERMINATED BY ','

You can use a here document:
# some bash script stuff
mysql ... <<EOF
SQL COMMANDS
GO HERE
EOF
# more bash script stuff

You can use Quartz to create a cronjob - for periodically updating your database. with the help of cronmaker (http://www.cronmaker.com/), you get to choose when and how often your database gets updated.
This is a sample SQL Script to import data into your MySQL database:
LOAD DATA INFILE 'c:/.../filename.csv'
INTO TABLE discounts
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
run the above script in your cronjob using your preferred language.

Related

Automatic export database tables from mariadb server to a linux server with .csv file format

I have to automatically generate a csv file of database tables from mariadb server to a linux server. Using maraidb 10.2.33
I created an event in mariadb but I could not find any files generated in the linux server.
my code:
CREATE EVENT `automatic_export_computers`
ON SCHEDULE EVERY 1 DAY_HOUR STARTS '2022-01-24 15:15:00'
ON COMPLETION PRESERVE
ENABLE
COMMENT 'Save Table to CSV for Excel'
Do SELECT * FROM `glpi`.`glpi_computers`
INTO OUTFILE '/data_computers2.csv'
FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n';
so is there any script using I can generate csv file.
plz help!
Thank you!
Make sure that the event scheduler is enabled. If not, enable it with set global_event_scheduler=ON or add the option to your configuration file.
Make sure that the user under which MariaDB is running, has permissions to write to the specified location. In your example writing to /data_computers2.csv will require that MariaDB server is running as root (you can specify ./data_computer2.csv so the file will be written to the data directory of MariaDB server).
Open the error log and check why the event wasn't executed successfully.

Import CSV File to MYSQL DB Automatically

I would like to ask for your help please. Here's the scenario.
From my web application: I will upload a CSV file everyday from my everyday sales data.
Next on the server where my file was saved, it should be automatically inserted to MySQL DB.
I've done the insertion but manually, using command line in ubuntu.
What I want to achieve is that every time I upload a new CSV file on my webpage, it will be automatically inserted to my MySQL Database.
I've done this to do the insert to DB: but this is only for a specific file. what if there is new file uploaded with different name?
mysql -uname -ppassword --local_infile=1 DASHBOARD -e "LOAD DATA LOCAL INFILE '/var/www/html/webapp/files/files.csv' INTO TABLE files_temp FIELDS TERMINATED BY ',' IGNORE 1 ROWS (column1. col2, col3, col4)"

Use LOAD DATA INFILE and SELECT INTO OUTFILE in Cloud SQL

I am testing Google Cloud SQL and I have to test the instructions LOAD DATA INFILE and SELECT...INTO OUTFILE.
I understood that the instruction LOAD DATA LOCAL INFILE could be used instead of LOAD DATA INFILE, but how can it be used in Cloud SQL?
Can the instruction SELECT...INTO OUTFILE be used as is?
Thank you Renzo for your answer but when I try to execute the instruction SELECT.. INTO OUTFILE when I am connected to the Cloud SQL instance, I get the following error message:
ERROR 1045 (28000) at line 1: Access denied for user 'root'#'%'
I tried to used the instruction LOAD DATA LOCAL INFILE from my computer connected to Cloud SQL instance on MySQL and it worked. I used a file located on my computer to import CSV data into a table successfully. But I am wondering if I can use a file present in a bucket to do the same...
I try to explain what I need to do:
We are migrating our Website to GCP App Engine/Cloud SQL. We extensively use both instructions LOAD DATA INFILE which select files from a folder "ftp" to load them into the databases. As well, we use the SELECT INTO OUTFILE to export data to CSV files into a folder of our Website. So my concern is to be able to use the same process on App Engine/Cloud SQL.
The difference is that the Website instance and database instance are separated on GCP. Should we use buckets on GCP to replace our folders ? Or should we create folders on App Engine / Cloud SQL instance instead? What is the best solution according to you?
Thanks in advance
According to the page about Importing data into Cloud SQL,
you can also use the LOAD DATA LOCAL INFILE statement in the mysql client, which loads a local file to the database
You can follow a sample steps in this link to start the mysql client by connecting to your Cloud SQL instance first through Cloud Shell. After connecting, you will be able to execute the import statement: LOAD DATA LOCAL INFILE.
The same steps can be applied for exporting and the syntax SELECT...INTO OUTFILE can be used as is. Together with other alternatives, this can be also found on Exporting data from Cloud SQL :
Exporting in CSV format is equivalent to running the following SQL statement:
SELECT <query> INTO OUTFILE ... CHARACTER SET 'utf8mb4'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
ESCAPED BY '\\' LINES TERMINATED BY '\n'
Once you have executed the operations, it is also advisable to check their status especially if it is long running or an error has occurred.

LOAD DATA LOCAL INFILE not working in RDS

I have wordpress website and have created plugin to import csv to a table. Database is in RDS. Here is the sql I have used
LOAD DATA LOCAL INFILE 'my.csv' INTO TABLE tablename CHARACTER SET UTF8 FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\r\n' IGNORE 1 LINES
(
ID,
Name
Address
)
When I run this sql in sqlyog in the same database ( database in RDS ) this works perfectly. Please note csv file used is on my windows folder and given absolute path. However when I run this sql from the plugin on the linux server (where the website is hosted), wordpress gives message saying Load data local infile is not supported. I have another website where this works fine and this is also hosted on AWS as this one and has the same configuration such as database in RDS and mysql version 5.5+ on both servers.
Am I missing anything here. Any help will be appreciated.
Thanks in advance.
Amazon RDS hosted service doesn't support the load from files. Hence it gives error.
Hence, you can't load the CSV.
Here is approach.
Convert your CSV data into insert into table(....) SQL data.
Load your data using command like below.
mysql -h <Host> -u <username> -p<Password> < Your_file.sql

How to import a csv file into MySQL workbench?

I have a CSV file. It contain 1.4 million rows of data, so I am not able to open that csv file in Excel because its limit is about 1 million rows.
Therefore, I want to import this file in MySQL workbench. This csv file contains columns like
"Service Area Code","Phone Numbers","Preferences","Opstype","Phone Type"
I am trying to create a table in MySQL workbench named as "dummy" containing columns like
ServiceAreaCodes,PhoneNumbers,Preferences,Opstyp,PhoneTyp.
The CSV file is named model.csv. My code in workbench is like this:
LOAD DATA LOCAL INFILE 'model.csv' INTO TABLE test.dummy FIELDS TERMINATED BY ',' lines terminated by '\n';
but I am getting an error like model.CSV file not found
I guess you're missing the ENCLOSED BY clause
LOAD DATA LOCAL INFILE '/path/to/your/csv/file/model.csv'
INTO TABLE test.dummy FIELDS TERMINATED BY ','
ENCLOSED BY '"' LINES TERMINATED BY '\n';
And specify the csv file full path
Load Data Infile - MySQL documentation
In case you have smaller data set, a way to achieve it by GUI is:
Open a query window
SELECT * FROM [table_name]
Select Import from the menu bar
Press Apply on the bottom right below the Result Grid
Reference:
http://www.youtube.com/watch?v=tnhJa_zYNVY
In the navigator under SCHEMAS, right click your schema/database and select "Table Data Import Wizard"
Works for mac too.
You can use MySQL Table Data Import Wizard
At the moment it is not possible to import a CSV (using MySQL Workbench) in all platforms, nor is advised if said file does not reside in the same host as the MySQL server host.
However, you can use mysqlimport.
Example:
mysqlimport --local --compress --user=username --password --host=hostname \
--fields-terminated-by=',' Acme sales.part_*
In this example mysqlimport is instructed to load all of the files named "sales" with an extension starting with "part_". This is a convenient way to load all of the files created in the "split" example. Use the --compress option to minimize network traffic. The --fields-terminated-by=',' option is used for CSV files and the --local option specifies that the incoming data is located on the client. Without the --local option, MySQL will look for the data on the database host, so always specify the --local option.
There is useful information on the subject in AWS RDS documentation.
If the server resides on a remote machine, make sure the file in in the remote machine and not in your local machine.
If the file is in the same machine where the mysql server is, make sure the mysql user has permissions to read/write the file, or copy teh file into the mysql schema directory:
In my case in ubuntu it was: /var/lib/mysql/db_myschema/myfile.csv
Also, not relative to this problem, but if you have problems with the new lines, use sublimeTEXT to change the line endings to WINDOWS format, save the file and retry.
It seems a little tricky since it really had bothered me for a long time.
You just need to open the table (right click the "Select Rows- Limit 10000") and you will open a new window. In this new window, you will find "import icon".
https://www.convertcsv.com/csv-to-sql.htm
This helped me a lot. You upload your excel (or .csv) file and it would give you an .sql file with SQL statements which you can execute - even in the terminal on Linux.