We update several tables of our database thanks to CSV files (exported from our ERP)
The problem is that these tables must be able to update several other tables
Let's take a fictitious example, when I make a LOAD DATA INFILE of the all_order table (containing all the commands), it must directly:
Update the KPI table order_kpi
Insert in the order_summary table, the number of orders recorded by commercial
I wanted to put a trigger on the insert of the table all_order, but I think that it executes the trigger for each line of the CSV (more than 15 000 lines ...) and makes me plant my local server
To note: It is a PHP script which manages the LOAD DATA INFILE (because I need it to be dynamic and that it corresponds to the needs of the user) and I do not wish to put my requests of insertion and update in this PHP script
So my question is:
How to create a trigger (or other mysql function) that runs after the LOAD DATA INFILE is complete ?
Related
I have created an empty table A in MySQL and inserted data from a local csv file, A.csv, using LOAD DATA LOCAL INFILE. Now I want to create more tables just like A and insert data from other different local csv files (they all have the same fields).
So for table B it will be something like:
CREATE TABLE B LIKE A;
LOAD DATA LOCAL INFILE 'mypath/B.csv'
...
I need to repeat this process for about 20 tables. How can I write a loop procedure to automate the process?
Any help on this would be much appreciated!
I know that the Load Data InFile SQL statement is not allowed to be executed when creating MySQL Events.
I am trying to find an alternative solution to this, but until now I cant.
https://dev.mysql.com/doc/mysql-reslimits-excerpt/5.6/en/stored-program-restrictions.html
Anyone has any idea if there is any other way that we can set to load external file into the tables on a scheduled basis?
Any reason you cant just do the Load Data into a temporary file of the expected input format? Add a few extra columns such as for pre-processing flags if that might help? Then that could be a single load without any other interactions of triggers and such. Then you could have a stored procedure run query / fetch / process / insert, etc. based on the already loaded records into a table. If you skip some, so be it, others process as normal? Just something I have done in the past and might be an option for you.
although I did not quite understand your question if you are on a Linux based system, you could create a simple bash script to load any external data into MySQL . Either dump it locally and load it or pull from external source and load it the same way you would load any file.
For example as below I am trying to import my data from a csv file to my customers table and I want to schedule it every 5 minutes.
Hence I include the load data infile under a mysql event. Code as below
However, it returns an error: ERROR Code 1314: LOAD DATA is not allowed in stored procedures.
Any alternative way that i can import a csv file data into a table and execute it during a schedule of every 5 minutes?
USE sales;
CREATE EVENT import_file_event
ON SCHEDULE EVERY 5 MINUTE
DO
LOAD DATA INFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/Importfile.csv'
INTO TABLE customers
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(#first_name,#last_name,#gender,#email_address,#number_of_complaints)
SET first_name = NULLIF(#first_name,''),
last_name = NULLIF(#last_name,''),
gender = NULLIF(#gender,''),
email_address = NULLIF(#email_address,'');
I have a series of CSV files that I had wanted to import into MySQL. To first populate the table I did the following;
mysql -u root -p apn -e "LOAD DATA LOCAL INFILE '/opt/cell.csv' INTO TABLE data FIELDS TERMINATED BY ',';"
Where the CSV contents as;
89xx,31xx,88xx,35xx,ACTIVATED,250.0MB,GPRS,96xx,0,0,2,false,DEFAULT
The one unique field is the first starting with '89xx' (which goes into column named 'iccid').
Now I want to do is update the table, but clueless how to use the first entry in the CSV to update the rest of the row? It will be like the 4th field that I need to get updated overtime as that is the value that will change (data usage for specific cellular device). I don't have that much of a problem emptying the table before doing a whole new import, I was thinking though it was a better practice to just update, I will eventually need update several times a day.
Since I have no practical skills in any language, or mysql for that matter, would it be best to just insert into a temp table and update from that?
You can use
REPLACE
before
INTO
keyword to update/replace your row.
LOCAL INFILE '/opt/cell.csv' REPLACE INTO TABLE data FIELDS TERMINATED BY ',';"
To ignore your duplicate index you can use
IGNORE
keyword before
INTO
Load data manual
I'm trying to extract information from 50 million HTML files into a MySQL database. My question is at what point during the process should I store the information into the MySQL database. For example, I'm considering these options:
Open each file and extract the information I need. Perform an INSERT after each file gets parsed.
Open each file and extract the information I need. Store the information into a CSV file as an intermediary. After all the files have been parsed into the CSV, perform a bulk upload using LOAD DATA INFILE
I know that LOAD DATA INFILE is much faster than individual INSERT statements if I already have the information in a CSV. However, if I don't have the information already in a CSV, I don't know if it's faster to create the CSV first.
At the crux of the question: Is writing to a local CSV faster or about the same as a single INSERT statement?
I'm using PHP in case it matters. Thanks in advance!
They key is not to do one insert per entry, but batch the entries in memory then perform a batch insert.
See: https://dev.mysql.com/doc/refman/5.7/en/insert.html
INSERT statements that use VALUES syntax can insert multiple rows. To do this, include multiple lists of column values, each enclosed within parentheses and separated by commas. Example:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
ORMs like SQLAlchemy or Hibernate are smart enough (depending on configuration) to automatically batch your inserts.
Suppose we have table with a DECIMAL column with values, for example: 128.98, 283.98, 21.20.
I want to import some CSV Files to this table. However, in the columns of these files, I have values like 235,69, 23,23, with comma instead of points.
I know I can REPLACE that column, but is there some way of doing that before LOAD INFILE?
I do not believe you can simultaneously replace that column and load the data. Looks like you will have to do multiple steps to get the results you want.
Load the data first into a raw table using the LOAD INFILE command. This table can be identical to the main table. You can use the Create Table like command to create the table.
Process the data (i.e. change the comma to a . where applicable) in the raw table.
select the data from the raw table and insert into main table either with row by row processing or bulk insert.
This can all be done in a stored procedure (SP) or by a 3rd party script written in python, php, etc...
If you want to know more about SP's in Mysql, Here is a useful link.