I'm trying to insert and update some data on MySql using PySpark SQL DataFrames and JDBC connection.
I've succeeded to insert new data using the SaveMode.Append. Is there a way to update the existing data and insert new data in MySql Table from PySpark SQL?
My code to insert is:
myDataFrame.write.mode(SaveMode.Append).jdbc(JDBCurl,mySqlTable,connectionProperties)
If I change to SaveMode.Overwrite it deletes the full table and creates a new one, I'm looking for something like the "ON DUPLICATE KEY UPDATE" available in MySql
Any help on this is highly appreciated.
Create a view in Mysql as create view <viewName> as select ...<tableName>
Create trigger in mysql to update after insert using -
CREATE TRIGGER trigger_name
AFTER INSERT
ON <viewName> FOR EACH ROW
BEGIN
-- statements
-- INSERT ... ON DUPLICATE KEY UPDATE Statement
END$$
ref - https://www.mysqltutorial.org/mysql-triggers/mysql-after-insert-trigger/
Write data to view <viewName> from spark
Related
Is it possible to insert data into an existing table in postgres database using apache drill.something similar to
insert into Post_db.test_schema.customer_account_holder_test select customer_id,source_system_id,salutation,first_name,middle_name,last_name,legal_name,gender,identity_proof_name,identity_proof_value from hive.schema_name.customer_account_holder limit 10
In apache drill, Insert is not supported.
You can creates a new table and populates the new table with rows returned from a SELECT query. Use the CREATE TABLE AS (CTAS) statement in place of INSERT INTO. CREATE TABLE name AS query;
I have the following sql syntax:
CREATE TRIGGER trigger_name
BEFORE INSERT
ON table FOR EACH ROW
BEGIN
END;
I need to check the table for values where field1='x' and field2='y'. If the values are found want the insertion to fail via php side. ($mysqli->error)
This can be done by attempting to select that value using mysql and count the results (num_rows). If the result returns 0, then you insert data into mysql table, as there is no data with the same value in your table.
I'm using Mysql in phpMyAdmin,where i have to delete a entry from tableA if i insert a row with same primary key.I thought to do it in trigger of tableA BEFORE INSERT
For Ex,
if the tableA contains
1 Hai Hello
here 1 is the primary key
And now if i insert a row 1 Bye Hello then the trigger BEFORE INSERT will delete the old entry and then the new row (2nd) will be inserted. But Mysql has restriction of not being able to update a table inside a trigger defined for that same table.
It gives the error
#1442 - Can't update table 'tableA' in stored
function/trigger because it is already used by statement which invoked
this stored function/trigger.
So i changed my way, i called a procedure from trigger BEFORE INSERT of tableA and in that procedure i do the task what i thought to do in trigger. But unfortunately i'm getting the same error.
In trigger BEFORE INSERT i simply called the procedure as
CALL proce1(new.Reg_No);
In procedure i have done this
DECLARE toup integer;
select count(*) into toup from tableA where Reg_No=reg;/*Here Reg_No is primary key */
if toup > 0 then
delete from tableA where Reg_No=reg;
end if;
Need some other Idea to achieve this. Help Me.....
I don't like to use triggers so much because they are hard to manage somethimes. Also they will cause you a downgrade in performance. I am not against trigger as they can be handy in some cases.
In your case have you though of using REPLACE(....) or INSERT INTO .... ON DUPLICATE KEY UPDATE or even INSERT INTO IGNORE .....
REPLACE(....) will delete a record if a record is found and insert a new one with the same ID.
INSERT INTO .... ON DUPLICATE KEY UPDATE will allow you to override existing field if a duplicate is found.
INSERT INTO IGNORE ..... will allow you to ignore the new inserted row if one already exists
Since you mentioned in the comments that you are importing records from a file, then try to use LOAD DATA INFILE logic which will allow you to REPLACE field on duplicate
LOAD DATA LOCAL INFILE 'x3export.txt'
REPLACE INTO TABLE x3export
I want to see if can I use mysql to manage a little stock.
table A contain all the movements:
art_code, qty_load, qty_unload, date
table B with the existence:
art_code, total_load, total_unload, available, date
I've created a trigger: (after update on)
INSERT INTO STOCK VALUES(NEW.ART_CODE, TOTAL_LOAD, TOTAL_UNLOAD, TOTAL_LOAD-TOTAL_UNLOAD, NOW());
but after the first correct run it says a row already exists, how could I replace the old row with the new row?
You can use INSERT ... ON DUPLICATE KEY UPDATE command. It will help you to insert and to update existed records using one statement.
INSERT ... ON DUPLICATE KEY UPDATE Syntax
Using MySQL and PHP, I have two tables ResponseTable and EntryTable
Response table has
RespID(pk,uq,ai, int), Response(string)
Entry table has
EntryID(pk,uq,ai,int), RespID(int), UserID(int)
I would like to insert a response into the ResponseTable where the Response doesn't exist, and then insert an entry into the EntryTable based on the RespID from the ResponseTable corresponding to the response.
How can this be done with the fewest statements?
EDIT:
Response is unique
The fewest statements from the front-end would be to use a stored procedure. Any way you do it, though, you will still need to have two INSERT statements. You just can't insert two different things with one query.
mysql supports TRIGGERS . you can add one to your ResponseTable that will activate "AFTER" "INSERT" and you can get it to use the values that are being inserted using the NEW key word.
the body of the trigger can be an insert into entry
something like:
CREATE TRIGGER `response_after_insert` AFTER INSERT ON `response`
FOR EACH ROW
BEGIN INSERT INTO entry SET RespID=NEW.RespID ON DUPLICATE KEY UPDATE operation=1;END;
So once you have your trigger set up, any time you do an insert on respnse, the trigger will get activated