I am requesting data from a service whose response in stored in a database.First, I have an empty table, whenever I make my very first request the records from the service comes to my database table.
from now, whenever I make second request, the service will provide me some records which may be same as my first response, may be new records, may be updated records etc.
my query is to how to update my table with respect to the responses coming from the service during my second request on-wards? so that Unchanged records will remain same, New records will be added, updated records will be updated.Do I need to write any stored procedure on my DB or any workaround ?what might be the scenario if I use Nomysql DB's like mongo DB ?
Thanks In Advance.
Locate whichever subset of the service response identifies a record; this should be set as the primary key in your database table, or else a secondary key over which a uniqueness constraint is enforced.
In MySQL, use INSERT ... ON DUPLICATE KEY UPDATE.
Related
I have a table (MySql) that some rows need to be updated when a user desires.
i know the right way is just using Sql UPDATE statement and i don't speak about 'Which is faster? Delete and insert or just update!'. but as my table update operation needs more time to write a code (cause of table's relations) why i don't delete the old row and insert updated field?
Yes, you can delete and insert. but what keeps the record in your database if the program crash a moment before it can insert data to Database?
Update keeps this from happening. It keeps the data in your database and change the value that needed to be changed. Maybe it is complicated to use in your database, but you can certain that your record still safe.
finally i get the answer!
in a RDBMS system there are relations between records and one record might have some dependencies. in such situations you cannot delete and insert new record because foreign key constraint cause data lose. records dependent (ie user posts) to main record (ie an user record) will be deleted!
if there are situations that you don't have records dependencies (not as exceptions! but in data models nature) (like no-sql) and you have some problems in updating a record (ie file checking) you can use this approach.
I have a query that runs everytime a user logins. Since this query regards information the user might have third-party updated recently I thought it would be a good idea to turn the user_id + information combo in the table unique. As so, everytime a user tried to save new information it would only save the one information I already didn't have. So, the first query being
INSERT INTO table VALUES ("1","cake"),("1","pie"),("1","bedsheets")
And as the user logins a second time and it being
INSERT INTO table VALUES ("1","cake"),("1","pie"),("1","bedsheets"),("1","chocolate")
It would only save ("1","chocolate") because (id,info) being an unique pair all other would not be inserted. I came upon the realization they all fail if only one fails. So my question is: is there any way to override this operation? Or do I have to query the db first to filter the information I already have? tyvm...
When you use the IGNORE Keyword, so every errors, in the execution are ignored. Example: if you have a duplicate or PRIMARY key error while executing a INSERT Statement, so it will ignored and the execution is not aborted
Use this:
I NSERT IGNORE INTO table VALUES ("1","cake"),("1","pie"),("1","bedsheets"),("1","chocolate");
I have an app that has to import TONS of data from a remote source. From 500 to 1500 entries per call.
Sometimes some of the data coming in will need to replace data already stored in the dB. If I had to guess, I would say once in 300 or 400 entries would one need to be replaced.
Each incoming entry has a unique ID. So I am trying to figure out if it is more efficient to always issue a delete command based on this ID or to check if there is already an entry THEN delete.
I found this SO post where it talks about the heavy work a dB has to do to delete something. But it is discussing a different issue so I'm not sure if it applies here.
Each incoming entry has a unique ID. So I am trying to figure out if it is more efficient to always issue a delete command based on this ID or to check if there is already an entry THEN delete.
Neither. Use INSERT ... ON DUPLICATE KEY UPDATE ....
Since you are using MySQL and you have a unique key then let MySQL do the work.
You can use
INSERT INTO..... ON DUPLICATE KEY UPDATE......
MySQL will try to insert a new record in the table, is the unique value exists in the table then MySQL will update all the field that you have set after the update
You can read more about the INSERT INTO..... ON DUPLICATE KEY UPDATE...... syntax on
http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html
I'm building a system that updates its local database from other APIs frequently. I have Python-scripts set as cron jobs, and they do the job almost fine.
However, the one flaw is, that the scripts take ages to perform. When they are ran for the first time, the process is quick, but after that it takes nearly 20 minutes to go through a list of 200k+ items received from the third-party API.
The problem is that the script first gets all the rows from the database and adds their must-be-unique column value to a list. Then, when going through the API results, it checks if the current items must-be-unique value exists in the list. This gets really heavy, as the list has over 200k values in it.
Is there a way to check in an INSERT-query that, based on a single column, there is no duplicate? If there is, simply not add the new row.
Any help will be appreciated =)
If you add a UNIQUE key to the column(s) that have to contain UNIQUE values, MySQL will complain when you insert a row that violates this constraint.
You then have three options:
INSERT IGNORE will try to insert, and in case of violation, do nothing.
INSERT ... ON DUPLICATE KEY UPDATE will try to insert, and in case of violation, update the row to the new values
REPLACE will try to insert, and in case of violation, DELETE the offending existing row, and INSERT the new one.
Finally reached data migration part of my Project and now trying to move data from MySQL to SQL Server.
SQL Server has new schema (mapping is not always one to one).
I am trying to use SSIS for the conversion, which I started learning today morning.
We have customer and customer location table in MySQL and equivalent table in SQL Server. In SQL server all my tables now have surrogate key column (GUID) and I am creating the same in Script Component.
Also note that I do have a primary key in current mysql tables.
What I am looking for is how I can add child records to customer location table with newly created guid as parent key.
I see that SSIS have Foreach loop container, is this of any use here.
if not another possibility that I can think of is create two Data Flow Task and [somehow] just before the master data is sent to Destination Component [Table] on primary dataflow task , add a variable with newly created GUID and another with old PrimaryID, which will be used to create source for DataTask Flow for child records.
May be to simplyfy , this can also be done once datatask for master is complete and then datatask for child reads this master data and inserts child records from MySQL to SQL Server table. This would though mean that I have to load all my parent table records back into memory.
I know this is all too confusing and it is mainly because I am very confused :-(, to bear with me and if you want more information let me know.
I have been through may links that i found through google search but none of them really explains( or I was not able to uderstand) how the process is carried out.
Please advise
regards,
Mar
** Edit 1**
after further searching and refining key words i found this link in SO and going through it to see if it can be used in my scenario
How to load parent child data found in EDI 823 lockbox file using SSIS?
OK here is what I would do. Put the my sql data into staging tables in sql server that have identity columns set up and an extra column for the eventual GUID which will start out as null. Now your records have a primary key.
Next comes the sneaky trick. Pick a required field (we use last_name) and instead of the real data insert the value form the id field in the staging table. Now you havea record that has both the guid and the id in it. Update the guid field in the staging table by joing to it on the ID and the required field you picked out. Now update the last_name field with the real data.
To avoid the sneaky trick and if this is only a onetime upload, add a column to your tables that contains the staging table id. Again you can use this to get the guid for inserting to related tables. Then when you are done, drop the extra column.
You are aware that there are performance issues involved with using GUIDs? Make sure not to make them the clustered index (as the PK they will be by default unless you specify differntly) and use newsequentialid() to populate them. Why are you using GUIDs? If an identity would work, it is usually better to use it.