How to update large database in mysql database using cronJobs - mysql

I am very thank full if anyone help me to resolve this issue.
Detail: I have a mysql table where 110000+ record stored with Unique application(column) number.
Now there is Detail API which will take only 1 application number at a time in URI parameter and return details of that application number it will take Approx 1 min to respond for now.
Now i need to update that records(multiple column) using Cron Job Scheduling to always updated record in database.
and it will progressively update.
Flow: Get application number from database -> Call detail API -> update there record on database.
Issue: there is large number of record in database so we can not call API for all application number in ones.
I am using Laravel 7 and Guzzle HTTP client 7.2 HTTP Client for API calling.
Any suggestion are welcome !
Update: (Tested) I am thinking to do something like this and agree with #bhucho comment to call cron in every 15 minute
We create one more column in table for last_updated_id default 1
and we will write a query to fetch application number something like and get 100 or 500 records in one slab using Laravel take methods from here Database Queries
$latestIdInColumn = myTableClassRef::max(last_updated_id);
$applications = myTableClassRef::select('application_number)->where('id', '>', $latestIdInColumn)
->take(100)
->get()->toArray();
Here we call detail API and update for each application number, when there is last application update done we will store that id in last_updated_id.
When again cron call we have that last_updated_id so we will apply same filter in next query where id '>' $latestIdInColumn. now we will get next 100/500 records where id is greater then $latestIdInColumn.
if ( $applications )
{
For all other available id
} else {
when record not found.
Here we will reset the last_updated_id = 1
}
Now function will fetch from id 1.
It is not tested yet i just planing to do this, i am happy to have
feedback on this.
Note: Now API not taking 1Min to respond I resolved this issue.
Update 2: It is working good.

Related

Datastage incremental charging

We would like to perform incremental loading in DataStage (in parallel environement). Exactly load only the delta between the previous load and the new one (for create, update, delete the records in DWH).
We would like to store the last key recovered during the previous load to be able to restart the request from the next record on a new loading.
We have already successfully used a parameter to filter the SQL load query at runtime. Unfortunately, we have not yet found the possibility to retrieve the last key (max (Key) - Aggregator?) And to store it in this parameter.
Which stage to use, to output a single value in the same parallel job, and then store to parameter ?
Any ideas ?
Thanks for your help.
Think about getting the max value from your target - it is most probably a database and a max() is easy to do.
Check out my post about getting some data from the "stream" to a parameter
Thank's Michael,
I've found Head stage to get the max(LastRowId) in the same job, with 'All rows (after skip) = False), and 'Number of Rows (Per partition)=1. And I run the job in sequential mode...
That's worked fine.

SQL row data mismatch for different users

I am using MySQL (version 5.5.43) as my database.
I have a RoR micro-service that does an update column to a model Active Record class:
model.update_columns(status: 0)
The next line is an api call to different micro-service that synchronously does a SQL query:
select * from model where status = 0;
The code runs without any errors but the latter query is not fetching the record that is being updated by the former. There is milliseconds of difference between the update and the read.
Both the services are connected to the same database as different users but same access.
I don't understand why this would happen? the update_column is obviously a commit to the db, then why would the select query not fetch the updated record. What am I missing here?

Update remote database based on conditional statement

I have a MySQL database with a REST API for my main application hosted on Azure. I am setting up a hardware sensor with an additional database that will capture data multiple times a second. When a value changes by a specific threshold of the current value or after a specific time interval I want to make an API call to update the main database.
ie) Threshold is 10%. Last value was 10 this value is 12; this will set a trigger to call API and add to main database.
Can a trigger be added to the second database to make a HTTP request? Is there benefit to using another RDBMS in this case instead of MySQL? Does PubNub/Firebase make sense in this situation?

How to do updates on all rows at every button click

I have a python app that has an admin dashboard.
There I have a button called "Update DB".
(The app uses MySQL and SQLAlchemy)
Once it's clicked it makes an API call and gets a list of data and writes that to the DB, and if there are new records returned by the API call it adds them and does not duplicate currently existing records.
However if API call returns less items, it does not delete them.
Since I don't even have a "starting to google" point I need some guidance of what type of SQL query should my app be making.
Like once button is clicked ,it needs to go through all the rows:
do the changes to the updated records that existed
add new ones if there are any returned by the API call
delete ones that API call did not return.
What is this operation called or how can I accomplish this in mysql?
Once I find out about this I'll see how can I do that in SQLAlchemy.
You may want to set a timestamp column to the time of latest action on the table and have a background thread remove old rows as another action. I don't know of any atomic action that may perform the desired data reformation. Another option might be satisfactory is to write the replacement batch to a staging table, rename both versions (swap) and drop the old table. HTH

Simple Update Function MySQL

I have a CMS coded in ColdFusion 9 with basic login functions to work with a socket server that I'm working with. Recently I setup a "VIP" subscription w/payment, but then I realized, I have no SQL to remove the subscription after 2 months have passed. Here's my basic request; I need some SQL to execute automatically if the timestamp value of date_purchased is 2 months old. It needs to delete the row in vip_purchases for the session user (Only the user), and update the column vip to 0 in users.
I need this to execute for the session username ONLY, and only if the timestamp is 2 weeks old, and ONLY if the value in VIP is 1.
I was thinking along the lines of:
update users set vip = 0 where username = session.username if vip = 1 if date_purchased = 2 months old
I realize this isn't SQL, that was just to attempt to make it a little more clear.
Close. :)
UPDATE users
SET vip = 0
WHERE username = sessionname
AND vip = 1
AND date_purchased <= (today - number of days wanted)
You'll need to replace the last part of the WHERE (the today - part) with whatever timeframe condition you want; your post is confusing, as it mentions 2 weeks old and 2 months old in the same context.
If you plan to run this reoccurring task via Coldfusion (as opposed to internal to the MySQL database), you should setup a Scheduled Task via the Coldfusion Administrator. The task could launch a script once daily to clean up all of the old accounts.
Once that is decided, you can create a .cfm or .cfc file that includes the task.
This can be a simple file that includes a cfquery or a cfstoredproc depending on where you want your logic to run. Cfquery will run the logic on the web server in Coldfusion, the cfstoredproc will ask the database server to run MySQL logic on the database server. It would likely be best to run it as a stored procedure.
In this case, you would not need to even check the username. Just SELECT then loop over and UPDATE all users where vip = 1 and the date_purchased has expired.