I am using MySQL (version 5.5.43) as my database.
I have a RoR micro-service that does an update column to a model Active Record class:
model.update_columns(status: 0)
The next line is an api call to different micro-service that synchronously does a SQL query:
select * from model where status = 0;
The code runs without any errors but the latter query is not fetching the record that is being updated by the former. There is milliseconds of difference between the update and the read.
Both the services are connected to the same database as different users but same access.
I don't understand why this would happen? the update_column is obviously a commit to the db, then why would the select query not fetch the updated record. What am I missing here?
Related
I am very thank full if anyone help me to resolve this issue.
Detail: I have a mysql table where 110000+ record stored with Unique application(column) number.
Now there is Detail API which will take only 1 application number at a time in URI parameter and return details of that application number it will take Approx 1 min to respond for now.
Now i need to update that records(multiple column) using Cron Job Scheduling to always updated record in database.
and it will progressively update.
Flow: Get application number from database -> Call detail API -> update there record on database.
Issue: there is large number of record in database so we can not call API for all application number in ones.
I am using Laravel 7 and Guzzle HTTP client 7.2 HTTP Client for API calling.
Any suggestion are welcome !
Update: (Tested) I am thinking to do something like this and agree with #bhucho comment to call cron in every 15 minute
We create one more column in table for last_updated_id default 1
and we will write a query to fetch application number something like and get 100 or 500 records in one slab using Laravel take methods from here Database Queries
$latestIdInColumn = myTableClassRef::max(last_updated_id);
$applications = myTableClassRef::select('application_number)->where('id', '>', $latestIdInColumn)
->take(100)
->get()->toArray();
Here we call detail API and update for each application number, when there is last application update done we will store that id in last_updated_id.
When again cron call we have that last_updated_id so we will apply same filter in next query where id '>' $latestIdInColumn. now we will get next 100/500 records where id is greater then $latestIdInColumn.
if ( $applications )
{
For all other available id
} else {
when record not found.
Here we will reset the last_updated_id = 1
}
Now function will fetch from id 1.
It is not tested yet i just planing to do this, i am happy to have
feedback on this.
Note: Now API not taking 1Min to respond I resolved this issue.
Update 2: It is working good.
I am developing high load web-application and trying to reduce the quantity of sql queries. Quite often I need to update one row and get the results. I think it would be nice to have possibility to run query and at the same time receive the values of updated fields without making 2 calls of mysql server. For example, I execute the following query:
update table set val=val+1 where id=1;
and function returns:
array("val"=>10)
Sure, I understand, that I can write my own function, which first makes update, than select and returns the result. But the problem is that in such case mysql server will have to seek data, update during first query, than comes the second query, which again requires to seek data and return it. And I am thinking about the way, where mysql seeks data, updates and returns updated data.
I'm getting data from an MSSQL DB ("A") and inserting into a MySQL DB ("B") using the date created in the MSSQL DB. I'm doing it with simple logics, but there's got to be a faster and more efficient way of doing this. Below is the sequence of logics involved:
Create one connection for MSSQL DB and one connection for MySQL DB.
Grab all of data from A that meet the date range criterion provided.
Check to see which of the data obtained are not present in B.
Insert these new data into B.
As you can imagine, step 2 is basically a loop, which can easily max out the time limit on the server, and I feel like there must be a way of doing this must faster and during when the first query is made. Can anyone point me to right direction to achieve this? Can you make "one" connection to both of the DBs and do something like below?
SELECT * FROM A.some_table_in_A.some_column WHERE
"it doesn't exist in" B.some_table_in_B.some_column
A linked server might suit this
A linked server allows for access to distributed, heterogeneous
queries against OLE DB data sources. After a linked server is created,
distributed queries can be run against this server, and queries can
join tables from more than one data source. If the linked server is
defined as an instance of SQL Server, remote stored procedures can be
executed.
Check out this HOWTO as well
If I understand your question right, you're just trying to move things in the MSSQL DB into the MySQL DB. I'm also assuming there is some sort of filter criteria you're using to do the migration. If this is correct, you might try using a stored procedure in MSSQL that can do the querying of the MySQL database with a distributed query. You can then use that stored procedure to do the loops or checks on the database side and the front end server will only need to make one connection.
If the MySQL database has a primary key defined, you can at least skip step 3 ("Check to see which of the data obtained are not present in B"). Use INSERT IGNORE INTO... and it will attempt to insert all the records, silently skipping over ones where a record with the primary key already exists.
This question is basically an extension of this question:
How can I tell if a database table is being accessed anymore? Want something like a "SELECT trigger"
In the query the author provided as a solution in his post, I assume a null value would mean that a table hasn't been accessed since the last time SQL Server was restarted.
My question is: Is there a way to tell when a table was last accessed, if the last access was before the last time SQL restarted? Also, how can I tell when the last time SQL was restarted?
For sql server start time in SQL Server 2008,
select sqlserver_start_time from sys.dm_os_sys_info
For last user access since server restart,
select DB_NAME(us.[database_id]) as [db], OBJECT_NAME(us.[object_id], us.[database_id]) as [object],
MAX(us.[last_user_lookup]) as [last_user_lookup], MAX(us.[last_user_scan]) as [last_user_scan], MAX(us.[last_user_seek]) as [last_user_seek]
from sys.dm_db_index_usage_stats as us
where us.[database_id] = DB_ID() AND us.[object_id] = OBJECT_ID('tblname')
group by us.[database_id], us.[object_id];
I log this table daily so I have it after restarts. It can also act as an index audit after they have been dropped.
So here is my situation: I have a vendor supplied DB we cannot modify and a custom db that imports data from the vendor app and acts on it. Once records are imported form the vendor app, they cannot appear on the list of records to be imported. Also we only want to display the 250 most recent records that have not been imported.
What I originally started with was select the list of ids that have been imported from the custom db, and then query the vendor db, using the list of ids in a .Where(x => !idList.Contains(x.Id)) clause on the remote query.
This worked up until we broke 2100 records imported into the custom db, as 2100 is the limit on the number of parameters that can be passed into SQL. After finding out this was the actual problem and not the 'invalid buffer'/'severe error' ADO.Net reported, my solution was to remove the first 2000 ids in the remote query, and then remove the remaining records in the local query.
Having to pull back a large number of irrelevant records, just to exclude them, so I can get the correct 250 records seems very inelegant. Is there a better way to do this, short of doing a cross db stored procedure?
Thanks in advance.
This might not be the best answer, depending on how many records you're dealing with, but you could force the SQL to execute and just deal with it as in-memory objects. Calling the ToList() method will execute the SQL and convert to an IEnumerable .
What I might suggest is to have started by querying the vendor database first ordering the results by some kind of criteria (perhaps a date field, oldest to most recent).
You could do a Skip().Take() to "skim" the results and then take each bulk set and insert them into the custom db where the ID doesn't already exist. That way you avoid the problem you have now.
If you have db-create access to the SQL Server that the vendor's db is running on (or if your custom db is on the same server), you could create a "has been imported" table in a different database on that same server, and then write a stored proc that does a cross-database join of that table against the vendor db, e.g.:
select top 250 from vendordb.to_be_imported
where not exists
(select 1 from customdb.has_been_imported where idWasImported = idToBeImported)
order by whatever;
You might even be able to do this in Linq 2 SQL -- I've never tried adding objects from different databases into a single DataContext...