I have a method which is called frequently from multiple threads. It involves writing to disk using await FileIO.WriteTextAsync. This works fine when it is called from a single thread, but once I start doing this in multiple threads, I get this error:
The process cannot access the file because it is being used by another process.
I know what the error means, but I'm not sure how to work around it. Normally, I would create a lock(object) statement, to ensure that the file is being accessed by only one thread at a time. However, this is an asynchronous method and as such I can't use the await operator in the body of the lock(object) statement.
Please advise on how to handle this scenario.
You can use SemaphoreSlim to act as an async-compatible lock:
SemaphoreSlim _mutex = new SemaphoreSlim(1);
async Task MyMethodAsync()
{
await _mutex.WaitAsync();
try
{
...
}
finally
{
_mutex.Release();
}
}
Personally, I don't like the finally, so I usually write my own IDisposable to release the mutex when disposed, and my code can look like this:
async Task MyMethodAsync()
{
// LockAsync is an extension method returning my custom IDisposable
using (await _mutex.LockAsync())
{
...
}
}
Related
I'm struggeling a bit with how to shoot several SQL update queries to the MySQL server from NodeJS. I need some kind of synchronous execution.
Situation:
I have an array of objects. (Lets say "formValues")
Some of these objects might have a corresponding row in a database table, others don't.
I need to do a MySQL query for each object in the array to know for which object a new row in MySQL needs to be created and which one only needs to be updated.
at the end I need to update / create the rows in MySQL table.
I need kind of one callback for the whole process.
This is more or less a general question on how to solve situations like this with NodeJS. As I understood, MySQL queries are executed asynchronously.
How can it be achieved, that I can loop through the array to build a list of entries which need to be updated an other to be created in MySQL table?
Is there any "synchronous" MySQL query?
Regards
Jens
You can probably really benefit from switching to an async/await MySQL client. Chances are that the client you are using already has support for this.
But even if yours doesn't, and you can't switch it's still by far the easiest to write a helper function that converts your callback-based mysql to a promise-based one.
Effectively the approach becomes:
for(const formValue of formValues) {
await mysql.query('....');
}
Doing this without async/await and with callbacks is significantly harder.
I imagine one approach might be something like:
function doQueriesForFormValue(formValues, cb) {
const queries = [];
for(const formvalue of formValues) {
queries.push('...');
}
runQueries(queries, cb);
}
function runQueries(queries, cb) {
// Grab the first query
const currentQuery = queries[0];
mysql.query(currentQuery, (res, err) {
if (err) {
cb(null, err);
}
if (currentQueries.length>1) {
// Do the next query
runQueries(currentQueries.slice(1), cb);
} else {
// call final callback
cb(true);
}
});
}
I wrote an article with some effective patterns for working with MySQL in Node.js. Perhaps it's helpful
I have a Spring application which updates particular entity details in MySQL DB using a #Transactional method, And within the same method, I am trying to call another endpoint using #Async which is one more Spring application which reads the same entity from MySql DB and updates the value in redis storage.
Now the problem is, every time I update some value for the entity, sometimes its updated in redis and sometimes it's not.
When I tried to debug I found that sometimes the second application when it reads the entity from MySql is picking the old value instead of updated value.
Can anyone suggest me what can be done to avoid this and make sure that second application always picks the updated value of that entity from Mysql?
The answer from M. Deinum is good but there is still another way to achieve this which may be simpler for you case, depending on the state of your current application.
You could simply wrap the call to the async method in an event that will be processed after your current transaction commits so you will read the updated entity from the db correctly every time.
Is quite simple to do this, let me show you:
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionSynchronization;
import org.springframework.transaction.support.TransactionSynchronizationManager;
#Transactional
public void doSomething() {
// application code here
// this code will still execute async - but only after the
// outer transaction that surrounds this lambda is completed.
executeAfterTransactionCommits(() -> theOtherServiceWithAsyncMethod.doIt());
// more business logic here in the same transaction
}
private void executeAfterTransactionCommits(Runnable task) {
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
public void afterCommit() {
task.run();
}
});
}
Basically what happens here is that we supply an implementation for the current transaction callback and we only override the afterCommit method - there are others methods there that might be useful, check them out. And to avoid typing the same boilerplate code if you want to use this in other parts or simply make the method more readable I extracted that in a helper method.
The solution is not that hard, apparently you want to trigger and update after the data has been written to the database. The #Transactional only commits after the method finished executing. If another #Async method is called at the end of the method, depending on the duration of the commit (or the actual REST call) the transaction might have committed or not.
As something outside of your transaction can only see committed data it might see the updated one (if already committed) or still the old one. This also depends on the serialization level of your transaction but you generally don't want to use an exclusive lock on the database for performance reason.
To fix this the #Async method should not be called from inside the #Transactional but right after it. That way the data is always committed and the other service will see the updated data.
#Service
public class WrapperService {
private final TransactionalEntityService service1;
private final AsyncService service2;
public WrapperService(TransactionalEntityService service1, AsyncService service2) {
this.service1=service1;
this.service2=service2;
}
public updateAndSyncEntity(Entity entity) {
service1.update(entity); // Update in DB first
service2.sync(entity); // After commit trigger a sync with remote system
}
}
This service is non-transactional and as such the service1.update which, presumable, is #Transactional will update the database. When that is done you can trigger the external sync.
So I'm currently fetching some remote JSON data through an async call. By using a completion handler I'm able to use the data outside the call.
But now I'm wondering how I could process this data the right way.
For example I'm having different functions that rely on this data. Every time I need the data I could call it inside these functions. E.g.:
func generateRandomItem() {
DataManager().requestData() { (result: [Model]) in
//Generate Random item from the results of call here
}
}
func listAlphabetically() {
DataManager().requestData() { (result: [Model]) in
//List all data alphabetically from the results of the call here
}
}
But this approach will be wasting API calls I think since the data will only change once a week.
Another approach I was thinking about, was calling this function on launch of the application and store everything in a global var. This var can then be used in every single function whenever I need it. E.g.:
var allItems = [Model]()
func getAllItems() {
DataManager().requestData() { (result: [Model]) in
self.allItems = result
}
}
The disadvantage of this is that the data is only pulled in once, and never updated but only on app launch.
So this isn't much of a technical question but more a fundamental one. I'm pretty new to Swift and looking for the best design patterns. If my completion handler on the async call is also a wrong approach I would like to know how I could improve it!
I need to update or create data in a mysql table from a large array (few 1000s objects) with sequelize.
When I run the following code it uses up almost all my cpu power of my db server (vserver 2gb ram / 2cpu) and clogs my app for a few minutes until it's done.
Is there a better way to do this with sequelize? Can this be done in the background somehow or as a bulk operation so it doesn't effect my apps performance?
data.forEach(function(item) {
var query = {
'itemId': item.id,
'networkId': item.networkId
};
db.model.findOne({
where: query
}).then(function(storedItem) {
try {
if(!!storedItem) {
storedItem.update(item);
} else {
db.model.create(item);
}
} catch(e) {
console.log(e);
}
});
});
Your first line of your sample code data.forEach()... makes a whole mess of calls to your function(item){}. Your code in that function fires off, in turn, a whole mess of asynchronously completing operations.
Try using the async package https://caolan.github.io/async/docs.htm and doing this
async = require('async');
...
async.mapSeries(data, function(item){...
It should allow each iteration of your function (which iterates once per item in your data array) to complete before starting the next one. Paradoxically enough, doing them one at a time will probably make them finish faster. It will certainly avoid soaking up your resources.
Weeks later I found the actual reason for this. (And unfortunately using async didn't really help after all) It was as simple as stupid: I didn't have an MYSQL index for itemId so with every iteration the whole table was queried which caused the high CPU load (obviously).
What is the purpose of the org.apache.hadoop.mapreduce.Mapper.run() function in Hadoop? The setup() is called before calling the map() and the clean() is called after the map(). The documentation for the run() says
Expert users can override this method for more complete control over the execution of the Mapper.
I am looking for the practical purpose of this function.
The default run() method simply takes each key / value pair supplied by the context and calls the map() method:
public void run(Context context) throws IOException, InterruptedException {
setup(context);
while (context.nextKeyValue()) {
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
cleanup(context);
}
If you wanted to do more than that ... you'd need to override it. For example, the MultithreadedMapper class
I just came up with a fairly odd case for using this.
Occasionally I've found that I want a mapper that consumes all its input before producing any output. I've done it in the past by performing the record writes in my cleanup function. My map function doesn't actually output any records, it just reads the input and stores whatever will be needed in private structures.
It turns out that this approach works fine unless you're producing a LOT of output. The best I can make out is that the mapper's spill facility doesn't operate during cleanup. So the records that are produced just keep accumulating in memory, and if there are too many of them you risk heap exhaustion. This is my speculation of what's going on - could be wrong. But definitely the problem goes away with my new approach.
That new approach is to override run() instead of cleanup(). My only change to the default run() is that after the last record has been delivered to map(), I call map() once more with null key and value. That's a signal to my map() function to go ahead and produce its output. In this case, with the spill facility still operable, memory usage stays in check.
Maybe it could be used for debugging purposes as well. You can then skip part of the input key-value pairs (=take a sample) to test your code.