using Async inside a transaction in Spring application - mysql

I have a Spring application which updates particular entity details in MySQL DB using a #Transactional method, And within the same method, I am trying to call another endpoint using #Async which is one more Spring application which reads the same entity from MySql DB and updates the value in redis storage.
Now the problem is, every time I update some value for the entity, sometimes its updated in redis and sometimes it's not.
When I tried to debug I found that sometimes the second application when it reads the entity from MySql is picking the old value instead of updated value.
Can anyone suggest me what can be done to avoid this and make sure that second application always picks the updated value of that entity from Mysql?

The answer from M. Deinum is good but there is still another way to achieve this which may be simpler for you case, depending on the state of your current application.
You could simply wrap the call to the async method in an event that will be processed after your current transaction commits so you will read the updated entity from the db correctly every time.
Is quite simple to do this, let me show you:
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionSynchronization;
import org.springframework.transaction.support.TransactionSynchronizationManager;
#Transactional
public void doSomething() {
// application code here
// this code will still execute async - but only after the
// outer transaction that surrounds this lambda is completed.
executeAfterTransactionCommits(() -> theOtherServiceWithAsyncMethod.doIt());
// more business logic here in the same transaction
}
private void executeAfterTransactionCommits(Runnable task) {
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
public void afterCommit() {
task.run();
}
});
}
Basically what happens here is that we supply an implementation for the current transaction callback and we only override the afterCommit method - there are others methods there that might be useful, check them out. And to avoid typing the same boilerplate code if you want to use this in other parts or simply make the method more readable I extracted that in a helper method.

The solution is not that hard, apparently you want to trigger and update after the data has been written to the database. The #Transactional only commits after the method finished executing. If another #Async method is called at the end of the method, depending on the duration of the commit (or the actual REST call) the transaction might have committed or not.
As something outside of your transaction can only see committed data it might see the updated one (if already committed) or still the old one. This also depends on the serialization level of your transaction but you generally don't want to use an exclusive lock on the database for performance reason.
To fix this the #Async method should not be called from inside the #Transactional but right after it. That way the data is always committed and the other service will see the updated data.
#Service
public class WrapperService {
private final TransactionalEntityService service1;
private final AsyncService service2;
public WrapperService(TransactionalEntityService service1, AsyncService service2) {
this.service1=service1;
this.service2=service2;
}
public updateAndSyncEntity(Entity entity) {
service1.update(entity); // Update in DB first
service2.sync(entity); // After commit trigger a sync with remote system
}
}
This service is non-transactional and as such the service1.update which, presumable, is #Transactional will update the database. When that is done you can trigger the external sync.

Related

Spring Data JPA - Pessimistic Locking Not Working

Using: Spring Boot 2.3.3, MySQL 5.7(currently via TestContainers), JUnit 5
I have a JpaRepository inside a Spring MVC application that has a method set to be #Lock(LockModeType.PESSIMISTIC_WRITE) and, while I do see the SELECT ... FOR UPDATE coming up in the resulting SQL, it doesn't seem to do much of anything.
I'll put the code below, but, if I try to spin up multiple threads that make the same call, each thread is able to read the same initial value in question and nothing ever seems to block/wait. And my understanding is that any "additionally" called methods that are also #Transactional (from the org.springframework.transaction namespace) are made part of the original transaction.
I can't figure out what I'm doing wrong. Any assistance would be appreciated, even if it means pointing out that my understanding/expectations are flawed.
Repository
public interface AccountDao extends JpaRepository<Account, Long> {
#Lock(LockModeType.PESSIMISTIC_WRITE)
public Optional<Account> findById(Long id);
}
Services
Account Service
#Service
public class AccountServiceImpl implements AccountService {
#Autowired
private FeeService feeService;
#Override
#Transactional // have also tried this with REQUIRES_NEW, but the same results occur
public void doTransfer(Long senderId, Long recipientId, TransferDto dto) {
// do some unrelated stuff
this.feeService.processFees(recipientId);
}
}
Fee Service
#Service
public class FeeServiceImpl implements FeeService {
#Autowired
private AccountDao accountDao;
#Override
#Transactional // have also tried removing this
public void processFees(Long recipientId) {
// this next line is actually done through another service with a #Transactional annotation, but even without that annotation it still doesn't work
Account systemAccount = this.accountDao.findById(recipientId);
System.out.println("System account value: " + systemAccount.getFunds());
systemAccount.addToFunds(5);
System.out.println("Saving system account value: " + systemAccount.getFunds());
}
}
Test
public class TheTest {
// starts a #SpringBootTest with ```webEnvironment = WebEnvironment.RANDOM_PORT``` so it should start up a dedicated servlet container
// also auto configures a WebTestClient
#Test
#Transactional
public void testLocking() {
// inserts a bunch of records to have some users and accounts to test with and does so via JPA, hence the need for #Transactional
// code here to init an ExecutorService and a synchronized list
// code here to create a series of threads via the ExecutorService that uses different user IDs as the sender, but the same ID for the recipient, hence the need for pessimistic locking
}
}
I can put in the testing code if necessary, but, I'm not sure what other details are necessary.
The resulting output (especially from the System.out.println calls in FeeServiceImpl) shows that the same "system account" value is read in across all threads, and the saved value is, therefore, also always the same.
When the application starts up, that value is 0, and all threads read that 0, with no apparent locking or waiting. I can see multiple transactions starting up and committing (I increased the logging level on Hibernate's TransactionImpl), but, it doesn't seem to matter.
Hopefully I'm overlooking or doing something dumb, but, I can't quite figure out what it is.
Thank you!
Of course, it was something buried that I wasn't expecting.
It turns out my tables had been created using MyISAM instead of InnoDB, oddly, since that hasn't been the default for table creation in MySQL in a long time.
So, here's what I did:
I thought I was using MySQL 8.0. Turns out TestContainers defaults (to 5.7.22 in my case) when using a JDBC connection string that doesn't specifically name the version. So I fixed that.
This still didn't fix things as MyISAM was still being used. It turns out this was because I had a legacy dialect setting in my configuration. Updating that to something like MySQL57Dialect corrected that.
This actually also explains the "weird" behaviour I was seeing in my JUnit tests, as values were popping into the DB right away and not rolling back, etc.
I hope this helps someone else in the future!

How to schedule Laravel 5 job to get data from external JSON file, and store value in database?

I'm currently working on a project in Laravel, and I want to schedule a job that grabs a value (the price of Bitcoin) from an external API (JSON file) and stores this value in my database every few minutes.
So far, I have created a job using the artisan command: artisan make:job UpdateBitcoinMarketPrice. But I've no idea what to include in my public function handle() in side of the Job class that was created.
I have fathomed that I can call this job regularly from App\Console\Kernel.php with the following function:
protected function schedule(Schedule $schedule){
// $schedule->command('inspire')
// ->hourly();
$schedule->job(new UpdateBitcoinMarketPrice)->everyFiveMinutes();}
Should I, for example, create a new Model that stores said value? Then create a new Object every-time this run?
Should I then call the first row of the database should I wish to return the value?
Job classes are very simple, normally containing only a handle() method which is called when the job is processed by the queue. You can use the contructor to inject any parameter or serialize a model so you can use it in your handle method.
So to keep it bold you can make the api call on the handle method and store the response in the databse. Knowing that this is going to fire the api call as a background job.
Something along the lines of:
public function __construct(User $user)
{
//In this case Laravel serilizes the User model example so you could use it on your background job.
//This can be anything that you need in order to make the call
$this->user = $user;
}
//Injecting as an example ExtrernalServieClass or Transformer(to transform api response).
public function handle(ExternalServiceClass $service, Transformer $transform)
{
//Here you can make the call to the api.
//Get the response parse it
// Store to database
$response = $service->postRequest($someUri, $someParams);
$parsedResponse = $transform->serviceResponse($response);
DatabaseModel::firstOrCreate($parsedResponse);
}
}
The handle method is called when the job is processed by the queue. Note that you are able to type-hint dependencies on the handle method of the job, like in the example above. The Laravel service container automatically injects these dependencies.
Now since you are going to run the job everyFiveMinutes() you have to be careful since if the previous job is not completed by default, scheduled tasks will be run even if the previous instance of the task is still running.
To prevent this, you may use the withoutOverlapping method:
$schedule->job(new UpdateBitcoinMarketPrice)->everyFiveMinutes()->>withoutOverlapping();

Returning values from InputFormat via the Hadoop Configuration object

Consider a running Hadoop job, in which a custom InputFormat needs to communicate ("return", similarly to a callback) a few simple values to the driver class (i.e., to the class that has launched the job), from within its overriden getSplits() method, using the new mapreduce API (as opposed to mapred).
These values should ideally be returned in-memory (as opposed to saving them to HDFS or to the DistributedCache).
If these values were only numbers, one could be tempted to use Hadoop counters. However, in numerous tests counters do not seem to be available at the getSplits() phase and anyway they are restricted to numbers.
An alternative could be to use the Configuration object of the job, which, as the source code reveals, should be the same object in memory for both the getSplits() and the driver class.
In such a scenario, if the InputFormat wants to "return" a (say) positive long value to the driver class, the code would look something like:
// In the custom InputFormat.
public List<InputSplit> getSplits(JobContext job) throws IOException
{
...
long value = ... // A value >= 0
job.getConfiguration().setLong("value", value);
...
}
// In the Hadoop driver class.
Job job = ... // Get the job to be launched
...
job.submit(); // Start running the job
...
while (!job.isComplete())
{
...
if (job.getConfiguration().getLong("value", -1))
{
...
}
else
{
continue; // Wait for the value to be set by getSplits()
}
...
}
The above works in tests, but is it a "safe" way of communicating values?
Or is there a better approach for such in-memory "callbacks"?
UPDATE
The "in-memory callback" technique may not work in all Hadoop distributions, so, as mentioned above, a safer way is, instead of saving the values to be passed back in the Configuration object, create a custom object, serialize it (e.g., as JSON), saved it (in HDFS or in the distributed cache) and have it read in the driver class. I have also tested this approach and it works as expected.
Using the configuration is a perfectly suitable solution (admittedly for a problem I'm not sure I understand), but once the job has actually been submitted to the Job tracker, you will not be able to amend this value (client side or task side) and expect to see the change on the opposite side of the comms (setting configuration values in a map task for example will not be persisted to the other mappers, nor to the reducers, nor will be visible to the job tracker).
So to communicate information back from within getSplits back to your client polling loop (to see when the job has actually finished defining the input splits) is fine in your example.
What's your greater aim or use case for using this?

How can I defind that object set was already created?

I'm working with entity framework and mysql. We created a class
public class DataBaseContext : ObjectContext, IDbContext
There is a method
public IEnumerable<T> Find<T>(Func<T, bool> whereClause) where T : class
{
return CreateObjectSet<T>().Where(whereClause);
}
Is there a way not to create ObjectSet every time when I call the method? Can I check that it is already exists?
Whooooo. That is so bad method. You are passing Func<>, not Expression<Func<>>. It means that every time you execute your method EF will pull all records from database table mapped to T and execute your filtering in memory of your application - creating object set is the last thing you should be afraid of.
Anyway creating object set should not be expensive operation and if you don't want to create it every time you need to implement some "local caching" inside your object context instance.

Why EclipseLink 's auto commit doesn't work with MySQL?

Using the following code:
EntityManager manager = factory.createEntityManager();
manager.setFlushMode(FlushModeType.AUTO);
PhysicalCard card = new PhysicalCard();
card.setIdentifier("012345ABCDEF");
card.setStatus(CardStatusEnum.Assigned);
manager.persist(card);
manager.close();
when code runs to this line, the "card" record does not appear in the database. However, if using the FlushModeType.COMMIT, and using transaction like this:
EntityManager manager = factory.createEntityManager();
manager.setFlushMode(FlushModeType.COMMIT);
manager.getTransaction().begin();
PhysicalCard card = new PhysicalCard();
card.setIdentifier("012345ABCDEF");
card.setStatus(CardStatusEnum.Assigned);
manager.persist(card);
manager.getTransaction().commit();
manager.close();
it works fine. From the eclipselink's log i can see the previous code doesn't issue an INSERT statement while the second code does.
Do I miss something here? I'm using EclipseLink 2.3 and mysql connection/J 5.1
I am assuming that you are using EclipseLink in a Java SE application, or in a Java EE application but with an application managed EntityManager instead of a container managed EntityManager.
In both scenarios, all updates made to the persistence context are flushed only when the transaction associated with the EntityManager commits (using EntityTransaction.commit), or when the EntityManager's persistence context is flushed (using EntityManager.flush). This is the reason why the second code snippet issues the INSERT as it invokes the EntityTransaction's begin and commit methods, while the first doesn't; an invocation of em.persist does not issue an INSERT.
As far as FlushModeType values are concerned, the API documentation states the following:
COMMIT
public static final FlushModeType COMMIT
Flushing to occur at transaction commit. The provider may flush at
other times, but is not required to.
AUTO
public static final FlushModeType AUTO
(Default) Flushing to occur at query execution.
Since, queries haven't been executed in the first case case, no flushing i.e. no INSERT statements corresponding to the persistence of the PhysicalCard entity will be issued. It is the explicit commit of the EntityTransaction in the second, that is resulting in the INSERT statement being issued.