Does jOOQ use different connections to execute different queries? - mysql

I have a service which does some CRUD over a DB. Service uses jOOQ. This is what my transaction code looks like:
dslContext.transaction(conf -> {
// Query1
dslContext.update(Tables.TABLE1).set(Tables.TABLE1.COLUMN1, column1Value).where(Tables.TABLE1.COLUMN2.eq(column2Value)).execute();
// Query2
dslContext.insertInto(Tables.TABLE2).set(record1).execute();
});
Note that dslContext is global and we are using Apache DBCP2 BasicDataSource.
At high load, when the connection pool gets exhausted, all the threads gets stuck and leaves the DB in a state where only Query1 is executed.
Assuming that the connection pool size is N and is configured to wait indefinitely for a new thread, Hypothesis is that
At t=t0, N simultaneous threads try to execute Query1 in Connection1
At t=t1, these N threads move on to Query2 but doesn't get a connection from the pool and waits indefinitely.
Since threads are stuck, transaction didn't end and commit is not called, due to which N threads get stuck forever and never frees up. Rollback also doesn't happen because only way to bring back the system is to restart it. This leaves the DB in an inconsistent state.
Is my hypothesis correct ?

You're not using the transactional conf reference, you're using the outer dslContext, which isn't transactional - meaning each statement is run in its own transaction, just as if you weren't calling the transaction() method.
Do this instead:
dslContext.transaction(conf -> {
// Query1
conf.dsl().update(Tables.TABLE1)
.set(Tables.TABLE1.COLUMN1, column1Value)
.where(Tables.TABLE1.COLUMN2.eq(column2Value)).execute();
// Query2
conf.dsl().insertInto(Tables.TABLE2).set(record1).execute();
});

Related

host.json; meaning of batchsize

Does it make sense to set batchSize = 1? In case I would like to process files one-at-a-time?
Tried batchSize = 1000 and batchSize = 1 - seems to have the same effect
{
"version": "2.0",
"functionTimeout": "00:15:00",
"aggregator": {
"batchSize": 1,
"flushTimeout": "00:00:30"
}
}
Edited:
Added into app setings:
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1
Still the function is triggered simultaneously - using blob trigger. Two more files were uploaded.
From https://github.com/Azure/azure-functions-host/wiki/Configuration-Settings
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1
Set a maximum number of instances that a function app can scale to. This limit is not yet fully supported - it does work to limit your scale out, but there are some cases where it might not be completely foolproof. We're working on improving this.
I think I can close this issue. There is no easy way how to set one-message-one-time feature in multiple function apps instances.
I think your misunderstand the batchSize meaning with aggregator. This batchSize means Maximum number of requests to aggregate. You could check here and about the aggregator it's configured to the runtime agregates data about function executions over a period of time.
From your description, it's similar to the Azure Queue batchSize. It sets the number of queue messages that the Functions runtime retrieves simultaneously and processes in parallel. And If you want to avoid parallel execution for messages received on one queue, you can set batchSize to 1(This means one-message-one-time).

Concurrent execution of streams and host functions

I have wrote a program which has two streams. Both streams operate on some data and write results back on the host memory.
Here is the generic structure of how i am doing this:
loop {
AsyncCpy(....,HostToDevice,Stream1);
AsyncCpy(....,HostToDevice,Stream2);
Kernel<<<...,Stream1>>>
Kernel<<<...,Stream2>>>
/* Write the results on the host memory */
AsyncCpy(....,DeviceToHost,Stream1);
AsyncCpy(....,DeviceToHost,Stream2);
}
I want to do some work on the CPU once i know that StreamX has finished copying the results back to the host memory. At the same time, i don't want to stop the loop from executing Async operations (memcpy or kernel execution).
If i insert my host functions, let say host_ftn1(..) and host_ftn2(..) like this
loop {
AsyncCpy(....,HostToDevice,Stream1);
AsyncCpy(....,HostToDevice,Stream2);
Kernel<<<...,Stream1>>>
Kernel<<<...,Stream2>>>
/* Write the results on the host memory to be processed by host_ftn1(..) */
AsyncCpy(....DeviceToHost,Stream1);
/* Write the results on the host memory to be processed by host_ftn2(..) */
AsyncCpy(....DeviceToHost,Stream2);
if(Stream1 results are copied to host)
host_ftn1(..);
if(Stream2 results are copied to host)
host_ftn2(..);
}
It will stop the execution of loop until it finishes the execution of host functions i.e. host_ftn1 and host_ftn2, but I don't want to stop the execution of GPU instructions i.e. AsyncCpy(..) and Kernel<<<....,StreamX>>> while the CPU is busy executing the host functions i.e. host_ftn1(..) and host_ftn2(..)
Any solution/approach regarding this problem
As huseyin tugrul buyukisik suggested, the stream callback worked in this scenario. I have tested this for two streams.
The final design is as following:-
loop {
AsyncCpy(....,HostToDevice,Stream1);
AsyncCpy(....,HostToDevice,Stream2);
Kernel<<<...,Stream1>>>
Kernel<<<...,Stream2>>>
/* Write the results on the host memory to be processed by host_ftn1(..) */
AsyncCpy(....DeviceToHost,Stream1);
/* Write the results on the host memory to be processed by host_ftn2(..) */
AsyncCpy(....DeviceToHost,Stream2);
callback1(..); // Work to be done on the host once stream1 completes
callback2(..); // Work to be done on the host once stream2 completes
}
See Stream Callbacks

Continuously read data from a serial port while loop is running

First, please refer to this block of code:
while(1) {
lt = time(NULL);
ptr = localtime(&lt);
int n = read (fd, buf, sizeof(buf));
strftime(str, 100, "%c", ptr);
int temp = sprintf(tempCommand, "UPDATE roomtemp SET Temperature='%s' WHERE Date='Today'", buf);
temp = sprintf(dateCommand, "UPDATE roomtemp SET Date='%s' WHERE Type='DisplayTemp'", str);
printf("%s", buf);
mysql_query(conn, tempCommand);
mysql_query(conn, dateCommand);
}
The read function is actually reading data coming in from a serial port. It works great, but the problem I am experiencing (I think) is the time it takes for the loop to execute. I have data being sent to the serial port every second. Suppose the data is "22" every second. What this loop does is read in "2222" or sometimes "222222". What I think is happening is that the loop takes too long to iterate, and that causes data to accumulate in the serial buffer. The read statement reads in everything in the buffer, hence giving me repeated values.
Is there any way to get around this? Perhaps at the end of the loop, I can flush the buffer. But I am not certain I know how to do this. Or perhaps there is some way to cut down the code inside the loop in order to reduce the overall time each iteration takes in the first place. My guess is that the MySQL queries are what take the most time anyway.
To start with you should check for errors from read, and also properly terminate the received "string".
To continue with your problem, there are a couple of ways to solve this. One it to put either the reading from the serial port or the database updates in a separate thread. Then you can pass "messages" between the the threads. Be careful though, as it seems your database is slow and the message queue might build up. This message-buildup can be averted by having a message queue of size one, which always contain the latest temperature read. Then you only need a single flag that the temperature reading thread sets, and the database updating thread checks and then clears.
Another solution is to modify the protocol used for the communication, so it includes a digit to tell how big the message is.

Why are the return addresses of prefetch abort and data abort different in ARM exceptions?

for prefetch, the return address is:
R14_abt = address of the aborted instruction + 4
and for data abort, the return address is:
R14_abt = address of the aborted instruction + 8
These offsets are due to the processor's pipelining and the fetch/decode/execute stages.
The processor's program counter (PC) is updated at specific points during execution. Exceptions can occur during different phases of fetching/decoding/execution.
In the case of the prefetch abort, the instruction cannot be (has not been) executed; the exception occurs only when the processor actually attempts to execute the instruction (some prefetched instructions may not be executed).
In the case of the data abort, the instruction is being executed, and the instruction's execution causes the exception.
From the ARM documentation:
Regarding prefetch abort:
[The prefetch abort exception] Occurs when the
processor attempts to execute an
instruction that has prefetched from
an illegal address, that is, an
address that the memory management
subsystem has determined is
inaccessible to the processor in its
current mode.
... Instructions already in the pipeline continue to execute until the invalid instruction is reached, at which point a prefetch abort is generated.
... because the program counter is not updated at the time the prefetch abort is issued, lr_ABT points to the instruction following the one that caused the exception. The handler must return to lr_ABT – 4
And regarding the data abort:
[The Data Abort exception] Occurs when
a data transfer instruction attempts
to load or store data at an illegal
address.
When a load or store instruction tries to access memory, the program counter has been updated. A stored value of (pc – 4) in lr_ABT points to the second instruction beyond the address where the exception was generated. When the MMU has loaded the appropriate address into physical memory, the handler should return to the original, aborted instruction so that a second attempt can be made to execute it. The return address is therefore two words (eight bytes) less than that in lr_ABT
So in other words, for the data abort, the handler must return to lr_ABT – 8 (two words/instructions previous)
I don't remember seeing an official explanation, but if you think about it, it's pretty logical.
Let's consider this example:
00000 INSN1 [PC = 08]
00004 INSN2 [PC = 0C]
00008 INSN3 [PC = 10]
If processor can't fetch the INSN3, the abort happens before executing it, so the PC value is still the one of INSN2, i.e. 0C.
If a data abort happens during execution of INSN3, the PC value is already updated to 10.
For prefetch abort
new_lr_value = if CPSR.T == ‘1’ then PC else PC-4
For data abort
new_lr_value = if CPSR.T == ‘1’ then PC+4 else PC;
Reference TRM TakePrefetchAbortException() and TakeDataAbortException()

Perl script multi thread not running parallel

I am completely new to Perl, like absolute newbie. I am trying to develop a system which reads a database and, according to the results, generates a queue which launches another script.
HERE is the source code.
Now the script works as expected, except I have noticed that it doesn't really do the threads parallel. Whether I use 1 thread or 50 threads, the execution time is the same; 1 thread is even faster.
When I have the script display which thread did what, I see the threads don't run at the same time, because it will do thread 1, then 2, then 3 etc.
Does anyone know what I did wrong here? Again the script itself works, just not in parallel threads.
You need to learn what semaphores actually are before you start using them. You've explicitly told the threads not to run in parallel:
my $s = Thread::Semaphore->new;
#...
while ($queue_id_list->pending > 0) {
$s->down;
my $info = $queue_id_list->dequeue_nb;
if (defined($info)) {
my #details = split(/#/, $info);
#my $result = system("./match_name db=user_".$details[0]." id=".$details[1]);
# normally the script above would be launched which is a php script run in php-cli and does some database things
sleep(0.1);
#print "Thread: ". threads->self->tid. " - Done user: ".$details[0]. " and addressbook id: ". $details[1]."\r\n";
#print $queue_id_list->pending."\r\n";
}
$s->up;
}
You've created a semaphore $s, which by default has a count of 1. Then in the function you're trying to run, you call $s->down at the start -- which decreases the count by 1, or blocks if the count is already <1, and $s->up at the end, which increases the count by 1.
Once a thread calls down, no other threads will run until it calls up again.
You should carefully read the Thread::Semaphore docs, and probably this wikipedia article on semaphores, too.