In any application, we can do error logging using flat file system.
How do we handle a scenario when there are multiple users having exceptions which are logged in the same flat file?
Many Thanks.
You'd acquire a write lock to the logfile before logging an exception. If someone else tries to log while you're in the middle of writing, their lock request will wait until you release it.
Alternatively, if such a thing is available, use an atomic file write operation to log the entire exception.
You could create one thread for writing exceptions, reading from a queue. The actual exception handling code would write the exception to the queue. Since there's only one thread writing the file, everything is serialized. Of course, you need a thread-safe queue implementation, but your language or framework probably provides one.
Related
A question that might be mostly theoretical, but I'd love to have my concerns put to rest (or confirmed).
I built a Script Component to pull data from RabbitMQ. On RabbitMQ, we basically set up a durable queue. This means messages will continue to be added to the queue, even when the server reboots. This construction allows us to periodically execute the package and grab all "new" messages since the last time we did so.
(We know RabbitMQ isn't set up to accommodate to this kind of scenario, but rather it expects there to be a constant listener to process messages. However, we are not comfortable having some task start when SQL Server starts, and pretty much running 24/7 to handle that, so we built something we can schedule to run every n minutes and empty the queue that way. If we'd not be able to run the task, we most likely are dealing with a failed SQL Server, and have different priorities).
The component sets up a connection, and then connects to the specific exchange + queue we are pulling messages from. Messages are in JSON format, so we deserialize the message into a class we defined in the script component.
For every message found, we disable auto-acknowledge, so we can process it and only acknowledge it once we're done with it (which ensures the message will be processed, and doesn't slip through). Then we de-serialize the message and push it onto the output buffer of the script component.
There's a few places things can go wrong, so we built a bunch of Try/Catch blocks in the code. However, seeing we're dealing with the queue aspect, and we need the information available to us, I'm wondering if someone can explain how/when a message that is sent to the output buffer is processed.
Is it batched up and then pushed? Is it sent straight away, and is the SSIS component perhaps not updating information back to SSIS in a timely fashion?
Would there be a chance for us to acknowledge a message, but that it somehow ends up not getting committed to our database, yet popped from the queue (as I think happens once a message is acknowledged)?
I have a Yii application with many concurrent console jobs writing to one database. Due to the high concurrency sometimes I get MySQL deadlock errors. Sometimes these can be too many. The console.log file becomes too big, and it translates to more expenses.
I want to prevent logging of specific CDbException instances, or at least suppress them altogether (I am handling the exceptions and can generate more compact log sentences from there).
YII__DEBUG is already commented out.
Can anyone please help me figure out how to do this?
Thanks a lot!!
Regards.
I decided to modify the log statement in yii/framwework/db/CDbCommand.php that was logging the failed SQL. I converted it into a trace statement:
Yii::trace(Yii::t('yii','CDbCommand::{method}() failed: {error}. The SQL statement executed was: {sql}.', array('{method}'=>$method, '{error}'=>$message, '{sql}'=>$this->getText().$par)),CLogger::LEVEL_ERROR,'system.db.CDbCommand');
I am anyway catching the exception and logging a more compact version of the sentence, so it is OK for me to do it.
This was the easiest way I could find. We don't upgrade Yii very often, so if and when we go to the next version I'll probably repeat the change.
I want to use mysql row level lock. I can't lock complete table. I want to avoid two process processing two different message for server at same time.
What I thought that I can have some table called:
server_lock and if one process start working on server it will insert a row in the table.
Problem with this approach is that if application crashes. We need to remove the lock manually.
Is there a way I may row level lock and lock will get released if application is crashing ?
Edit
I am using C++ as language.
My application is similar to message queue. But difference is that there is two queue which are getting populated by one process for each queue. After action if action belong to same object and both are processing same object it may result in wrong data. So I want a locking mechanism b/w these two queue so that both processor don't modify same object at same time.
I can think of two ways:
Implement some error handler on your program where you remove the lock. Without knowing anything about your program it is hard to say how to do this, but most languages have some method to do some work before exiting upon a crash. This is dangerous, because a crash happens when something is not right. If you continue to do any work, it is possible that you corrupt the database or something like that.
Periodically update the lock. Add a thread on your program that periodically reacquires the lock, or reacquire the lock in some loop you are doing. Then, when a lock is not updated in a while, you know that it belonged to a program that crashed.
I'm currently designing a system for processing uploaded files.
The files are uploaded through a LAMP web frontend and must be processed through several stages some of which are sequential and others which may run in parallel.
A few key points:
The clients uploading the files only care about safely delivering the files not the results of the processing so it can be completely asynchronous.
The files are max 50kb in size
The system must scale up to processing over a million files a day
It is critical that no files may be lost or go unprocessed
My assumption is MySQL, but I have no issue with NoSQL if this could offer an advantage.
My initial idea was to have the front end put the files straight into a MySQL DB and then have a number of worker processes poll the database setting flags as they completed each step. After some rough calculations I realised that this wouldn't scale as the workers polling would start to cause locking problems on the upload table.
After some research it looks like Gearman might be the solution to the problem. The workers can register with the Gearman server and can poll for jobs without crippling the DB.
What I am currently puzzling over is how to dispatch jobs in the most efficient manner. There are three ways I can see to do this:
Write a single dispatcher to poll the database and then send jobs to Gearman
Have the upload process fire off an asynchronous Gearman job when it receives a file
Use the Gearman MySQL UDF extension to make the DB fire off jobs when files are inserted
The first approach will still hammer the DB somewhat but it could trivially recover from a failure.
The second two approaches would seem to require enabling Gearman queue persistence to recover from faults, but I am concerned that if I enable this I will loose the raw speed that attracts me to Gearman and shift the DB bottleneck downstream.
Any advice on which of these approaches would be the most efficient (or even better real world examples) would be much appreciated.
Also feel free to pitch in if you think I'm going about the whole thing the wrong way.
This has been open for a little while now so I thought I would provide some information on the approach that I took.
I create a gearman job every time a file is uploaded for a "dispatch" worker which understands the sequence of processing steps required for each file. The dispatcher queues gearman jobs for each of the processing steps.
Any jobs that complete write back a completion timestamp to the DB and call the dispatcher which can then queue any follow on tasks.
The writing of timestamps for each job completion means the system can recover its queues if processing is missed or fails without having to have the burden of persistent queues.
I would save the files to disk, then send the filename to Gearman. As each part of the process completes, it generates another message for the next part of the process, you could move the file into a new work-in-process directory for the next stage to work on it.
We all know that logging is very important, and that there are a multitude of potential places to log to. (e.g. a file, a database, the event log, ...)
However, what do you do when the logging itself throws an exception? If we try to log to a file, but don't have permissions, or the file is locked, we can log to the event log. I don't know how it would happen, but I assume that there is some scenario that could cause logging to the event log to also fail. How would you handle exceptions that occur while logging to ensure that it is reported somewhere?
What methods of logging are least likely to fail?
What methods of logging are most likely to fail?
My current scheme is for all logging to go to a file, with a little bit also going to the event log. If logging to the file fails, I would log that to the event log. I don't really have a good plan for the event log failing.
I'd say you're going too deep. Logging frameworks should take care of themselves. That means: no exceptions, unless something really horrid is going on behind the scenes.
File locking exceptions should never appear. If they do, your logging framework has a flaw. That means you're using a wrong framework, since that would be a very fundamental flaw.
Secondly, file permissions. It is YOU who decides where logged files will appear. If you don't take into account file and directory permissions, it is your fault. You must make sure your logger can log where you tell it to log.
Bottom line: log to files. This is the most convenient way. Also, the fastest way. Logging to a database can always fail. If the framework is robust enough, it won't throw any exceptions. You must ensure that file permissions are set up properly. And that's all there is to it.