The following error occurs when running the sandbox:
io.grpc.netty.NettyServerHandler onStreamError
WARNING: Stream Error
io.netty.handler.codec.http2.Http2Exception$HeaderListSizeException: Header size exceeded max allowed size (8192)
What could the cause of this be?
I have seen this error numerous times, and it is a consequence of having a transaction failure in a complex DAML model/transaction when running on the Sandbox. When you experience a transaction failure (fetch/exercise an inactive contract, lookupByKey returned a stale cid, head [], divide-by-zero, etc) the sandbox helpfully tries to provide transaction trace information in the error result.
This is normally fine for relatively simple models. With more complex models this trace can exceed the maximum header size producing the error you see. When this happens I have found the trace in the sandbox.log file, sometimes along with other errors that help explain what is going on.
The trace is an unformatted dump, so it can take a bit of effort to decode manually, but I have done it many times myself and the information I needed to identify the issue has always been there —— and to be honest, generally just knowing the choice I was exercising + the specific class of error is normally enough to point me in the right direction.
I believe there is some tooling being built to help with this sort of diagnosis; however, I don't know how advanced the work on that is.
Related
I have a few functions and I share user session among the functions using Cloud Memorystore. I used connect-redis package and I modified it to work with Memorystore.
It works without issues mostly. However, I have found that at times, the cloud functions were unable to access the session. It doesn't happen frequently and I have faced this issue maybe three to four times in the last one and a half month. There are no errors in the functions and I have rigorously checked my functions.
I have always found that redeploying the functions, even without any changes to the code fixes the issue. I have only been working with GCP products for over two months now and I am not sure if these two products are incompatible or are there any edge cases that are being triggered that results in the following issue.
Due to sudden nature of the error I am also not sure if I can replicate the events leading to the error. What can I do to debug this error and have a more concrete understanding of what's happening?
According to this the usage of Cloud Functions with Memorystore should work normally without any issues.
It could be caused by many factors. Possibly connection timeout, cold start of the Function, maybe misuse of Memorystore leading to an issue there preventing it from working as expected.
What I suggest is that you add logging before and after every part of the code that completes a big part.
So basically try to locate which parts of the code are causing the issue or not showing expected results when the issue occurs, then split that part into smaller parts to find what is causing the issue. If even with logging everything seems okay with the Cloud Function, most likely something is going on on Memorystore side.
It might also be worth it then to open a Public Issue for further investigation if the issue seems to be something irrelevant to your code or configuration. Issue Tracker
I'm really looking for your help people...
I'm currently in a transition of switching from Splunk MINT (previously Bugsense) to HockeyApp for beta distribution and bug reporting of my Windows 8.1 application.
My problem is that I've several crash reports without any stacktrace which makes it pretty much impossible to resolve this errors.
I'm not an expert in this field so I'm curious how is it possible that nearly all of the received crashes include a stacktrace but only a few are without any detailed information.
Also the exact exception report may vary for the two different Bugreporting frameworks.
A few examples:
System.ArgumentException: The paramter is incorrect.
(System.ArgumentException: Value does not fall within the expected range.)
No stacktrace!
HockeyApp reports it as "parameter is incorrect" while the Splunk report says "Value does not fall within the expected range".
It's definitely the same crash since it occurs at the exact same timestamps and for the same devices. I don't get why the reports are different
Occurs about ~1 every 1000 sessions
System.Exception: The operation cannot be completed because the window is being closed.
Same report on Splunk & HockeyApp
Minimal stacktrace on Splunk (No stacktrace on HockeyApp):
1 Windows.UI.Core.CoreDispatcher.RunAsync
2 System.Threading.Tasks.AwaitTaskContinuation.RunCallback
Occurs about ~1 every 1000 sessions
System.Exception: Exception of type 'System.Exception' was thrown.
No stacktrace!
It seems like there are some reports where only one of the two crashreporting frameworks displays a stacktrace. For others neither does.
Has anybody any idea for the absence of stacktraces at all?
Many thanks for any help in advance!!
I keep hitting this error a low though non negligible percentage of the time ( low single digits in my testing ).
I am positive it isn't my code since I was able to repro this with a test case of just the same API call to check the changes feed without any changes to the data set used for my account.
<errors xmlns='http://schemas.google.com/g/2005'><error><domain>GData</domain><code>ServiceException</code><internalReason>An unknown error has occurred.</internalReason></error></errors>
How does one go about debugging this issue given that there is no further detail in the error message ?
Even though it happens ~5% of the time, it is high enough that the failure is noticeable.
I see very similar behavior with drive.changes.list(), which again seems to fail at a comparable rate.
I'm want to get an idea how I should handle end-user visible error messages in my web application.
How much information do you give in
error messages?
Do you redirect all errors,
regardless of type, to a common error
page, or do you have a small set of pages (404, 403, all others)?
Do you give error codes that the user
could reference/give to you that only
you understand?
Do you give any technical details?
As I stated, my users are non-technical regular Joe folks.
Display a nice error to the user, Log a detailed error for yourself.
I try to do the following:
make sure you never run the risk of passwords or connection strings appearing in error messages.
Make sure the errors get logged to a persistable medium. I prefer a database so that I can query by time range and other paramaters. I don't log 404s.
If the application is an internal app that does not need to be pretty, it may be ok to have the error info on the page. Even if you are logging this stuff, it is nice to be able to have your users email you a screen shot or copy/paste.
If 3 seems distasteful, have some error info written as HTML comments. Then you can at least see the info by viewing source.
In general I try to give users as much information needed to help them solve their problems themselves. For example, in the case of a 404, you might want to let them know to double check that the URL they are looking for is correct.
They obviously wont need stack traces, and the like, but it will make sense for you to log that level of detail somewhere for diagnostics and debugging.
for fatal errors, keep them short, so they can repeat them over the phone or e-mail: can't connect to database, etc.
for non-fatal errors, describe the condition fully: Error, can not save the invoice without an invoice date.
I also always log everything, the parameters to the function and any internal values that may be of use.
I try to show users enough information that they know it's an issue they need to tell someone about, but try to avoid showing them so much it scares them!
If possible the error message should tell them what just failed e.g did their save just fail, or has it saved fine, but the refresh of the screen afterwards had an issue. Extra error information (e.g. stack traces) should be logged somewhere where you can get at it without the user having to send it to you.
When it comes to displaying errors for the end user, I find it a good practise to display a errorcode (so me and administrators know what error it is) and a typical "ops something went wrong, please contact an administrator"
It can be good to give a bit more information for common errors that could be the cause of the users actions. But usually too much information can scare or confuse the user.
None, just show give a reference number so user can give it to you, and you can check the details from the application logs (obviously you need to keep a copy of error logs).
Your web application's error messages should always (at
least) be the answers these 3 questions (in that order):
What happened?
Why did it happen?
What can be done about it?
I have used it for many years, originally from Apple's
"Human Interface Guidelines: The Apple Desktop Interface". Newer version.
Microsoft has similar guidelines.
This also makes it easy to write them - this structured
approach makes it faster to write them as one can just
answer the questions.
The error messages should also be specific. Any information
that the web application know about and that the user may
need to resolve the problem should be in the error message.
The (infamous) error message "An error happend." is simply
not acceptable.
Optional: more technical information that the user may not
understand can be placed at the end. But it should be marked
as such.
I suspect that all non-trivial software is likely to experience situations where it hits an external problem it cannot work around and thus needs to fail. This might be due to bad configuration, an external server being down, disk full, etc.
In these situations, especially if the software is running in non-interactive mode, I expect that all one can really do is log an error and wait for the admin to read the logs and fix the problem. If someone happens to interact with the software in the meantime, e.g. a request comes in to a server that failed to initialize properly, then perhaps an appropriate hint can be given to check the logs and maybe even the error can be echoed (depending on whether you can tell if they're a technical guy as opposed to a business user). For the moment though let's not think too hard about this part.
My question is, to what extent should the software be responsible for trying to explain the meaning of the fatal error? In general, how much competence/knowledge are you allowed to presume on administrators of the software, and how much should you include troubleshooting information and potential resolution steps when logging fatal errors? Of course if there's something that's unique to the runtime context this should definitely be logged; but lets assume your software needs to talk to Active Directory via LDAP and gets back an error "[LDAP: error code 49 - 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 525, vece]". Is it reasonable to assume that the maintainers will be able to Google the error code and work out what it means, or should the software try to parse the error code and log that this is caused by an incorrect user DN in the LDAP config?
I don't know if there is a definitive best-practices answer for this, so I'm keen to hear a variety of views.
The approach I tend to agree with is that you should explain as much as possible if the fatal error is caused by some code in your own responsibility (i.e. not third party). Otherwise if the error is caused "further down", for example at the database level, then the administrators should be passed up the error returned without adding much further information. So if the database server dies, then your connector with throw some exception, and you would log the error code in the exception.
The administrator or support staff should then have sufficient knowledge to resolve the issue with the provided information.
When you do provide too much details on errors which are not caused by your own code you run the risk of having error details NOT matching the cause of the actual error, especially if the error codes stop matching between versions.
Of course, there are exceptions. We have worked with open source libraries that were so poorly documented that we ended up writing wrappers around the libraries just to provide decent logging of what actually is going on.
Just my 2c
The answer, as for all broad questions, is "it depends."
If you're looking at a configuration error, then by all means you should try to explain what was wrong (in the logs). If it's an out-of-memory error, there's not much you can do -- and you may not even be able to write a log message.
One thing you said concerns me:
If someone happens to interact with
the software in the meantime, e.g. a
request comes in to a server that
failed to initialize properly, then
perhaps an appropriate hint can be
given to check the logs
If this is truly a fatal error, the server should not be running, and therefore any incoming request should fail with absolutely no warning or explanation.
You should at least provide the message from the exception and a stack trace so you can find out where in the code it occurred. If possible, you should also explain what you were attempting to do and what you think may have happened depending on the exception type.
I guess it depends on how much time you have before delivering the software to your customers.
Yes, it would be nice to parse the error and give a more explicit message but, in this day and age, Google is not always very far.
So unless, you have time to create the code to parse errors, I would leave them as is.
IMHO you can never provide too much information in these case.
In the real world it comes down to cost-benefit analysis, though. What's the impact of the error to you, your app, your business, etc. How much time is it worth spending on it.
In a business critical app my first point applies. Everything else is a sliding scale.
I think it depends on who is using the application.
If the application is used by tech savvy people then show more technical details, so they will be able to troubleshoot the problem if they want. I've had some users go to great lengths to solve issues. It can be very helpful, especially for issues that are specific to certain configurations.
If your user base is more of the average Joe then technical details will confuse them in most cases. You should show them a simple error message, and try to offer some solutions if possible.
You could also merge the two techniques. Show a simple error message by default and allow the user to view more detailed error information if they want.
You just don't want to overwhelm the user with too much information that they don't understand. It just frustrates and confuses them in the majority of cases.
There are two aspects I think all errors and exceptions should have:
1) Enough information in the error to help debug the problem. Stacktrace, class/method name, type of exception etc fall in this category.
2) A human understandable message, ideally clear enough for say Ops team or Sysadmins engineer to know who to call or forward that error message. Typically it is of the form "so and so module failed" or "network call failed" etc. Something that will come as close to you explaining the problem to customer, in non technical jargon.
Now with all the time constraints etc it may not be possible to have both messages programmed in. Then I would go out on a limb and say we should have the second type of error message. Remember, the sysadmin would probably be able to call you and since you helped write the code you can maybe pinpoint the error. But if the customer is on phone asking about the error, the sysadmin better be able to explain the possible cause :)
On a different note, all products need a clear exception/error handling mechanism decided at architecture level. And the exceptions NEED to adhere to that design. There are few things more frustrating than trying to debug an error based on a design only to find out a day later that its a one of a kind error message based on completely different design.
See https://meta.stackexchange.com/questions/3122/formatting-sandbox