Google Stackdriver Error Reporting not picking up errors - google-compute-engine

Question

Logs with severity “ERROR” is not identified by Error and Reporting tool. Application logs are being directed at Google Stackdriver Logging using fluentd agent and some of these are third party java components.
{
insertId: "14sf3lvg3ccncgh"
jsonPayload: {
class: "o.a.w.MarkupContainer"
message: "Unable to find component with id 'search2' in [Form [Component id = form]]
Expected: 'form:search2'.
Found with similar names: 'form:search'
at org.apache.wicket.markup.MarkupStream.throwMarkupException(MarkupStream.java:526) ~[wicket-core-6.22.0.jar:6.22.0]
at org.apache.wicket.MarkupContainer.renderNext(MarkupContainer.java:1438) ~[wicket-core-6.22.0.jar:6.22.0]
at org.apache.wicket.MarkupContainer.renderAll(MarkupContainer.java:1557) ~[wicket-core-6.22.0.jar:6.22.0]
at org.apache.wicket.MarkupContainer.renderComponentTagBody(MarkupContainer.java:1532) ~[wicket-core-6.22.0.jar:6.22.0]
at org.apache.wicket.MarkupContainer.onComponentTagBody(MarkupContainer.java:1487) ~[wicket-core-6.22.0.jar:6.22.0]"
milsec: "576"
reportLocation: {…}
serviceContext: {…}
tag: "test.gui"
thread: "[ajp-apr-8009-exec-5]"
}
labels: {…}
logName: "projects/myservice/logs/test.gui"
receiveTimestamp: "2017-08-29T15:20:16.847782870Z"
resource: {…}
severity: "ERROR"
timestamp: "2017-08-29T15:20:11Z"
}
Using the following configuration allows for my application logs to be forwarded correctly to Googles Stackdriver logging and all entries are correctly identified.
<source>
#type tail
path /var/log/test/test_gui/test_gui.log
pos_file /var/lib/google-fluentd/pos/test_gui-multiline.pos
read_from_head true
tag test.gui
format multiline
time_format %Y-%m-%d %H:%M:%S
format_firstline /\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{1,3}\s(?<severity>\S*)/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}),(?<milsec>\d{1,3})\s(?<severity>\S*)\s(?<class>\S*)\s(?<thread>\[\S*\])\s(?<message>.*)/
</source>
However for severity ERROR, Error Reporting never noticed these entries.
The output was identified as textPayLoad, I used the following filter, which ensured output was jsonPayload
<filter test.gui>
#type record_transformer
<record>
serviceContext {"service": "test.gui", "version": "1"}
reportLocation {"filePath": "test_gui.log", "lineNumber": "unknown", "functionName": "unknown"}
tag ${tag}
</record>
</filter>
Still the error jsonPayload is being ignored.
If I replace the message using the filter, then suddenly Error Reporting is working
<filter test.gui>
#type record_transformer
<record>
serviceContext {"service": "test.gui", "version": "1"}
reportLocation {"filePath": "test_gui.log", "lineNumber": "unknown",
"functionName": "unknown"}
message "java.lang.TestError: msg
at com.example.TestClass.test (TestClass.java:51)
at com.example.AnotherClass (AnotherClass.java:25)"
tag ${tag}
</record>
</filter>
How can I force Error Reporting to pick these error entries as my next step would be to implement some form of alerting.

Answer 1

Third parties did not produce correct Java Stack trace. I needed the reportLocation, but this needs to be in context.
I changed the following line:
reportLocation {"filePath": "test_gui.log", "lineNumber": "unknown", "functionName": "unknown"}
to
context { "reportLocation" : {"filePath": "test_gui.log", "lineNumber": 1, "functionName": "unknown"} }
which ensured that the logs are now picked up by Stackdriver Error Reporting.
This is the final version of my filter:
<filter test.gui>
#type record_transformer
<record>
serviceContext {"service": "test.gui", "version": "1"}
context { "reportLocation" : {"filePath": "test_gui.log", "lineNumber": 1, "functionName": "unknown"} }
tag ${tag}
</record>
</filter>