Due to reasons outside of my control, newlines aren't handled properly by some logging infrastructure I have to use.
A workaround is to replace the \n with another character, e.g. _newline_
This can be done in logback by configuring the pattern:
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<!-- Standard pattern -->
<!-- <pattern>%coloredLevel - %logger - %message%n%xException{15}</pattern> -->
<!-- With newlines removed -->
<pattern>%coloredLevel - %logger - %replace(%message){'\n', '_newline_'}_newline_%replace(%xException){'\n', '_newline_'}%nopex%n</pattern>
</encoder>
</appender>
However, this adds a superfluous _newline_ in logslines when there isn't an exception. (and adds an extra newline to stack traces, but this isn't a large problem)
Is there a way to only output the _newline_ in between the message and the if there is an exception?
Found a bit easier solution to conditionally output newline between log message and exception message. It seems that exception message is always followed by newline, no matter how many lines of stack trace you print. What could you do is print an exception with 0 lines of stack trace (%ex{0}), and then truncate from left all the characters until the length is no more than 1 (%.1ex{0}). This effectively leaves a \n if there's an exception and produces empty string in case there's no exception.
So this:
// logger.error("Log message", new RuntimeException("Exception message", new RuntimeException("Cause")));
// logger.error("Log message without exception");
<pattern>%-5p %m%.1ex{0}%ex%nopex%n</pattern>
Results in:
ERROR Log message
java.lang.RuntimeException: Exception message
[ redacted ]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Cause
... 10 common frames omitted
ERROR Log message without exception
Now the fun thing. Remember the fact that exceptions are always followed by newline? This makes the log events with exceptions to be followed by empty line (one newline from exception and one from the pattern), unless you know a trick how to print something only when there are no exceptions...
P.S. you can wrap everything in replace(){} to escape newlines, too:
<pattern>%-5p %replace(%m%.1ex{0}%ex){'[\r\n]+', 'LF'}%nopex%n</pattern>
I solved it very hackily, by replace an extra exception stack trace with a single _newline_
<pattern>%coloredLevel - %logger - %replace(%message){'\n', '_newline_'} %replace(%replace(%ex{0}){'\n',''}){'.+', '<nl>'}%replace(%xException){'\n', '_newline_'}%nopex%n</pattern>
Related
I have below footer.ftl file as below,
3<#t>
~<#t>
<#-- Record -->
${incrementDate}<#t>
~<#t>
<#-- redeemRecords -->
0<#t>
~
and my test case ExtractFile.txt contains below record.
3~08DEC2018~0~
when doing assert on extractFile.txt and record created by footer.ftl i am getting below error as /n is getting added in extractFile.txt but not in footer.ftl
java.lang.AssertionError: Invalid extract file content
Expected: "3~08DEC2018~0~\n"
but: was "3~08DEC2018~0~"
Here \n is missing.
When running on unix i am getting above error.
when running locally it appends \r\n. \r seems to be because of env difference though so we can ignore it.
Any help on this please.
Tried to add custom Logback pattern in order to log Exception stacktraces into a single line (where new line character is replaced with unicode Line separator \u2028), as:
%date{"yyyy-MM-dd'T'HH:mm:ss.SSSZ", UTC} %5p %t %c{5}:%L [log_framework=logback;app_name=${APP_NAME};app_version=${APP_VERSION};instance_id=${CF_INSTANCE_INDEX}] %m MULTIEXCEPTION %replace(%xException){'\n','\u2028'}%nopex%n
Note: See spring-config.xml config file in Github
In the console, \n is replaced, the Exception stacktrace is in one line but, instead with "Line separator" character (\u2028), \n is replaced with the string "u2028".
If I try to log directly this "Line separator" character (via Logback, as log message) - it is printed in the console correctly.
What could be the problem?
I've managed to do this by entering the "Line separator" unicode character (
) directly:
%date{"yyyy-MM-dd'T'HH:mm:ss.SSSZ", UTC} %5p %t %c{5}:%L [log_framework=logback;app_name=${APP_NAME};app_version=${APP_VERSION};instance_id=${CF_INSTANCE_INDEX}] %m MULTIEXCEPTION %replace(%xException){'\n','
'}%nopex%n
Note: You can also manage to make Exception stacktraces "single-lined" in Spring Boot application by adding next application property:
logging.exception-conversion-word: "%replace(%xException){'\\n','\u2028'}%nopex"
I'm having a hard time dealing with some parsing issues related to Emojis.
I have a json requested through the brandwatch site using urllib.(1) Then, I must decode it in utf-8, however, when I do so, I'm getting surrogate keys and the json.loader cannot deal with them. (2)
I've tried to use BeautifulSoup4, which works great, however, when there's a " on the site result, it is transformed to ", and then, the json.loader cannot deal with it for it says that a , is missing. After tons of search, I gave up trying to escape the " which would be the ideal.(3)
So now, I'm stuck with both "solutions/problems". Any ideas on how to proceed?
Obs: This is a program that fetchs data from the brandwatch and put it inside an MySQL database. So performance is an issue here.
Obs2: PyJQ is a JQ for Python with does the request and I can change the opener.
(1) - Dealing with the first approach using urllib, these are the relevants parts of the code used for it:
def downloader(url):
return json.loads(urllib.request.urlopen(url).read().decode('utf8'))
...
parsed = pyjq.all(jqparser,url=url, vars={"today" : start_date}, opener=downloader)
Error Casted:
Exception ignored in: '_pyjq.pyobj_to_jv'
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 339: surrogates not allowed
*** Error in `python': munmap_chunk(): invalid pointer: 0x00007f5f806303f0 ***
If I print the result of urllib.request.urlopen(url).read().decode('utf8') instead of sending it to json.loader, that's what appears. These keys seems to be Emojis.
"fullname":"Botinhas\uD83D\uDC62"
(2) Dealing with the second approach using BeautifulSoup4, here's the relevant part of the code. (Same as above, just changed the downloader function)
def downloader(url):
return json.loads(BeautifulSoup(urllib.request.urlopen(url), 'lxml').get_text())
...
parsed = pyjq.all(jqparser,url=url, vars={"today" : start_date}, opener=downloader)
And this is the error casted:
Expecting ',' delimiter: line 1 column 4814765 (char 4814764)
Doing the print, the " before Diretas Já should be escaped.
"title":"Por "Diretas Já", manifestações pelo país ocorrem em preparação ao "Ocupa Brasília" - Sindicato dos Engenheiros no Estado do Rio de Janeiro"
I've thought of running a regex, however, I'm not sure whether this would be the most appropriate solution to this case as performance is an issue.
(3) - Part of Brandwatch result with the " problem mentioned above
UPDATE:
As Martin stated in the comments, I ran a replace swapping " for nothing. Then, it raised the former problem, of the emoji.
Exception ignored in: '_pyjq.pyobj_to_jv'
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 339: surrogates not allowed
*** Error in `python': munmap_chunk(): invalid pointer: 0x00007f5f806303f0 ***
UPDATE2:
I've added this to the downloader function:
re.sub(r'\\u(d|D)([a-z|A-Z|0-9]{3})', "", urllib.request.urlopen(url).read().decode('utf-8','ignore'))
It solved the issue, however, I don't think it's the best way to solve it. If anybody knows a better option.
I am using EMBOSSwin's needle() command line function which performs pairwise global alignments, but I encounter a strange warning.
So I have 24 pairs of amino acid sequences that need aligning, I run the needle() command from python using "subprocess.call()" - whilst this process occurs (seemingly smoothly) I get the following warning:
Warning: Sequence character string not found in ajSeqCvtKS
EXTRA CLUES:
Albeit this strange warning... alignments are successfully generated by needle() in .fasta format as you can see...
... BUT... I am getting unexplained "AssertionErrors" when trying to read these alignments back into python - using biopython's AlignIO.read() function (see: http://bit.ly/1aHK9w7 for my question directly related to this AssertionError)...
*To be clear: these AlignIO() AssertionErrors may not be related to the needle() warning, but I am treating the warning as a prime lead in the investigation...!
Whats the error in this thing -:
var decodeChars:Vector.<int> = new <int>[-1, -1, -1, -1, -1];
I get four complier errors three saying that "1084: Syntax error: expecting rightbrace before end of program." and the fourth saying that "1100: Syntax error: XML does not have matching begin and end tags.".
Whats the actual problem? thanks for help
Your code appears to be properly formed as demonstrated at http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/Vector.html#Vector()
1: Turn on debugging mode before compiling (Publish Settings > Flash > Permit debugging). From the errors given, It doesn't sound like this line is the cause of the issue. Debugging mode will tell you which line is throwing errors.
2: As The_asMan already mentioned, 1084 is indicating that you have a shortage of close braces. Make sure you properly indent your code, and this issue should be apparant.
3: 1100 is indicating that an XML file you loaded is malformed. Run your XML through a syntax validator such as http://validator.w3.org/