shell script parsing issue: unexpected INVALID_CHARACTER (Windows cmd shell quoting issues?) - json

Question

What I am doing?
I have one JSON file as sonar-report.json. I want to iterate sonar-report.json in shell script, to read values of json.
To parse JSON file I am using jq https://stedolan.github.io/jq/
So Following code I was trying to execute in shell script
alias jq=./jq-win64.exe
for key in $(jq '.issues | keys | .[]' sonar-report.json); do
echo "$key"
line=$(jq -r ".issues[$key].line" sonar-report.json)
done
Problem
When i execute this, console give me error:
jq: error: syntax error, unexpected INVALID_CHARACTER (Windows cmd shell quoting issues?) at <top-level>, line 1:
If I update my above script, and add static index of array then script works fine
alias jq=./jq-win64.exe
for key in $(jq '.issues | keys | .[]' sonar-report.json); do
echo "$key"
line0=$(jq -r ".issues[0].line" sonar-report.json)
line1=$(jq -r ".issues[1].line" sonar-report.json)
done
so at the end what i want :
I want to iterate values and print in console like
alias jq=./jq-win64.exe
for key in $(jq '.issues | keys | .[]' sonar-report.json); do
line=$(jq -r ".issues[$key].line" sonar-report.json)
echo $line
done
so the output should be
15
This is my JSON file as sonar-report.json
{
"issues": [
{
"key": "016B7970D27939AEBD",
"component": "bits-and-bytes:src/main/java/com/catalystone/statusreview/handler/StatusReviewDecisionLedHandler.java",
"line": 15,
"startLine": 15,
"startOffset": 12,
"endLine": 15,
"endOffset": 14,
"message": "Use the \"equals\" method if value comparison was intended.",
"severity": "MAJOR",
"rule": "squid:S4973",
"status": "OPEN",
"isNew": true,
"creationDate": "2019-06-21T15:19:18+0530"
},
{
"key": "AWtqCc-jtovxS8PJjBiP",
"component": "bits-and-bytes:src/test/java/com/catalystone/statusreview/service/StatusReviewInitiationSerivceTest.java",
"message": "Fix failing unit tests on file \"src/test/java/com/catalystone/statusreview/service/StatusReviewInitiationSerivceTest.java\".",
"severity": "MAJOR",
"rule": "common-java:FailedUnitTests",
"status": "OPEN",
"isNew": false,
"creationDate": "2019-06-18T15:32:08+0530"
}
]
}
please help me, Thanks in advance

Answer 1

This looks to me like an instance of Windows/Unix line-ending incompatibility, indicated in jq bugs 92 (for Cygwin) and 1870 (for MSYS2).
Any of the workarounds indicated in those bug reports should work, but once the fix gets into the release binary (presumably v1.7), the simplest solution is to use the new -b command-line option. (The option is available in recent jq preview builds; see the second bug report listed above):
for key in $(jq -b '.issues | keys | .[]' sonar-report.json); do
line=$(jq -rb ".issues[$key].line" sonar-report.json)
# I added quotes in the next line, because it's better style.
echo "$line"
done
Until the next version of jq is available, or if you don't want to upgrade for some reason, a good workaround is to just remove the CRs by piping the output of jq through tr -d '\r':
for key in $(jq -'.issues | keys | .[]' sonar-report.json | tr -d '\r'); do
line=$(jq -r ".issues[$key].line" sonar-report.json | tr -d '\r')
echo "$line"
done
However, as pointed out in a comment by Cyrus, you probably don't need to iterate line-by-line in a shell loop, which is incredibly inefficient since it leads to reparsing the entire JSON input many times. You can use jq itself to iterate, with the much simpler:
jq '.issues[].line' solar-response.json
which will parse the JSON file just once, and then produce each .line value in the file. (You probably still want to use the -b command-line option or other workaround, depending on what you intend to do with the output.)