Jq variable add extra \ - json

Hi i have this bash code
#!/bin/bash
textb="\n 1 \n 2 \n 3 \n 4"
jq --arg textb "$textb" '. | {plain_text: ( $textb + desc.envi )'
When i run the comand this giveme the next example
#!/bin/bash
\\n1 \\n2 \\n3 \\n4
Why jq add and extra "\"? What i m going wrong?
I try some like this
textb="\n" 1 "\n" 2 "\n" 3 "\n" 4"
But i have this result
n1 n2 n3 n4
Thx


\n does not mean linefeed/newline in a bash double quoted string. It's just backslash+lowercase n.
If you use linefeeds instead of backslashes and Ns, they will encode the way you want:
textb="
1
2
3
4"
jq -n --arg textb "$textb" '."the string is:" = $textb'
outputs:
{
"the string is:": "\n1\n2\n3\n4"
}
Here are few other equivalent ways of putting literal linefeeds into a bash variable:
textb=$'\n1\n2\n3\n4'
textb=$(printf '\n%s' {1..4})

Related

Is there a fix for this Expression to get the key value pairs right

I have this python script which prints this output in raw string format.
{"A":"ab3241c","B":"d12e31234f","c":"[g$x>Q)M&.N+v8"}
I am using jq to set the values of A,B and c
Something like this
expression='. | to_entries | .[] | .key + "=\"" + .value + "\""'
eval "$(python script.py | jq -r "$expression")"
This works fine A and B. And when I do something like
echo $A
ab3241c
But the problem is with c where i get the output as
[g\u003eQ)M\u0026.N+v8
so $x> and & are getting converted to unicode.
Can I fix the expression to avoid this?
I fixed it using
expression as
expression='to_entries | map("\(.key)=\(.value | #sh)") | .[]'

Using eval here is like shooting yourself in the foot.
Why not just pipe the output of your python command to the jq command?
Consider:
function mypython {
cat <<"EOF"
{"A":"ab3241c","B":"d12e31234f","c":"[g$x>Q)M&.N+v8"}
EOF
}
expression='to_entries[] | .key + "=\"" + .value + "\""'
mypython | jq -r "$expression"
Note that using expression here seems pointless as well. In general, it would be better either to "inline" it, or put it in a file and use jq's -f command-line option.
(Notice also that the first "." in your expression is not needed.)

If you really must use eval, then consider:
function mypython {
cat <<"EOF"
{"A":"ab3241c","B":"d12e31234f","c":"[g$x>Q)M&.N+v8"}
EOF
}
expression='to_entries[] | .key + \"=\\\"\" + .value + \"\\\"\"'
eval "mypython | jq -r \"$expression\""
Output
A="ab3241c"
B="d12e31234f"
c="[g$x>Q)M&.N+v8"

Arithmetic in web scraping in a shell

so, I have the example code here:
#!/bin/bash
clear
curl -s https://www.cnbcindonesia.com/market-data/currencies/IDR=/USD-IDR |
html2text |
sed -n '/USD\/IDR/,$p' |
sed -n '/Last updated/q;p' |
tail -n-1 |
head -c+6 && printf "\n"
exit 0
this should print out some number range 14000~15000
lets start from the very basic one, what I have to do in order to print result + 1 ? so if the printout is 14000 and increment it to 1 become 14001. I suppose the result of the html2text is not calculatable since it should be something like string output not integer.
the more advance thing i want to know is how to calculate the result of 2 curl results?

What I would do, bash + xidel:
$ num=$(xidel -se '//div[#class="mark_val"]/span[1]/text()' 'https://url')
$ num=$((${num//,/}+1)) # num was 14050
$ echo $num
Output
14051
 Explanations
$((...))
is an arithmetic substitution. After doing the arithmetic, the whole thing is replaced by the value of the expression. See http://mywiki.wooledge.org/ArithmeticExpression
Command Substitution: "$(cmd "foo bar")" causes the command 'cmd' to be executed with the argument 'foo bar' and "$(..)" will be replaced by the output. See http://mywiki.wooledge.org/BashFAQ/002 and http://mywiki.wooledge.org/CommandSubstitution
Bonus
You can compute directly in xidel, thanks Reino using xquery syntax :
$ xidel -s <url> e 'replace(//div[#class="mark_val"]/span[1],",","") + 1'
And to do addition arithmetic of 2 values :
$ xidel -s <url> -e '
let $num:=replace(//div[#class="mark_val"]/span[1],",","")
return $num + $num
'

unescape backslash in jq output

https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/cid/5317139/property/IsomericSMILES/JSON
For the above JSON, the following jq prints 5317139 CCC/C=C\\1/C2=C(C3C(O3)CC2)C(=O)O1.
.PropertyTable.Properties
| .[]
| [.CID, .IsomericSMILES]
| #tsv
But there are two \ before the first 1. Is it wrong, should three be just one \? How to get the correct number of backslash?

The extra backslash in the output is the result of the request to produce TSV, since "\" has a special role to play in jq's TSV (e.g. "\t" signifies the tab character).
By contrast, consider:
jq -r '
.PropertyTable.Properties
| .[]
| [.CID, .IsomericSMILES]
| join("\t")' smiles.json
5317139 CCC/C=C\1/C2=C(C3C(O3)CC2)C(=O)O1

How do I print line number with the output for JQ in json file

I know there is a -n option and tried many combinations but couldn't get it to work. I'd like to print the line number and the length of the each line for the json file
cat -n traffictest.json | jq '. |length'
jq -C . | cat -n traffictest.json | jq '. |length'

jq has a built-in filter, input_line_number, which emits the line number of the input being read. For example, given this input:
[1,2]
"abcd"
{"a":1,"b":1,"c":1,"d":1, "e":1}
the invocation:
jq -r "\(input_line_number): \(length)"
yields:
1: 2
3: 4
5: 5

If you are just interested in line-number and length, I would use awk instead, e.g.:
awk '{ print NR, length, $0 }' traffictest.json
Or if you want to keep the syntax highlighting:
paste <(jq . traffictest.json | awk '{ print NR, length }' OFS='\t') \
<(jq -C . traffictest.json)

Convert ps aux to json

I try to convert the output of ps aux into Json format without using Perl or Python! For these I have read about jq. But I have success to convert the commandline output into json.
How to convert a simpe ps aux to Json?

ps aux | awk '
BEGIN { ORS = ""; print " [ "}
{ printf "%s{\"user\": \"%s\", \"pid\": \"%s\", \"cpu\": \"%s\"}",
separator, $1, $2, $3
separator = ", "
}
END { print " ] " }';
Just adjust columns which you need from ps aux output.

jq can read non-JSON input. You'll want to pre-process the input with awk first:
ps aux |
awk -v OFS=, '{print $1, $2}' |
jq -R 'split(",") | {user: .[0], pid: .[1]}'
If you want an array instead of a sequence of objects, pipe the output through jq --slurp 'add'. (I swear there's a way to do that without an extra call to jq, but it escapes me at the moment.)

Here's an only-jq solution based on tokenization.
Tokenization can be done using:
def tokens:
def trim: sub("^ +";"") | sub(" +$";"");
trim | splits(" +");
For illustration and brevity, let's consider only the first 10 tokens:
[tokens] | .[0:9]
Invocation:
$ ps aux | jq -c -R -f tokens.jq
Or as a one-liner, you could get away with:
$ ps aux | jq -cR '[splits(" +")] | .[0:9]'
First few lines of output:
["USER","PID","%CPU","%MEM","VSZ","RSS","TT","STAT","STARTED"]
["p","1595","55.9","0.4","2593756","32832","??","R","24Jan17"]
["p","12472","26.6","12.6","4951848","1058864","??","R","Sat01AM"]
["p","13239","10.9","1.5","4073756","128324","??","R","Sun12AM"]
["p","12482","7.8","1.2","3876628","101736","??","R","Sat01AM"]
["p","32039","7.7","1.4","4786968","118424","??","R","12Feb17"]
["_windowserver","425","7.6","0.8","3445536","65052","??","Ss","24Jan17"]
Using the headers as object keys
See e.g.
https://github.com/stedolan/jq/wiki/Cookbook#convert-a-csv-file-with-headers-to-json

I have a gist that to convert ps output to json. It uses jq under the covers so you need to install that. But you do not need to know jq

Dumps the fields specified by the -o flag as an array of PID objects:
ps ax -o "stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,wchan,sz,pcpu,command" \
| jq -sRr ' sub("\n$";"") | split("\n") | ([.[0]|splits(" +")]) as $header | .[1:] | [.[] | [. as $x | range($header|length) | {"key": $header[.], "value": (if .==($header|length-1) then ([$x|splits(" +")][.:]|join(" ")|tojson|.[1:length-1]) else ([$x|splits(" +")][.]) end) } ] | from_entries]'
This builds an array of the header fields, maps an array of {key, value} objects per output object, and then uses the built-in from_entries filter to object-aggregate these into the outputs.