Translate JSON into Markdown list with jq - json

I've got a JSON structure for projects and tasks:
{
"project1": [
"task1a",
"task1b"
],
"project2": [
"task2a",
"task2b",
"task2c"
]
}
which I'd like to convert to MD-like list:
* project1
* task1a
* task1b
* project2
* task2a
* task2b
* task2c
mainly using jq and sh/bash
the best I can do is:
jq '. | to_entries | .[] | .value |= " * "+join("\n * ")+"\n" | .key = "* "+.key'
but then I'm stuck trying to properly print .value string along with .key


Yet another solution, without the update operator |=, without the join() builtin and without explicit newline characters \n:
jq -r 'to_entries[] | "* \(.key)", " * \(.value[])"'
* project1
* task1a
* task1b
* project2
* task2a
* task2b
* task2c
Demo


Another solution, using join() and string interpolation:
to_entries[] | "* \(.key)\n * \(.value | join("\n * "))"
Will produce:
* project1
* task1a
* task1b
* project2
* task2a
* task2b
* task2c
jq -r 'to_entries[] | "* \(.key)\n * \(.value | join("\n * "))"' input
JqPlay Demo


You're very close. Add -r to output raw strings instead of JSON, and use | .key, .value to output the key and value strings as separate items so they are shown on adjacent lines.
jq -r '. | to_entries | .[] | .value |= " * "+join("\n * ") | .key = "* "+.key | .key, .value'

Related

Bash script with jq wont get date difference from strings, and runs quite slowly on i7 16GB RAM

Need to find the difference between TradeCloseTime and TradeOpenTime time in dd:hh:mm format for the Exposure column in the following script.
Also the script runs super slow (~4 mins for 800 rows of json, on Core i7 16gb RAM machine)
#!/bin/bash
echo "TradeNo, TradeOpenType, TradeCloseType, TradeOpenSource, TradeCloseSource, TradeOpenTime, TradeCloseTime, PNL, Exposure" > tradelist.csv
tradecount=$(jq -r '.performance.numberOfTrades|tonumber' D.json)
for ((i=0; i<$tradecount; i++))
do
tradeNo=$(jq -r '.trades['$i']|[.tradeNo][]|tonumber' D.json)
entrySide=$(jq -r '.trades['$i'].orders[0]|[.side][]' D.json)
exitSide=$(jq -r '.trades['$i'].orders[1]|[.side][]' D.json)
entrySource=$(jq -r '.trades['$i'].orders[0]|[.source][]' D.json)
exitSource=$(jq -r '.trades['$i'].orders[1]|[.source][]' D.json)
tradeEntryTime=$(jq -r '.trades['$i'].orders[0]|[.placedTime][]' D.json | tr -d 'Z' | tr -s 'T' ' ')
tradeExitTime=$(jq -r '.trades['$i'].orders[1]|[.placedTime][]' D.json | tr -d 'Z' | tr -s 'T' ' ')
profitPercentage=$(jq -r '(.trades['$i']|[.profitPercentage][0]|tonumber)*(100)' D.json)
echo $tradeNo","$entrySide","$exitSide","$entrySource","$exitSource","$tradeEntryTime","$tradeExitTime","$profitPercentage | tr -d '"' >> tradelist.csv
done
json file looks like this
{"market":{"exchange":"BINANCE_FUTURES","coinPair":"BTC_USDT"},"strategy":{"name":"","type":"BACKTEST","candleSize":15,"lookbackDays":6,"leverageLong":1.00000000,"leverageShort":1.00000000,"strategyName":"ABC","strategyVersion":35,"runNo":"002","source":"Personal"},"strategyParameters":[{"name":"DurationInput","value":"87.0"}],"openPositionStrategy":{"actionTime":"CANDLE_CLOSE","maxPerSignal":1.00000000},"closePositionStrategy":{"actionTime":"CANDLE_CLOSE","minProfit":"NaN","stopLossValue":0.07000000,"stopLossTrailing":true,"takeProfit":0.01290000,"takeProfitDeviation":"NaN"},"performance":{"startTime":"2019-01-01T00:00:00Z","endTime":"2021-11-24T00:00:00Z","startAllocation":1000.00000000,"endAllocation":3478.58904150,"absoluteProfit":2478.58904150,"profitPerc":2.47858904,"buyHoldRatio":0.62426630,"buyHoldReturn":4.57228387,"numberOfTrades":744,"profitableTrades":0.67833109,"maxDrawdown":-0.20924885,"avgMonthlyProfit":0.05242718,"profitableMonths":0.70370370,"avgWinMonth":0.09889897,"avgLoseMonth":-0.05275563,"startPrice":null,"endPrice":57623.08000000},"trades":[{"tradeNo":0,"profit":-5.48836165,"profitPercentage":-0.00549085,"accumulatedBalance":994.51163835,"compoundProfitPerc":-0.00548836,"orders":[{"side":"Long","placedTime":"2019-09-16T21:15:00Z","placedAmount":0.09700000,"filledTime":"2019-09-16T21:15:00Z","filledAmount":0.09700000,"filledPrice":10300.49000000,"commissionPaid":0.39965901,"source":"SIGNAL"},{"side":"CloseLong","placedTime":"2019-09-17T19:15:00Z","placedAmount":0.09700000,"filledTime":"2019-09-17T19:15:00Z","filledAmount":0.09700000,"filledPrice":10252.13000000,"commissionPaid":0.39778264,"source":"SIGNAL"}]},{"tradeNo":1,"profit":-3.52735800,"profitPercentage":-0.00356403,"accumulatedBalance":990.98428035,"compoundProfitPerc":-0.00901572,"orders":[{"side":"Long","placedTime":"2019-09-19T06:00:00Z","placedAmount":0.10000000,"filledTime":"2019-09-19T06:00:00Z","filledAmount":0.10000000,"filledPrice":9893.16000000,"commissionPaid":0.39572640,"source":"SIGNAL"},{"side":"CloseLong","placedTime":"2019-09-19T06:15:00Z","placedAmount":0.10000000,"filledTime":"2019-09-19T06:15:00Z","filledAmount":0.10000000,"filledPrice":9865.79000000,"commissionPaid":0.39463160,"source":"SIGNAL"}]},{"tradeNo":2,"profit":-5.04965308,"profitPercentage":-0.00511770,"accumulatedBalance":985.93462727,"compoundProfitPerc":-0.01406537,"orders":[{"side":"Long","placedTime":"2019-09-25T10:15:00Z","placedAmount":0.11700000,"filledTime":"2019-09-25T10:15:00Z","filledAmount":0.11700000,"filledPrice":8430.00000000,"commissionPaid":0.39452400,"source":"SIGNAL"},{"side":"CloseLong","placedTime":"2019-09-25T10:30:00Z","placedAmount":0.11700000,"filledTime":"2019-09-25T10:30:00Z","filledAmount":0.11700000,"filledPrice":8393.57000000,"commissionPaid":0.39281908,"source":"SIGNAL"}]}

You can do it all (extracts, conversions and formatting) with one jq call:
#!/bin/sh
echo 'TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure'
query='
.trades[]
| [
.tradeNo,
.orders[0].side,
.orders[1].side,
.orders[0].source,
.orders[1].source,
(.orders[0].placedTime | fromdate | strftime("%Y-%m-%d %H:%M:%S")),
(.orders[1].placedTime | fromdate | strftime("%Y-%m-%d %H:%M:%S")),
.profitPercentage * 100,
(
(.orders[1].placedTime | fromdate) - (.orders[0].placedTime | fromdate)
| (. / 86400 | floor | tostring) + (. % 86400 | strftime(":%H:%M"))
)
]
|#csv
'
jq -r "$query" < D.json > tradelist.csv
example of JSON (cleaned of all irrelevant keys):
{
"trades": [
{
"tradeNo": 0,
"profitPercentage": -0.00549085,
"orders": [
{
"side": "Long",
"placedTime": "2018-12-16T21:34:46Z",
"source": "SIGNAL"
},
{
"side": "CloseLong",
"placedTime": "2019-09-17T19:15:00Z",
"source": "SIGNAL"
}
]
}
]
}
output:
TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure
0,"Long","CloseLong","SIGNAL","SIGNAL","2018-12-16 21:34:46","2019-09-17 20:15:00",-0.549085,"274:22:40"
If you want to get rid of the double quotes that jq adds when generating a CSV (which are completely valid, but you need a real parser to read the CSV) then you can replace #csv with #tsv and post-process the output with tr '\t' ',', like this:
query='
...
|#tsv
'
jq -r "$query" < D.json | tr '\t' ',' > tradelist.csv
and you'll get:
TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure
0,Long,CloseLong,SIGNAL,SIGNAL,2018-12-16 21:34:46,2019-09-17 20:15:00,-0.549085,274:22:40
note: This method of getting rid of the " in the CSV is only accurate when there is no \n \t \r \ , or " characters in the input data.

Regarding the main question (regarding computing time differences), you're in luck as jq provides the built-in function fromdateiso8601 for converting ISO times to "the
number of seconds since the Unix epoch (1970-01-01T00:00:00Z)".
With your JSON sample,
.trades[]
| [ .orders[1].placedTime, .orders[0].placedTime]
| map(fromdateiso8601)
| .[0] - .[1]
produces the three differences:
79200
900
900
And here's a function for converting seconds to "hh:mm:ss" format:
def hhmmss:
def l: tostring | if length < 2 then "0\(.)" else . end;
(. % 60) as $ss
| ((. / 60) | floor) as $mm
| (($mm / 60) | floor) as $hh
| ($mm % 60) as $mm
| [$hh, $mm, $ss] | map(l) | join(":");

I prefer using an intermediate structure of the "entry" and "exit" JSON. This helps with debugging the jq commands. Formatted for readability over performance:
#!/usr/bin/env bash
echo "TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure" > tradelist.csv
jq -r '
.trades[]
|{tradeNo,
profitPercentage,
entry:.orders[0],
exit:.orders[1],
entryTS:.orders[0].placedTime|fromdate,
exitTS:.orders[1].placedTime|fromdate}
|[.tradeNo,
.entry.side,
.exit.side,
.entry.source,
.exit.source,
(.entry.placedTime|strptime("%Y-%m-%dT%H:%M:%SZ")|strftime("%Y-%m-%d %H:%M:%S")),
(.exit.placedTime|strptime("%Y-%m-%dT%H:%M:%SZ")|strftime("%Y-%m-%d %H:%M:%S")),
(.profitPercentage*100),
(.exitTS-.entryTS|todate|strptime("%Y-%m-%dT%H:%M:%SZ")|strftime("%d:%H:%M"))]|#csv
' D.json | tr -d '"' >> tradelist.csv
WARNING: This formatting assumes Exposure is LESS THAN 1 MONTH. Good luck with that!

How to get selected values from JSON output

From the output shown, I was looking for solr : xcfrtgyhujikolpu in a shell script:
[{"username":"zookeeper","password":"sdfrtghuioklhq"},
{"username":"solr","password":"xcfrtgyhujikolpu"}]
I tried:
| jq -r .[].password

.[] | select( .username == $username ) | .username + " : " + .password
Used as
jq --arg username solr -r '.[] | select( .username == $username ) | .username + " : " + .password'
jqplay

How to convert json values to prometheus?

After runing this command:
mtr -jnbz www.google.com |jq .report.hubs|jq -r 'keys_unsorted[] as $k | "\(.[$k])"'
I get this result:
{"count":"1","host":"1.1.1.1","ASN":"AS???","Loss%":0,"Snt":10,"Last":36.28,"Avg":39.43,"Best":34.77,"Wrst":62.37,"StDev":8.15}
{"count":"2","host":"2.2.2.2","ASN":"AS???","Loss%":100,"Snt":10,"Last":0,"Avg":0,"Best":0,"Wrst":0,"StDev":0}
How can I get this result(prometheus format):
mtr_loss{"count"="1","host"="1.1.1.1","ASN"="AS???"} 0
mtr_snt{"count"="1","host"="1.1.1.1","ASN"="AS???"} 10
mtr_last{"count"="1","host"="1.1.1.1","ASN"="AS???"} 36.28
mtr_avg{"count"="1","host"="1.1.1.1","ASN"="AS???"} 39.43
mtr_best{"count"="1","host"="1.1.1.1","ASN"="AS???"} 34.77
mtr_wrst{"count"="1","host"="1.1.1.1","ASN"="AS???"} 67.37
mtr_stdev{"count"="1","host"="1.1.1.1","ASN"="AS???"} 8.15
And so on for the second string
I will be grateful to you for any tips and tricks
Regards

The following is perhaps easier to follow, maintain, and reuse:
# produce the {"k"="value", ...} representation:
def kv:
. as $in
| reduce keys_unsorted[] as $k ([]; . + ["\"\($k)\"=\"\($in[$k])\""] )
| join(",")
| "{" + . + "}" ;
# downcase and remove %
def ht($s):
keys_unsorted[] as $key
| .[$key] as $value
| "mtr_\($key|ascii_downcase|gsub("%";""))\($s) \($value)";
({count,host,ASN} | kv) as $s
| {"Loss%", Snt, Last, Avg, Best, Wrst, StDev}
| ht($s)

It was quite easy :) :
mtr -jnbz www.google.com |
tr -d "%" |
jq -r '.report.hubs[]
| "mtr_loss{count=\"\(.count)\",host=\"\(.host)\",asn=\"\(.ASN)\"} \(.Loss)\nmtr_sent{count=\"\(.count)\",host=\"\(.host)\",asn=\"\(.ASN)\"} \(.Snt)\nmtr_last{count=\"\(.count)\",host=\"\(.host)\",asn=\"\(.ASN)\"} \(.Last)\nmtr_avg{count=\"\(.count)\",host=\"\(.host)\",asn=\"\(.ASN)\"} \(.Avg)\nmtr_best{count=\"\(.count)\",host=\"\(.host)\",asn=\"\(.ASN)\"} \(.Best)\nmtr_wrst{count=\"\(.count)\",host=\"\(.host)\",asn=\"\(.ASN)\"} \(.Wrst)\nmtr_stdev{count=\"\(.count)\",host=\"\(.host)\",asn=\"\(.ASN)\"} \(.StDev)" '

jq to_entries string and number cannot be added

I'm not understanding how to_entries works in jq.
I have the following json payload in payload.json
{"REGION":"us-east-1","EMAIL":"contain","UPDATE":1}
which I want to convert into = delimited keypairs, like so;
REGION=us-east-1
EMAIL=contain
UPDATE=1
I was using
jq -r 'to_entries | .[] | .key + "=" + .value' < payload.json
But I get an error
jq: error (at <stdin>:0): string ("UPDATE=") and number (1) cannot be added
If I understand correctly, the issue is that the update value is a number, not a string (ie, having them not match types is an issue) so I tried the following, both with the same error;
string interpolation:
jq -r 'to_entries | .[] | (.key) + "=" + (.value)' < payload.json
tostring:
jq -r 'to_entries | .[] | .key + "=" + .value|tostring' < payload.json
What am I missing?

what am I missing?
A pair of parentheses:
.key + "=" + ( .value|tostring )
Alternatively, you could use string interpolation, e.g.
"\(.key)=\(.value)"

How to make paths to leafs of a JSON?

Say we have the following JSON:
[
{
"dir-1": [
"file-1.1",
"file-1.2"
]
},
"dir-1",
{
"dir-2": [
"file-2.1"
]
}
]
And we want to get the next output:
"dir-1/file-1.1"
"dir-1/file-1.2"
"dir-1"
"dir-2/file-2.1"
i.e. to get the paths to all leafs, joining items with /. Is there a way to do that on JQ?
I tried something like this:
cat source-file | jq 'path(..) | [ .[] | tostring ] | join("/")'
But it doesn't produce what I need even close.

You could take advantage of how streams work by merging the path with their values. Streams will only emit path, value pairs for leaf values. Just ignore the numbered indices.
$ jq --stream '
select(length == 2) | [(.[0][] | select(strings)), .[1]] | join("/")
' source-file
returns:
"dir-1/file-1.1"
"dir-1/file-1.2"
"dir-1"
"dir-2/file-2.1"

Here is a solution similar to Jeff Mercado's which uses tostream and flatten
tostream | select(length==2) | .[0] |= map(strings) | flatten | join("/")
Try it online at jqplay.org
Another way is to use a recursive function to walk the input such as
def slashpaths($p):
def concat($p;$k): if $p=="" then $k else "\($p)/\($k)" end;
if type=="array" then .[] | slashpaths($p)
elif type=="object" then
keys_unsorted[] as $k
| .[$k] | slashpaths(concat($p;$k))
else concat($p;.) end;
slashpaths("")
Try it online at tio.run!

Using --stream is good but the following is perhaps less esoteric:
paths(scalars) as $p
| getpath($p) as $v
| ($p | map(strings) + [$v])
| join("/")
(If using jq 1.4 or earlier, and if any of the leaves might be numeric or boolean or null, then [$v] above should be replaced by [$v|tostring].)
Whether the result should be regarded as "paths to leaves" is another matter...