Need to find the difference between TradeCloseTime and TradeOpenTime time in dd:hh:mm format for the Exposure column in the following script.
Also the script runs super slow (~4 mins for 800 rows of json, on Core i7 16gb RAM machine)
#!/bin/bash
echo "TradeNo, TradeOpenType, TradeCloseType, TradeOpenSource, TradeCloseSource, TradeOpenTime, TradeCloseTime, PNL, Exposure" > tradelist.csv
tradecount=$(jq -r '.performance.numberOfTrades|tonumber' D.json)
for ((i=0; i<$tradecount; i++))
do
tradeNo=$(jq -r '.trades['$i']|[.tradeNo][]|tonumber' D.json)
entrySide=$(jq -r '.trades['$i'].orders[0]|[.side][]' D.json)
exitSide=$(jq -r '.trades['$i'].orders[1]|[.side][]' D.json)
entrySource=$(jq -r '.trades['$i'].orders[0]|[.source][]' D.json)
exitSource=$(jq -r '.trades['$i'].orders[1]|[.source][]' D.json)
tradeEntryTime=$(jq -r '.trades['$i'].orders[0]|[.placedTime][]' D.json | tr -d 'Z' | tr -s 'T' ' ')
tradeExitTime=$(jq -r '.trades['$i'].orders[1]|[.placedTime][]' D.json | tr -d 'Z' | tr -s 'T' ' ')
profitPercentage=$(jq -r '(.trades['$i']|[.profitPercentage][0]|tonumber)*(100)' D.json)
echo $tradeNo","$entrySide","$exitSide","$entrySource","$exitSource","$tradeEntryTime","$tradeExitTime","$profitPercentage | tr -d '"' >> tradelist.csv
done
json file looks like this
{"market":{"exchange":"BINANCE_FUTURES","coinPair":"BTC_USDT"},"strategy":{"name":"","type":"BACKTEST","candleSize":15,"lookbackDays":6,"leverageLong":1.00000000,"leverageShort":1.00000000,"strategyName":"ABC","strategyVersion":35,"runNo":"002","source":"Personal"},"strategyParameters":[{"name":"DurationInput","value":"87.0"}],"openPositionStrategy":{"actionTime":"CANDLE_CLOSE","maxPerSignal":1.00000000},"closePositionStrategy":{"actionTime":"CANDLE_CLOSE","minProfit":"NaN","stopLossValue":0.07000000,"stopLossTrailing":true,"takeProfit":0.01290000,"takeProfitDeviation":"NaN"},"performance":{"startTime":"2019-01-01T00:00:00Z","endTime":"2021-11-24T00:00:00Z","startAllocation":1000.00000000,"endAllocation":3478.58904150,"absoluteProfit":2478.58904150,"profitPerc":2.47858904,"buyHoldRatio":0.62426630,"buyHoldReturn":4.57228387,"numberOfTrades":744,"profitableTrades":0.67833109,"maxDrawdown":-0.20924885,"avgMonthlyProfit":0.05242718,"profitableMonths":0.70370370,"avgWinMonth":0.09889897,"avgLoseMonth":-0.05275563,"startPrice":null,"endPrice":57623.08000000},"trades":[{"tradeNo":0,"profit":-5.48836165,"profitPercentage":-0.00549085,"accumulatedBalance":994.51163835,"compoundProfitPerc":-0.00548836,"orders":[{"side":"Long","placedTime":"2019-09-16T21:15:00Z","placedAmount":0.09700000,"filledTime":"2019-09-16T21:15:00Z","filledAmount":0.09700000,"filledPrice":10300.49000000,"commissionPaid":0.39965901,"source":"SIGNAL"},{"side":"CloseLong","placedTime":"2019-09-17T19:15:00Z","placedAmount":0.09700000,"filledTime":"2019-09-17T19:15:00Z","filledAmount":0.09700000,"filledPrice":10252.13000000,"commissionPaid":0.39778264,"source":"SIGNAL"}]},{"tradeNo":1,"profit":-3.52735800,"profitPercentage":-0.00356403,"accumulatedBalance":990.98428035,"compoundProfitPerc":-0.00901572,"orders":[{"side":"Long","placedTime":"2019-09-19T06:00:00Z","placedAmount":0.10000000,"filledTime":"2019-09-19T06:00:00Z","filledAmount":0.10000000,"filledPrice":9893.16000000,"commissionPaid":0.39572640,"source":"SIGNAL"},{"side":"CloseLong","placedTime":"2019-09-19T06:15:00Z","placedAmount":0.10000000,"filledTime":"2019-09-19T06:15:00Z","filledAmount":0.10000000,"filledPrice":9865.79000000,"commissionPaid":0.39463160,"source":"SIGNAL"}]},{"tradeNo":2,"profit":-5.04965308,"profitPercentage":-0.00511770,"accumulatedBalance":985.93462727,"compoundProfitPerc":-0.01406537,"orders":[{"side":"Long","placedTime":"2019-09-25T10:15:00Z","placedAmount":0.11700000,"filledTime":"2019-09-25T10:15:00Z","filledAmount":0.11700000,"filledPrice":8430.00000000,"commissionPaid":0.39452400,"source":"SIGNAL"},{"side":"CloseLong","placedTime":"2019-09-25T10:30:00Z","placedAmount":0.11700000,"filledTime":"2019-09-25T10:30:00Z","filledAmount":0.11700000,"filledPrice":8393.57000000,"commissionPaid":0.39281908,"source":"SIGNAL"}]}
You can do it all (extracts, conversions and formatting) with one jq call:
#!/bin/sh
echo 'TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure'
query='
.trades[]
| [
.tradeNo,
.orders[0].side,
.orders[1].side,
.orders[0].source,
.orders[1].source,
(.orders[0].placedTime | fromdate | strftime("%Y-%m-%d %H:%M:%S")),
(.orders[1].placedTime | fromdate | strftime("%Y-%m-%d %H:%M:%S")),
.profitPercentage * 100,
(
(.orders[1].placedTime | fromdate) - (.orders[0].placedTime | fromdate)
| (. / 86400 | floor | tostring) + (. % 86400 | strftime(":%H:%M"))
)
]
|#csv
'
jq -r "$query" < D.json > tradelist.csv
example of JSON (cleaned of all irrelevant keys):
{
"trades": [
{
"tradeNo": 0,
"profitPercentage": -0.00549085,
"orders": [
{
"side": "Long",
"placedTime": "2018-12-16T21:34:46Z",
"source": "SIGNAL"
},
{
"side": "CloseLong",
"placedTime": "2019-09-17T19:15:00Z",
"source": "SIGNAL"
}
]
}
]
}
output:
TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure
0,"Long","CloseLong","SIGNAL","SIGNAL","2018-12-16 21:34:46","2019-09-17 20:15:00",-0.549085,"274:22:40"
If you want to get rid of the double quotes that jq adds when generating a CSV (which are completely valid, but you need a real parser to read the CSV) then you can replace #csv with #tsv and post-process the output with tr '\t' ',', like this:
query='
...
|#tsv
'
jq -r "$query" < D.json | tr '\t' ',' > tradelist.csv
and you'll get:
TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure
0,Long,CloseLong,SIGNAL,SIGNAL,2018-12-16 21:34:46,2019-09-17 20:15:00,-0.549085,274:22:40
note: This method of getting rid of the " in the CSV is only accurate when there is no \n \t \r \ , or " characters in the input data.
Regarding the main question (regarding computing time differences), you're in luck as jq provides the built-in function fromdateiso8601 for converting ISO times to "the
number of seconds since the Unix epoch (1970-01-01T00:00:00Z)".
With your JSON sample,
.trades[]
| [ .orders[1].placedTime, .orders[0].placedTime]
| map(fromdateiso8601)
| .[0] - .[1]
produces the three differences:
79200
900
900
And here's a function for converting seconds to "hh:mm:ss" format:
def hhmmss:
def l: tostring | if length < 2 then "0\(.)" else . end;
(. % 60) as $ss
| ((. / 60) | floor) as $mm
| (($mm / 60) | floor) as $hh
| ($mm % 60) as $mm
| [$hh, $mm, $ss] | map(l) | join(":");
I prefer using an intermediate structure of the "entry" and "exit" JSON. This helps with debugging the jq commands. Formatted for readability over performance:
#!/usr/bin/env bash
echo "TradeNo,TradeOpenType,TradeCloseType,TradeOpenSource,TradeCloseSource,TradeOpenTime,TradeCloseTime,PNL,Exposure" > tradelist.csv
jq -r '
.trades[]
|{tradeNo,
profitPercentage,
entry:.orders[0],
exit:.orders[1],
entryTS:.orders[0].placedTime|fromdate,
exitTS:.orders[1].placedTime|fromdate}
|[.tradeNo,
.entry.side,
.exit.side,
.entry.source,
.exit.source,
(.entry.placedTime|strptime("%Y-%m-%dT%H:%M:%SZ")|strftime("%Y-%m-%d %H:%M:%S")),
(.exit.placedTime|strptime("%Y-%m-%dT%H:%M:%SZ")|strftime("%Y-%m-%d %H:%M:%S")),
(.profitPercentage*100),
(.exitTS-.entryTS|todate|strptime("%Y-%m-%dT%H:%M:%SZ")|strftime("%d:%H:%M"))]|#csv
' D.json | tr -d '"' >> tradelist.csv
WARNING: This formatting assumes Exposure is LESS THAN 1 MONTH. Good luck with that!
Hi I have two file for example:
file1.json
{
"id": "001",
"name" : "my_policy",
"list_1": ["111111111", "22222222","33333333"],
"list_2": ["a", "b","c"],
.....
}
Then I have file2.json (not always has the same field as f1)
{
"list_1": ["111111111","111111122","33333333"],
"list_2": ["a", "b","c","d","e"],
.....
}
How I can via jq merge same keys values in the two file json and in addiction to the merge operation remove from file1 keys the values non present in file2 ?
So get this result:
{
"id": "001",
"policy" : "my_policy",
"list_1": ["111111111","111111122","33333333"],
"list_2": ["a", "b","c","d","e"],
.....
}
I solved the merge operation via:
jq -s 'reduce .[] as $item ({}; reduce ($item | keys_unsorted[]) as $key (.; $item[$key] as $val | ( $val | type) as $ type | .[$key] = if ( $type == "array") then (.[$key] + $val | unique) elif ($type == "object") then (.[$key] + $val) else $val end ))' file1.json f2.json
How I can solve? Or is impossible via jq?
It is quite simple once you figure out how to find the difference among two list items and add/unique them. One way would be to
jq --slurpfile s 2.json -f script.jq 1.json
Where my script contents are
#!/usr/bin/env jq -f
# backup list_1, list_2 from 1.json
.list_1 as $l1 |
.list_2 as $l2 |
# Perform the operation of removing file1 keys not present in 2.json
# for both list_1 and list_2
( ( $l1 - ( $l1 - $s[].list_1 ) ) + $s[].list_1 | unique ) as $f1 |
( ( $l2 - ( $l2 - $s[].list_2 ) ) + $s[].list_2 | unique ) as $f2 |
# Update the original result 1.json with the modified content
.list_1 |= $f1 |
.list_2 |= $f2
or directly from the command line as
jq --slurpfile s 2.json '
.list_1 as $l1 |
.list_2 as $l2 |
( ( $l1 - ( $l1 - $s[].list_1 ) ) + $s[].list_1 | unique ) as $f1 |
( ( $l2 - ( $l2 - $s[].list_2 ) ) + $s[].list_2 | unique ) as $f2 |
.list_1 |= $f1 |
.list_2 |= $f2
' 1.json
I have a lengthy JSON file and I execute the command to get the output shown below:
jq -s '.[]
| ."lrouter/show"[]
| del( . | select(.type == "TUNNEL-VRF"))
| del(.ports[] | select(.type == "blackhole" or .type == "cpu-port" or .type == "loopback"))
| "Name: \(.name)" ,
"UUID: \(.uuid)" ,
(.ports[] | {Port_Name: .name,
Port_Type: .type,
Port_Peer: .peer,
Port_IPs: .ips[],
Port_Admin_Up: .admin_up,
Port_Op_State: .op_state_up } )' ec-state
"Name: SR-t0-uplink"
"UUID: 23354d26-6994-46d9-b78c-bb565a1c13f2"
{
"Port_Name": "uplink",
"Port_Type": "uplink",
"Port_Peer": "d78089f6-71b5-4c8e-a477-69ee01f17c5c",
"Port_IPs": "1.1.13.5/24",
"Port_Admin_Up": true,
"Port_Op_State": true
}
{
"Port_Name": "bp-sr0-port",
"Port_Type": "backplane",
"Port_Peer": null,
"Port_IPs": "169.254.0.2/28",
"Port_Admin_Up": false,
"Port_Op_State": false
}
jq: error (at ec-state:1028): Cannot iterate over null (null)
I get the desired result however, I also get the jq error at the end of the result. Just curious to know what am I doing incorrectly with the query.
Since your input is large, you might consider adding assertions or equivalent. Since your program evidently expects arrays at various points, you could instrument it with a function such as:
def q($n; $msg):
if type == "array" or type == "object"
then .
else error("\($msg): composite expected # \($n) vs \(.)")
end;
Your program could then be instrumented as follows:
range(0;length) as $n
| .[$n]
| ."lrouter/show" | q($n; 2) | .[]
| del( . | select(.type == "TUNNEL-VRF"))
| del(.ports | q($n; 3) | .[] | select(.type == "blackhole" or .type == "cpu-port" or .type == "loopback"))
| "Name: \(.name)" ,
"UUID: \(.uuid)" ,
(.ports[] | {Port_Name: .name,
Port_Type: .type,
Port_Peer: .peer,
Port_IPs: (.ips | q($n; 4) |.[]),
Port_Admin_Up: .admin_up,
Port_Op_State: .op_state_up } )
Say we have the following JSON:
[
{
"dir-1": [
"file-1.1",
"file-1.2"
]
},
"dir-1",
{
"dir-2": [
"file-2.1"
]
}
]
And we want to get the next output:
"dir-1/file-1.1"
"dir-1/file-1.2"
"dir-1"
"dir-2/file-2.1"
i.e. to get the paths to all leafs, joining items with /. Is there a way to do that on JQ?
I tried something like this:
cat source-file | jq 'path(..) | [ .[] | tostring ] | join("/")'
But it doesn't produce what I need even close.
You could take advantage of how streams work by merging the path with their values. Streams will only emit path, value pairs for leaf values. Just ignore the numbered indices.
$ jq --stream '
select(length == 2) | [(.[0][] | select(strings)), .[1]] | join("/")
' source-file
returns:
"dir-1/file-1.1"
"dir-1/file-1.2"
"dir-1"
"dir-2/file-2.1"
Here is a solution similar to Jeff Mercado's which uses tostream and flatten
tostream | select(length==2) | .[0] |= map(strings) | flatten | join("/")
Try it online at jqplay.org
Another way is to use a recursive function to walk the input such as
def slashpaths($p):
def concat($p;$k): if $p=="" then $k else "\($p)/\($k)" end;
if type=="array" then .[] | slashpaths($p)
elif type=="object" then
keys_unsorted[] as $k
| .[$k] | slashpaths(concat($p;$k))
else concat($p;.) end;
slashpaths("")
Try it online at tio.run!
Using --stream is good but the following is perhaps less esoteric:
paths(scalars) as $p
| getpath($p) as $v
| ($p | map(strings) + [$v])
| join("/")
(If using jq 1.4 or earlier, and if any of the leaves might be numeric or boolean or null, then [$v] above should be replaced by [$v|tostring].)
Whether the result should be regarded as "paths to leaves" is another matter...