I have this JSON file:
{
"system.timestamp": "{system.timestamp}",
"error.state": "{error.state}",
"system.timestamp": "{system.timestamp}",
"error.state": "{error.state}",
"system.timestamp": "{system.timestamp}",
"error.state": "{error.state}",
"error.content": "{custom.error.content}"
}
I would like to get only the last object of the JSON file as I need to check that in every case, the last object is error.content. Attached part of code is just a sample file, every file that will be generated in reality will contain at least around 40 to 50 objects, so in every case I need to check that the last object is error.content.
I have calculated the length by using jq '. | length'. How do I do it using the jq command in Linux?
Note: it's a plain JSON file without any arrays.
Objects with duplicate keys can be handled in jq using the --stream option, e.g.:
$ jq -s --stream '.[length-2] | { (.[0][0]): (.[1]) }' input.json
{
"error.content": "{custom.error.content}"
}
For large files, the following would probably be better as it avoids "slurping" the input file:
$ jq 'first(tostream) | {(.[0][0]): .[1]} ' input.json
Related
I am having trouble accessing bash variable inside 'jq'.
The snippet below shows my bash loop to check for missing keys in a Json file.
#!/bin/sh
for key in "key1" "key2.key3"; do
echo "$key"
if ! cat ${JSON_FILE} | jq --arg KEY "$key" -e '.[$KEY]'; then
missingKeys+=${key}
fi
done
JSON_FILE:
{
"key1": "val1",
"key2": {
"key3": "val3"
}
}
The script works correctly for top level keys such as "key1". But it does not work correctly (returns null) for "key2.key3".
'jq' on the command line does return the correct value
cat input.json | jq '.key2.key3'
"val3"
I followed answers from other posts to come to this solution. However can't seem to figure out why it does not work for nested json keys.
Using --arg prevents your data from being incorrectly parsed as syntax. Usually, a shell variable you're passing into jq contains literal data, so this is the correct thing.
In this case, your variable contains syntax, not literal data: The . isn't part of the string you want to do a lookup by, but is instead an instruction to jq to do two separate lookups one after the other.
So, in this case, you should do the more obvious thing, instead of using --arg:
jq -e ".$KEY"
I have a below file in txt format. I want to arrange the data in json array format in linux and append more such data with for/while loop in the same json array based on condition. Please help me with the best way to achieve this.
File
Name:Rock
Name:Clock
{“Array" :[
{
"Name": "Rock",
},
{
"Name”: "Clock”,
}
]
Suppose your initial file is object.json and that it contains an empty object, {};
and that at the beginning of each iteration, the key:value pairs are defined in another file, kv.txt.
Then at each iteration, you can update array.json using the invocation:
< kv.txt jq -Rn --argfile object object.json -f program.jq | sponge object.json
where program.jq contains the jq program:
$object | .Array +=
reduce inputs as $in ([]; . + [$in | capture("(?<k>^[^:]*): *(?<v>.*)") | {(.k):.v} ])
(sponge is part of the moreutils package. If it cannot be used, then you will have to use another method of updating object.json.)
I have the date in the file which looks like
test,test
test1,test1
I want to convert it into like:
{"test":"test","test1":"test1"}
I have tried jq for this purpose jq -R -s -c 'split("\n")'
But its oupting in the format ["test,test","test1,test1",""]
jq 1.5 has inputs, which allows a simple and efficient solution:
jq -R -n -c '[inputs|split(",")|{(.[0]):.[1]}] | add' input.txt
Important: don't forget the -n (--null-input) option, otherwise you'll lose the first line.
Alternative
If your jq does not have inputs, then it's time to upgrade if at all possible. Otherwise:
jq -R -s '
split("\n")
| map(if index(",") then split(",")|{(.[0]):.[1]}
else empty end)
| add' input.txt
As #peak indicates, use the inputs with the split function. But to merge the key/values into one single object, use the reduce method:
jq -Rn '[inputs|split(",")| {(.[0]): .[1]}] | reduce .[] as $obj ({}; . + $obj) ' input.csv
The reduce method reduces each item in the array into a single item. In this case, we indicate that each item should be assigned to the $obj variable, and that we start out with the empty {} object. The second argument to the reduce method indicates how to "reduce" things down to a single item. In this case, we are adding/merging the $obj we assigned with the {} object we started out with and then returning the resulting object to be used in the next iteration. After all the iterations have completed, the final item (in this case, the combined object) is returned.
What you ask is possible to achieve with just standar unix shell utilities (assuming your input in file.txt):
bash $ echo { \"$(<file.txt sed 's/,/":"/g' | paste -s -d, - | sed 's/,/","/g')\" }
{ "test":"test","test1":"test1" }
bash $
resulting output is a valid json
I have a problem in working JSON file. I launch curl in AutoIt sciript to download a json file from web and then convert it to csv format by jq-win
jq-win32 -r ".[]" -c class.json>class.txt
and the json is in the following format:
[
{
"id":"1083",
"name":"AAAAA",
"channelNumber":8,
"channelImage":""},
{
"id":"1084",
"name":"bbbbb",
"channelNumber":7,
"channelImage":""},
{
"id":"1088",
"name":"CCCCCC",
"channelNumber":131,
"channelImage":""},
{
"id":"1089",
"name":"DDD,DDD",
"channelNumber":132,
"channelImage":""},
]
after jq-win, the file should become:
{"id":"1083","name":"AAAAA","channelNumber":8,"channelImage":""}
{"id":"1084","name":"bbbbb","channelNumber":7,"channelImage":""}
{"id":"1088","name":"CCCCCC","channelNumber":131,"channelImage":""}
{"id":"1089","name":"DDD,DDD","channelNumber":132,"channelImage":""}
and then the csv file will be further process by the AutoIt script and become:
AAAAA,1083
bbbbb,1084
CCCCCC,1088
DDD,DDD,1089
The json has around 300 records and among them, 5~6 record has comma in it eg DDD,DDD
so when I tried read in the csv file by _FileReadToArray, the comma in DDD,DDD cause trouble.
My question is: can I replace comma in the field using jq-win ?
(I tried use fart.exe but it will replace all comma in json file which is not suitable for me.)
Thanks a lot.
Regds
LAM Chi-fung
can I replace comma in the field using jq-win ?
Yes. For example, use gsub, pretty much as you’d use awk’s gsub, e.g.
gsub(","; "|")
If you want more details, please provide more details as per [mcve].
Example
With the given JSON input, the jq program:
.[]
| .name |= gsub(",";";")
| [.[]]
| map(tostring)
| join(",")
yields:
1083,AAAAA,8,
1084,bbbbb,7,
1088,CCCCCC,131,
1089,DDD;DDD,132,
I have been using the wonderful JQ library to parse and extract JSON data to facilitate re-importing. I am able to extract a range easily enough, but am unsure as to how you could loop through in a script and detect the end of the file, preferably in a bash or fish shell script.
Given a JSON file that is wrapped in a "results" dictionary, how can I detect the end of the file?
From testing, I can see that I will get an empty array nested in my desired structure, but how can you detect the end of file condition?:
jq '{ "results": .results[0:500] }' Foo.json > 0000-0500/Foo.json
Thanks!
I'd recommend using jq to split-up the array into a stream of the JSON objects you want (one per line), and then using some other tool (e.g. awk) to populate the files. Here's how the first part can be done:
def splitup(n):
def _split:
if length == 0 then empty
else .[0:n], (.[n:] | _split)
end;
if n == 0 then empty elif n > 0 then _split else reverse|splitup(-n) end;
# For the sake of illustration:
def data: { results: [range(0,20)]};
data | .results | {results: splitup(5) }
Invocation:
$ jq -nc -f splitup.jq
{"results":[0,1,2,3,4]}
{"results":[5,6,7,8,9]}
{"results":[10,11,12,13,14]}
{"results":[15,16,17,18,19]}
For the second part, you could (for example) pipe the jq output to:
awk '{ file="file."++n; print > file; close(file); }'
A variant you might be interested in would have the jq filter emit both the filename and the JSON on alternate lines; the awk script would then read the filename as well.