Lets say I have an I/P json file as below. And I want to extract the O/P in a CSV format with the below fields. Specifically, I want to get the value of the key "Gamma" in the o/p if the key "Gamma" exists in "tags" map. If the key doesn't exists, it should just print a NULL value. The expected o/p is below.
generated_time,platform,id,,
2021-09-09:12:03:12,earth,2eeee67748,Ray,2021-08-25 09:41:06
2021-09-09:12:03:12,sun,xxxxx12334,NULL,2021-08-25 10:11:31
[
{
"generated_time": "generated_time",
"platform": "platform",
"id": "id"
},
{
"generated_time": "2021-09-09:12:03:12",
"platform": "earth",
"id": "2eeee67748",
"tags": {
"app": "map",
"Gamma": "Ray",
"null": [
"allow-all-humans"
]
},
"created": "2021-08-25 09:41:06"
},
{
"generated_time": "2021-09-09:12:03:12",
"platform": "sun",
"id": "xxxxx12334",
"tags": {
"component": "machine",
"environment": "hot",
"null": [
"aallow-all-humans"
]
},
"created": "2021-08-25 10:11:31"
}
]
jq has a builtin #csv which renders an array
as CSV with double quotes for strings, and quotes escaped by repetition.
If the additional quoting (as compared to your expected output) isn't an issue, the following
jq --raw-output '
# produce an array for each element in the input array
.[] | [
# containing the first three columns unchanged
.generated_time, .platform, .id,
# if the input element has a field named "tags"
if has("tags")
# then add two more columns and replace an inexistant Gamma with "NULL"
then (.tags.Gamma // "NULL", .created)
# otherwise add two empty columns instead
else (null, null) end
# and convert the array into CSV format
] | #csv
' input.json
will produce
"generated_time","platform","id",,
"2021-09-09:12:03:12","earth","2eeee67748","Ray","2021-08-25 09:41:06"
"2021-09-09:12:03:12","sun","xxxxx12334","NULL","2021-08-25 10:11:31"
Related
I have this JSON:
{
"time": "2022-02-28T22:00:55.196Z",
"severity": "INFO",
"params": [
{"key": "state", "value": "pending"},
{"key": "options", "value": "request"},
{"key": "description", "value": "[FILTERED]"}
],
"content_length": "231"
}
I want to print key value pairs of where the key matches to state and options and also the time and its value. I am able to print the time and all key value pairs by using below command, but not sure how to extract specific key value pairs.
jq '"time:\(.time)" ,[.params[] | "key:\(.key)" ,"value:\(.value)"]' test.json
This gives the output:
"time:2022-02-28T22:00:55.196Z"
[
"key:state",
"value:pending",
"key:options",
"value:request",
"key:description",
"value:[FILTERED]"
]
But my desired output is:
"time:2022-02-28T22:00:55.196Z"
"key:state",
"value:pending",
"key:options",
"value:request"
One solution to the stated problem would be:
< test.json jq '
"time:\(.time)",
[.params[] | select(.key|IN("state","options"))
| "key:\(.key)" ,"value:\(.value)"]
' | sed '/^[][]$/d'
However, it would almost certainly be better to modify the requirements slightly so that the output format is less idiosyncratic. This should also make it easier to formulate a cleaner (e.g. only-jq) solution.
You can use #csv (comma separated values).
Filter
"time:\(.time)",
(.params |
[map(select(.key=="state" or .key=="options"))[]
| "key:\(.key)", "value:\(.value)"]
| #csv)
Input
{
"time": "2022-02-28T22:00:55.196Z",
"severity": "INFO",
"params": [
{"key": "state", "value": "pending"},
{"key": "options", "value": "request"},
{"key": "description", "value": "[FILTERED]"}
],
"content_length": "231"
}
Output
time:2022-02-28T22:00:55.196Z
"key:state","value:pending","key:options","value:request"
Demo
https://jqplay.org/s/F_3QP6-EvK
I want to process data with a bash script but have trouble to get the InfluxDB output to the desired csv output with all tags and fields.
Below an example output from an influx query:
{
"results": [
{
"series": [
{
"name": "tickerPrice",
"tags": {
"symbol": "AAVE",
"symbolTo": "EUR"
},
"columns": [
"time",
"priceMean"
],
"values": [
[
1614402874120627200,
282.398263888889
]
]
},
{
"name": "tickerPrice",
"tags": {
"symbol": "BTC",
"symbolTo": "EUR"
},
"columns": [
"time",
"priceMean"
],
"values": [
[
1614402874120627200,
39189.756944444445
]
]
}
]
}
]
}
And I would like to transform it to:
"name","symbol","symbolTo","time","priceMean"
"tickerPrice","AAVE","EUR",1614402874120627200,282.398263888889
"tickerPrice","BTC","EUR",1614402874120627200,39189.756944444445
I have managed (google) to get the fields to a csv format but till now not managed to get all data in the csv. Here is the commands that I use for that:
$ jq -r '(.results[0].series[0].columns), (.results[0].series[].values[])'
Because this is not the only query I want to do it would be nice that it is universal for the content, so the number of fields and tags could be different.
Why you just don't specify csv format directly in influxdb CLI https://docs.influxdata.com/influxdb/v1.8/tools/shell/ :
-format 'json|csv|column' Specifies the format of the server responses.
So you won't need any result post processing.
The following produces the required output in a way that
allows for multiple values of "time" in each .values array, but does not refer to the specific headers except for "name":
def headers:
(.tags | keys_unsorted) as $tags
| (["name"] + $tags + .columns);
.results[0]
| (.series[0] | headers),
(.series[] | ([.name, .tags[]] + .values[]))
| #csv
This of course assumes that the separate "series" are conformal.
I have multiple JSON files one.json, two.json, three.json with the below format and I want to create a consolidated array from them using jq. So, from all the files I want to extract Name and Value field inside the Parameters and use them to create an array where the id value will be constructed from the Name value and value field will be constructed using Value field value.
input:
one.json:
{
"Parameters": [
{
"Name": "id1",
"Value": "one",
"Version": 2,
"LastModifiedDate": 1581663187.36
}
]
}
two.json
{
"Parameters": [
{
"Name": "id2",
"Value": "xyz",
"Version": 2,
"LastModifiedDate": 1581663187.36
}
]
}
three.json
{
"Parameters": [
{
"Name": "id3",
"Value": "xyz",
"Version": 2,
"LastModifiedDate": 1581663187.36
}
]
}
output:
[
{
"id": "id1",
"value": "one"
},
{
"id": "id2",
"value": "xyz"
},
{
"id": "id3",
"value": "xyz"
}
]
How to achieve this using jq
You can use a reduce expression instead of slurping the whole file into memory (-s); by iterative manipulation of the input file contents and then appending the required fields one at a time.
jq -n 'reduce inputs.Parameters[] as $d (.; . + [ { id: $d.Name, value: $d.Value } ])' one.json two.json three.json
The -n flag is to ensure that we construct the output JSON data from scratch over the input file contents made available over the inputs function. Since reduce works in an iterative manner, for each of the object in the input, we create a final array, creating the KV pair as desired.
I have the following json
[
{
"certname": "server1",
"environment": "production",
"name": "memorysize",
"value": "62.76 GiB"
},
{
"certname": "server1",
"environment": "production",
"name": "processorcount",
"value": 12
},
{
"certname": "server2",
"environment": "production",
"name": "memorysize",
"value": "62.76 GiB"
},
{
"certname": "server2",
"environment": "production",
"name": "processorcount",
"value": 10
}
]
And I want to convert to this format where it's grouped by the certname. The challenge is I need to use value for to make it as the key as follow
[
{
"certname": "server1",
"memorysize": "62.76 GiB",
"processorcount": 12
},
{
"certname": "server2",
"memorysize": "62.76 GiB",
"processorcount": 10
}
]
How do I do this using jq? I have tried to_entries but it doesn't help either.
Thanks
The following is a commented jq script. Feel free to use it as is, or strip out the newlines and comments and use it as is.
# First, we construct an object that maps each `$certname` to `{certname: $certname}`. We name it $init.
(map({key:.certname, value: {certname}}) | unique | from_entries) as $init |
# Next, we take each object of the input in turn (name it $attr) and assign its
# `name:value` into one of the objects.
# $init is the dictionary above
# Reduce will pass the current dictionary as . for each invocation, and the assignment
# returns the input object.
reduce .[] as $attr ($init; .[$attr.certname][$attr.name] = $attr.value) |
# Our initial dictionary has now been expanded with attributes.
# Map it back to an array of objects. .[] is a stream of objects,
# we capture that in an outer array.
[.[]]
I have some JSON output I am trying to parse with jq. I read some examples on filtering but I don't really understand it and my output it more complicated than the examples. I have no idea where to even begin beyond jq '.[]' as I don't understand the syntax of jq beyond that and the hierarchy and terminology are challenging as well. My JSON output is below. I want to return the value for Valid where the ItemName equals Item_2. How can I do this?
"1"
[
{
"GroupId": "1569",
"Title": "My_title",
"Logo": "logo.jpg",
"Tags": [
"tag1",
"tag2",
"tag3"
],
"Owner": [
{
"Name": "John Doe",
"Id": "53335"
}
],
"ItemId": "209766",
"Item": [
{
"Id": 47744,
"ItemName": "Item_1",
"Valid": false
},
{
"Id": 47872,
"ItemName": "Item_2",
"Valid": true
},
{
"Id": 47872,
"ItemName": "Item_3",
"Valid": false
}
]
}
]
"Browse"
"8fj9438jgge9hdfv0jj0en34ijnd9nnf"
"v9er84n9ogjuwheofn9gerinneorheoj"
Except for the initial and trailing JSON scalars, you'd simply write:
.[] | .Item[] | select( .ItemName == "Item_2" ) | .Valid
In your particular case, to ensure the top-level JSON scalars are ignored, you could prefix the above with:
arrays |