Json to CSV conversion with value as headers - json

I have a below JSON file and need to convert to CSV file with some values as headers and below that values should get populated. Below is the sample json
{
"environments" : [ {
"dimensions" : [ {
"metrics" : [ {
"name" : "count",
"values" : [ "123" ]
}, {
"name" : "response_time",
"values" : [ "15.7" ]
}],
"name" : "abcd"
}, {
"metrics" : [ {
"name" : "count",
"values" : [ "456" ]
}, {
"name" : "response_time",
"values" : [ "18.7" ]
}],
"name" : "xyzz"
}
This is what I have tried already
jq -r '.environments[].dimensions[] | .name as $p_name | .metrics[] | .name as $val_name | if $val_name == "response_time" then ($p_name,$val_name, .values[])' input.json
Expected out as
name,count,response_time
abcd, 123, 15.7
xyzz, 456, 18.7

If the goal is to rely on the JSON itself to supply the header names in whatever order the "metrics" arrays present them,
then consider:
.environments[].dimensions
| ["name", (.[0] | .metrics[] | .name)], # first emit the headers
( .[] | [.name, (.metrics[].values[0])] ) # ... and then the data rows
| #csv

Generating the headers is easy, so I'll focus on generating the rest of the CSV.
The following has the advantage of being straightforward and will hopefully be more-or-less self-explanatory, at least with the jq manual at the ready. A tweak with an eye to efficiency follows.
jq -r '
# name,count,response_time
.environments[].dimensions[]
| .name as $p_name
| .metrics
| [$p_name]
+ map(select(.name == "count") | .values[0] )
+ map(select(.name == "response_time") | .values[0] )
| #csv
'
Efficiency
Here's a variant of the above which would be appropriate if the .metrics array had a large number of items:
jq -r '
# name,count,response_time
.environments[].dimensions[]
| .name as $p_name
| INDEX(.metrics[]; .name) as $dict
| [$p_name, $dict["count"].values[0], $dict["response_time"].values[0]]
| #csv
'

Related

How to spare & group objects according key-value

I'm new on this, those are my first steps. I guess I've started with a not simple case.
Let's see:
I have objects, with an ID (name) and a resource group (rgs). Each object may be part of several groups. And what a do need is to get the intersections of the groups.
It is important to say that the object may part of several groups, which are parent-child groups, and I just need to get the parent group. It is easy to identify the parenthoods as they share prefixes.
e.g. Group PROM_FD_ARCNA contains the child groups PROM_FD_ARCNA_TGM and PROM_FD_ARCNA_TGM_TGA.
And the child groups contains the objects itself. But, as long as I can get the information from object, it is over.
The parent groups are PROM_FD_ARCNA, PROM_JOB_ICMP and PROM_JOB_WIN. That is to say, I need to get those objects which belong to the intersections of those groups.
The JSON file which looks like:
[
{
"id_ci": "487006",
"name": "LABTNSARWID625",
"id_ci_class": "host",
"rgs": "PROM_FD_ARCNA, PROM_FD_ARCNA_TGM, PROM_FD_ARCNA_TGM_TGA"
},
{
"id_ci": "5706",
"name": "HCCQ2001",
"id_ci_class": "host",
"rgs": "PROM_JOB_ICMP"
},
{
"id_ci": "9106",
"name": "HCC02155",
"id_ci_class": "host",
"rgs": "PROM_FD_ARCNA, PROM_FD_ARCNA_TGA, PROM_JOB_ICMP"
},
{
"id_ci": "2306",
"name": "VM00006",
"id_ci_class": "host",
"rgs": "PROM_FD_ARCNA, PROM_FD_ARCNA_TGA, PROM_JOB_WIN, PROM_JOB_WIN_TGA"
}
]
If my explanation was not good, I need to get a JSON like this:
PROM_FD_ARCNA, PROM_JOB_ICMP
{
"HCC02155"
}
PROM_FD_ARCNA, PROM_JOB_WIN
{
"VM00006"
}
As those are the intersections.
So far, I tried this:
jq '[.[] | select(.id_ci_class == "host") | select (.rgs | startswith("PROM_FD_ARCNA")) | .rgs = "PROM_FD_ARCNA"]
| group_by(.rgs) | map({"rgs": .[0].rgs, "Hosts": map(.name)}) ' ./prom_jobs.json >> Step0A.json
jq '[.[] | select(.id_ci_class == "host") | select (.rgs | startswith("PROM_JOB_WIN")) | .rgs = "PROM_JOB_WIN"]
| group_by(.rgs) | map({"rgs": .[0].rgs, "Hosts": map(.name)}) ' ./prom_jobs.json >> Step0A.json
jq '[.[] | select(.id_ci_class == "host") | select (.rgs | startswith("PROM_JOB_ICMP")) | .rgs = "PROM_JOB_ICMP"]
| group_by(.rgs) | map({"rgs": .[0].rgs, "Hosts": map(.name)}) ' ./prom_jobs.json >> Step0A.json
And the result is:
[
{
"rgs": "PROM_FD_ARCNA",
"Hosts": [
"LABTNSARWID625",
"HCC02155",
"VM00006"
]
}
]
[
{
"rgs": "PROM_JOB_WIN",
"Hosts": [
"VM00006"
]
}
]
[
{
"rgs": "PROM_JOB_ICMP",
"Hosts": [
"HCCQ2001",
"HCC02155"
]
}
]
Of course, the full JSON is quite long and I need to process this as lightweight as possible. Don't know if I've started well or bad.
def to_set(s): reduce s as $_ ( {}; .[ $_ ] = true );
[ "PROM_FD_ARCNA", "PROM_JOB_ICMP", "PROM_JOB_WIN" ] as $roots |
map(
{
name,
has_rg: to_set( .rgs | split( ", " )[] )
}
) as $hosts |
[
range( 0; $roots | length ) as $i | $roots[ $i ] as $g1 |
range( $i+1; $roots | length ) as $j | $roots[ $j ] as $g2 |
{
root_rgs: [ $g1, $g2 ],
names: [
$hosts[] |
select( .has_rg[ $g1 ] and .has_rg[ $g2 ] ) |
.name
]
} |
select( .names | length > 0 )
]
produces
[
{
"root_rgs": [
"PROM_FD_ARCNA",
"PROM_JOB_ICMP"
],
"names": [
"HCC02155"
]
},
{
"root_rgs": [
"PROM_FD_ARCNA",
"PROM_JOB_WIN"
],
"names": [
"VM00006"
]
}
]
Demo on jqplay

Is there a way to filter a JSON object using jq to only include those with a key matching a value from a known list?

I have a JSON array, and another text file that contains a list of values.
[
{
"key": "foo",
"detail": "bar"
},
...
]
I need to filter the array elements to only those that have a "key" value that is found in the list of values.
The list of values is a text file containing a single item per-line.
foo
baz
Is this possible to do using jq?
You can use the following:
jq --rawfile to_keep_file to_keep.txt '
( [ $to_keep_file | match(".+"; "g").string | { (.): true } ] | add ) as $to_keep_lkup |
map(select($to_keep_lkup[.key]))
' to_filter.json
or
(
jq -sR . to_keep.txt
cat to_filter.json
) | jq -n '
( [ input | match(".+"; "g").string | { (.): true } ] | add ) as $to_keep_lkup |
inputs | map(select($to_keep_lkup[.key]))
'
The former requires jq v1.6, the first version to provide --rawfile.
jqplay

How to get the first object after filtering an array in jq?

Given the following JSON
{
"tags": [
{
"key": "env",
"value": "foo"
},
{
"key": "env",
"value": "bar"
}
]
}
I am trying to find out the first tag where the key is env. I have this-
.tags[] | select (.key=="env") |.[0]
but that gives me an error Cannot index object with number
Use first(expr) to provide an expression that satisfies your usecase.
first(.tags[]? | select(.key == "env") .value)
You could wrap the results of your query in an array and then pick the first one
[.tags[] | select(.key=="env")] | .[0]
jq -r 'first( .tags[] | select(.key=="env") ).value'
jqplay
.tags[] flattens the array into a stream of values. You're applying .[0] to each of the values, not a filtered array. To filter an array, you'd use
.tags | map(select(...)) | .[0]
or
.tags | map(select(...)) | first
map(...) is a shorthand for [ .[] | ... ], so the above is equivalent to
.tags | [ .[] | select(...) ] | first
and
[ .tags[] | select(...) ] | first
Finally, [ ... ] | first can be written as first(...).
first( .tags[] | select(...) )

JQ to CSV issues

I previously got some help on here for some jq to csv issues. I ran into an issue where a few json files had some extra values that breaks the jq command
Here is the json data. The repairs section is what breaks the jq command
[
{
"Name": "John Doe",
"Car": [
"Car1",
"Car2"
],
"Location": "Texas",
"Repairs: {
"RepairLocations": {
"RepairsCompleted":[
"Fix1",
"Fix2"
]
}
}
},
{
"Name": "Jane Roe",
"Car": "Car1",
"Location": [
"Illinois",
"Kansas"
]
}
]
Here is the command
def expand($keys):
. as $in
| reduce $keys[] as $k ( [{}];
map(. + {
($k): ($in[$k] | if type == "array" then .[] else . end)
})
) | .[];
(.[0] | keys_unsorted) as $h
| $h, (.[] | expand($h) | [.[$h[]]]) | #csv
This is the end result i am trying to get. This data isnt actual data.
Name,Car,Location,Repairs:RepairLocation
John Doe,Car1,Texas,RepairsCompleted:Fix1
John Doe,Car1,Texas,RepairsCompleted:Fix2
John Doe,Car2,Texas,RepairsCompleted:Fix1
John Doe,Car2,Texas,RepairsCompleted:Fix2
Jane Roe,Car1,Illinois,
Jane Roe,Car1,Kansas,
Any advice on this would be great. I am struggling to figure jq out
A simple solution can be obtained using the same technique shown in one of the answers to the similar question you already asked. The only difference is fulfilling your requirements in the case where the "Repairs" key does not exist:
["Name", "Car", "Location", "Repairs:RepairLocation"],
(.[]
| [.Name]
+ (.Car|..|scalars|[.])
+ (.Location|..|scalars|[.])
+ (.Repairs|..|scalars
| [if . == null then . else "RepairsCompleted:\(.)" end]) )
| #csv
Avoiding the repetition with a helper function
def s: .. | scalars | [.];
["Name", "Car", "Location", "Repairs:RepairLocation"],
(.[]
| [.Name]
+ (.Car|s)
+ (.Location|s)
+ (.Repairs|s|map(if . == null then . else "RepairsCompleted:\(.)" end)))
| #csv

jq: change multiple values within a select object

I have an array of JSON objects, and I am trying change the name and version on the object of a given #type, with the following input
[
{
"name": "oldname",
"version": "oldversion",
"#type": "Project"
},
{
"name": "bomname",
"version": "bomversion",
"#type": "BOM"
},
{
"name": "componentname",
"version": "componentversion",
"#type": "Component"
}
]
I found many examples for changing a single value, and I can successfully do this by chaining multiple select statements together.
$ cat original.json | jq '[ .[] | (select(.["#type"] == "Project") | .name ) = "newname" | (select(.["#type"] == "Project") | .version ) = "newversion ] ' > renamed.json
But I was hoping I could condense this so I only have perform the select once to change both values.
Using your approach:
[ .[]
| if .["#type"] == "Project"
then .name = "newname" | .version = "newversion"
else . end ]
or if you want to use select, you could write:
map( (select(.["#type"] == "Project") | .name = "newname" | .version = "newversion" ) // .)
or more exotically:
(.[] | select(["#type"] == "Project"))
|= (.name = "newname" | .version = "newversion" )
Merge an object with the new values.
map(select(.["#type"] == "Project") * {name: "newname", version: "newversion"} // .)