I want to fetch some data from below JSON code:
I'm able to get the output using below command but now I want to format it in such a way that my output will look like the expected output.
Command:
cat dump | jq -r '["name","IP","NAT","location","method"],
(.objects[] | [.name, ."ipv4-address", ."nat-settings"."ipv4-address", ."nat-settings"."install-on", ."nat-settings".method])
| #csv'
| sed -e 's/"//g'
After using #csv I got below output:
name,IP,NAT,location,method
H_103.109.135.25,103.109.135.25,1.1.1.1,All,static
H_103.109.135.250,103.109.135.250,,,
and whenever I use #tsv I get "jq: error: tsv is not a valid format"
Can any one suggest me how can I achieve below output:
Expected Output:
Raw JSON Code:
{
"from" : 1,
"to" : 2,
"total" : 2,
"objects" : [ {
"uid" : "73b7036d-e8ec-47b7-99b5-19ca89eb5fd0",
"name" : "H_103.109.135.25",
"type" : "host",
"domain" : {
"uid" : "41e821a0-3720-11e3-aa6e-0800200c9fde",
"name" : "SMC User",
"domain-type" : "domain"
},
"ipv4-address" : "103.109.135.25",
"interfaces" : [ ],
"nat-settings" : {
"auto-rule" : true,
"ipv4-address" : "1.1.1.1",
"ipv6-address" : "",
"install-on" : "All",
"method" : "static"
},
"comments" : "",
"color" : "black",
"icon" : "Objects/host",
"tags" : [ ],
"meta-info" : {
"lock" : "unlocked",
"validation-state" : "ok",
"last-modify-time" : {
"posix" : 1674820459413,
"iso-8601" : "2023-01-27T17:24+0530"
},
"last-modifier" : "admin",
"creation-time" : {
"posix" : 1674818326777,
"iso-8601" : "2023-01-27T16:48+0530"
},
"creator" : "admin"
},
"read-only" : false,
"available-actions" : {
"edit" : "true",
"delete" : "true",
"clone" : "true"
}
}, {
"uid" : "7300c38a-a496-497a-b9e3-5701fa081393",
"name" : "H_103.109.135.250",
"type" : "host",
"domain" : {
"uid" : "41e821a0-3720-11e3-aa6e-0800200c9fde",
"name" : "SMC User",
"domain-type" : "domain"
},
"ipv4-address" : "103.109.135.250",
"interfaces" : [ ],
"nat-settings" : {
"auto-rule" : false
},
"comments" : "",
"color" : "black",
"icon" : "Objects/host",
"tags" : [ ],
"meta-info" : {
"lock" : "unlocked",
"validation-state" : "ok",
"last-modify-time" : {
"posix" : 1674818341888,
"iso-8601" : "2023-01-27T16:49+0530"
},
"last-modifier" : "admin",
"creation-time" : {
"posix" : 1674818341888,
"iso-8601" : "2023-01-27T16:49+0530"
},
"creator" : "admin"
},
"read-only" : false,
"available-actions" : {
"edit" : "true",
"delete" : "true",
"clone" : "true"
}
} ]
}
Note:
It's not mandatory that the output should be printed in table using jq only. "awk" or "sed" is also fine.
I have extracted data that required from the below raw json data:
Extracted data:
{
"name": "H_103.109.135.25",
"IP": "103.109.135.25",
"NAT": "1.1.1.1",
"location": "All",
"method": "static"
},
{
"name": "H_103.109.135.250",
"IP": "103.109.135.250",
"NAT": "NA",
"location": "NA",
"method": "NA"
}
I now just need to format this data into table like below or somewhat similar:
| name | IP | NAT | location | method |
|-------------------|-----------------|---------|------------|----------|
| H_103.109.135.25 | 103.109.135.25 | 1.1.1.1 | All | static |
| H_103.109.135.250 | 103.109.135.250 | NA | NA | NA |
There is jbtl which may produce what you're looking for. If you have this in output.jq for example:
.objects
| map(
{ name, IP: ."ipv4-address" } +
(."nat-settings" | {
NAT: (."ipv4-address" // "NA"),
location: (."install-on" // "NA"),
method: (.method // "NA")
})
)
then passing the data through this filter and piping it into jtbl with the -m option, like this:
cat dump | jq -f output.jq | jtbl -m
gives this
| name | IP | NAT | location | method |
|-------------------|-----------------|---------|------------|----------|
| H_103.109.135.25 | 103.109.135.25 | 1.1.1.1 | All | static |
| H_103.109.135.250 | 103.109.135.250 | NA | NA | NA |
miller is handy for pretty-printing output.
echo 'name,IP,NAT,location,method
H_103.109.135.25,103.109.135.25,1.1.1.1,All,static
H_103.109.135.250,103.109.135.250,,,' \
| mlr --c2p --barred put 'for (i,v in $*) {if (v == "") {$[i] = "NA"}}'
--c2p is a shortcut for --icsv --opprint which reads CSV input and outputs pretty-printed tabular form.
+-------------------+-----------------+---------+----------+--------+
| name | IP | NAT | location | method |
+-------------------+-----------------+---------+----------+--------+
| H_103.109.135.25 | 103.109.135.25 | 1.1.1.1 | All | static |
| H_103.109.135.250 | 103.109.135.250 | NA | NA | NA |
+-------------------+-----------------+---------+----------+--------+
The miller put verb takes an awk-like script.
See https://miller.readthedocs.io/en/latest/
A bit more functional style:
mlr --c2p --barred put '$* = apply($*, func(k,v) {return {k: v == "" ? "NA" : v}})'
I'd suggest removing quotes and adding "NA" inside jq, and then pipe the output to column
jq -r '
[
["name","IP","NAT","location","method"],
( .objects[]
| {"nat-settings": {"ipv4-address": "NA", "install-on": "NA", method: "NA"}} * .
| [.name, ."ipv4-address"] + (."nat-settings" | [."ipv4-address", ."install-on", .method])
)
][] | join(",")
' dump | column -s, -t
That assumes that the "nat-settings" object is missing the "ipv4-address", etc, keys.
I would recommend using jq's #tsv and the very standard tool, column, e.g. as follows:
< dump jq -r '
["name","IP","NAT","location","method"],
(.objects[] | [.name, ."ipv4-address", ."nat-settings"."ipv4-address", ."nat-settings"."install-on", ."nat-settings".method])
| #tsv' | column -t
Related
I'm new on this, those are my first steps. I guess I've started with a not simple case.
Let's see:
I have objects, with an ID (name) and a resource group (rgs). Each object may be part of several groups. And what a do need is to get the intersections of the groups.
It is important to say that the object may part of several groups, which are parent-child groups, and I just need to get the parent group. It is easy to identify the parenthoods as they share prefixes.
e.g. Group PROM_FD_ARCNA contains the child groups PROM_FD_ARCNA_TGM and PROM_FD_ARCNA_TGM_TGA.
And the child groups contains the objects itself. But, as long as I can get the information from object, it is over.
The parent groups are PROM_FD_ARCNA, PROM_JOB_ICMP and PROM_JOB_WIN. That is to say, I need to get those objects which belong to the intersections of those groups.
The JSON file which looks like:
[
{
"id_ci": "487006",
"name": "LABTNSARWID625",
"id_ci_class": "host",
"rgs": "PROM_FD_ARCNA, PROM_FD_ARCNA_TGM, PROM_FD_ARCNA_TGM_TGA"
},
{
"id_ci": "5706",
"name": "HCCQ2001",
"id_ci_class": "host",
"rgs": "PROM_JOB_ICMP"
},
{
"id_ci": "9106",
"name": "HCC02155",
"id_ci_class": "host",
"rgs": "PROM_FD_ARCNA, PROM_FD_ARCNA_TGA, PROM_JOB_ICMP"
},
{
"id_ci": "2306",
"name": "VM00006",
"id_ci_class": "host",
"rgs": "PROM_FD_ARCNA, PROM_FD_ARCNA_TGA, PROM_JOB_WIN, PROM_JOB_WIN_TGA"
}
]
If my explanation was not good, I need to get a JSON like this:
PROM_FD_ARCNA, PROM_JOB_ICMP
{
"HCC02155"
}
PROM_FD_ARCNA, PROM_JOB_WIN
{
"VM00006"
}
As those are the intersections.
So far, I tried this:
jq '[.[] | select(.id_ci_class == "host") | select (.rgs | startswith("PROM_FD_ARCNA")) | .rgs = "PROM_FD_ARCNA"]
| group_by(.rgs) | map({"rgs": .[0].rgs, "Hosts": map(.name)}) ' ./prom_jobs.json >> Step0A.json
jq '[.[] | select(.id_ci_class == "host") | select (.rgs | startswith("PROM_JOB_WIN")) | .rgs = "PROM_JOB_WIN"]
| group_by(.rgs) | map({"rgs": .[0].rgs, "Hosts": map(.name)}) ' ./prom_jobs.json >> Step0A.json
jq '[.[] | select(.id_ci_class == "host") | select (.rgs | startswith("PROM_JOB_ICMP")) | .rgs = "PROM_JOB_ICMP"]
| group_by(.rgs) | map({"rgs": .[0].rgs, "Hosts": map(.name)}) ' ./prom_jobs.json >> Step0A.json
And the result is:
[
{
"rgs": "PROM_FD_ARCNA",
"Hosts": [
"LABTNSARWID625",
"HCC02155",
"VM00006"
]
}
]
[
{
"rgs": "PROM_JOB_WIN",
"Hosts": [
"VM00006"
]
}
]
[
{
"rgs": "PROM_JOB_ICMP",
"Hosts": [
"HCCQ2001",
"HCC02155"
]
}
]
Of course, the full JSON is quite long and I need to process this as lightweight as possible. Don't know if I've started well or bad.
def to_set(s): reduce s as $_ ( {}; .[ $_ ] = true );
[ "PROM_FD_ARCNA", "PROM_JOB_ICMP", "PROM_JOB_WIN" ] as $roots |
map(
{
name,
has_rg: to_set( .rgs | split( ", " )[] )
}
) as $hosts |
[
range( 0; $roots | length ) as $i | $roots[ $i ] as $g1 |
range( $i+1; $roots | length ) as $j | $roots[ $j ] as $g2 |
{
root_rgs: [ $g1, $g2 ],
names: [
$hosts[] |
select( .has_rg[ $g1 ] and .has_rg[ $g2 ] ) |
.name
]
} |
select( .names | length > 0 )
]
produces
[
{
"root_rgs": [
"PROM_FD_ARCNA",
"PROM_JOB_ICMP"
],
"names": [
"HCC02155"
]
},
{
"root_rgs": [
"PROM_FD_ARCNA",
"PROM_JOB_WIN"
],
"names": [
"VM00006"
]
}
]
Demo on jqplay
I have a schema named reports in mongo and a collection named totals.
The keys in it looks like:
{ "_id" : { "dt" : "2018-12-02", "dt2" : "2018-04-08", "num" : 1312312312 }, "str" : 1 }
I would like to use DMS to migrate this collection into mysql instance on aws. The table should look like:
create table tab(
dt date,
dt2 date,
num bigint)
Currently, I'm using dms with simple rule:
{
"rules": [
{
"rule-type": "transformation",
"rule-id": "1",
"rule-name": "1",
"rule-target": "table",
"object-locator": {
"schema-name": "reports",
"table-name": "totals"
},
"rule-action": "rename",
"value": "tab",
"old-value": null
},
{
"rule-type": "selection",
"rule-id": "2",
"rule-name": "2",
"object-locator": {
"schema-name": "reports",
"table-name": "totals"
},
"rule-action": "include",
"filters": []
}
]
}
The result is not what I wanted:
MySQL [stats]> desc tab;
+-------+----------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+----------+------+-----+---------+-------+
| _doc | longtext | YES | | NULL | |
+-------+----------+------+-----+---------+-------+
MySQL [(none)]> select * from tab limit 1;
+------------------------------------------------------------------------------------------+
| _doc |
+------------------------------------------------------------------------------------------+
| { "_id" : { "dt" : "2018-12-02", "dt2" : "2018-04-08", "num" : 1312312312 }, "str" : 1 } |
+------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
Endpoint needed to have nestingLevel=ONE; instead of nestingLevel=NONE;.
Basically it means look at the data as a table instead of a document.
I have complex configuration file in JSON:
{
"config": {
...,
"extra": {
...
"auth_namespace.com": {
...
"name": "some_name",
"id": 1,
...
}
},
...,
"endpoints": [
{ ...,
"extra": {
"namespace_1.com": {...},
"namespace_auth.com": { "scope": "scope1" }
}},
{ ...
# object without "extra" property
...
},
...,
{ ...
"extra": {
"namespace_1.com": {...},
"namespace_auth.com": { "scope": "scope2" }
}},
{ ...
"extra": {
# scopes may repeat
"namespace_auth.com": { "scope": "scope2" }
}}
]
}
}
And I want to get the output object with properties "name", "id", "scopes". Where "scopes" is an array of unique values.
Something like this:
{
"name": "some_name",
"id": 1,
"scopes": ["scope1", "scope2" ... "scopeN"]
}
I can get these properties separately. But I don't know how to combine them together.
[
.config |
(
.extra["auth_namespace.com"] |
select(.name) |
{name, id}
) as $name_id |
.endpoints[] |
.extra["namespace_auth.com"].scope |
select(.)
] | unique | {scopes: .}
Perhaps the following is closer to what you're looking for:
.config
| (.extra."auth_namespace.com" | {id, name})
+ {scopes: .endpoints
| map( select(has("extra"))
| .extra."namespace_auth.com"
| select(has("scope"))
| .scope )
| unique }
Well, I found a solution. It's ugly, but it works.
Would be grateful if someone could write a more elegant version.
.config
| (
.endpoints
| map(.extra["namespace_auth.com"] | select(.scope) | .[])
| unique
) as $s
| .extra["auth_namespace.com"] | select(.name)
| {name, id, scopes: $s}
I have a below JSON file and need to convert to CSV file with some values as headers and below that values should get populated. Below is the sample json
{
"environments" : [ {
"dimensions" : [ {
"metrics" : [ {
"name" : "count",
"values" : [ "123" ]
}, {
"name" : "response_time",
"values" : [ "15.7" ]
}],
"name" : "abcd"
}, {
"metrics" : [ {
"name" : "count",
"values" : [ "456" ]
}, {
"name" : "response_time",
"values" : [ "18.7" ]
}],
"name" : "xyzz"
}
This is what I have tried already
jq -r '.environments[].dimensions[] | .name as $p_name | .metrics[] | .name as $val_name | if $val_name == "response_time" then ($p_name,$val_name, .values[])' input.json
Expected out as
name,count,response_time
abcd, 123, 15.7
xyzz, 456, 18.7
If the goal is to rely on the JSON itself to supply the header names in whatever order the "metrics" arrays present them,
then consider:
.environments[].dimensions
| ["name", (.[0] | .metrics[] | .name)], # first emit the headers
( .[] | [.name, (.metrics[].values[0])] ) # ... and then the data rows
| #csv
Generating the headers is easy, so I'll focus on generating the rest of the CSV.
The following has the advantage of being straightforward and will hopefully be more-or-less self-explanatory, at least with the jq manual at the ready. A tweak with an eye to efficiency follows.
jq -r '
# name,count,response_time
.environments[].dimensions[]
| .name as $p_name
| .metrics
| [$p_name]
+ map(select(.name == "count") | .values[0] )
+ map(select(.name == "response_time") | .values[0] )
| #csv
'
Efficiency
Here's a variant of the above which would be appropriate if the .metrics array had a large number of items:
jq -r '
# name,count,response_time
.environments[].dimensions[]
| .name as $p_name
| INDEX(.metrics[]; .name) as $dict
| [$p_name, $dict["count"].values[0], $dict["response_time"].values[0]]
| #csv
'
Let's say I have two tables in my database, employee and car defined thusly.
employee:
+--------------+------------+
| col_name | data_type |
+--------------+------------+
| eid | int |
| name | string |
| salary | int |
| destination | string |
+--------------+------------+
car:
+------------+----------------+
| col_name | data_type |
+------------+----------------+
| cid | int |
| name | string |
| model | string |
| cylinders | int |
| price | int |
+------------+----------------+
I would like to export this schema to a JSON object so that I can populate an HTML dropdown menu based on the table - for instance, the table menu would have employee and car. Selecting employee would populate another dropdown with the column names and types corresponding to that table.
Given this use case, would the optimal json representation of the database be this?
{
"employee": {
"salary": "int",
"destination": "string",
"eid": "int",
"name": "string"
},
"car": {
"price": "int",
"model": "string",
"cylinders": "int",
"name": "string",
"cid": "int"
}
}
EDIT:
Or would this be more appropriate?
{
"employee": [
{
"type": "int",
"colname": "eid"
},
{
"type": "string",
"colname": "name"
},
{
"type": "int",
"colname": "salary"
},
{
"type": "string",
"colname": "destination"
}
],
"car": [
{
"type": "int",
"colname": "cid"
},
{
"type": "string",
"colname": "name"
},
{
"type": "string",
"colname": "model"
},
{
"type": "int",
"colname": "cylinders"
},
{
"type": "int",
"colname": "price"
}
]
}
In the first example, all your data is stored in objects. Assuming the structure is stored in a var mytables, you can get the names with Object.keys(mytables), which returns ['employee', 'car']. Equivalent for the columns inside: Object.keys(mytables['employee'].cols) returns ['salary','destination','eid','name'].
In the second example I would suggest to also store the tables in an array as the columns, like
[name: 'employee',
cols: [ {
"type": "int",
"colname": "cid"
}, ...]
Then you can easily iterate over the arrays and get the names by accessing mytables[i].name
for (t in tables){
console.log(tables[t].name);
for (c in tables[t].cols)
console.log(" - ",tables[t].cols[c].colname, ": ", tables[t].cols[c].type);
}