Related
I have the following printout,
{
"metric": {
"container": "container1",
"namespace": "namespace1",
"pod": "pod1"
},
"values": [
[
1664418600,
"1"
],
[
1664418900,
"2"
],
[
1664419200,
"6"
],
[
1664419500,
"8"
],
[
1664419800,
"7"
],
[
1664420100,
"9"
]
]
}
{
"metric": {
"container": "container2",
"namespace": "namespace2",
"pod": "pod2"
},
"values": [
[
1664420100,
"1"
]
]
}
What I want:
container=container1,namespace=namespace1,pod=pod1
1 1664418600
2 1664418900
6 1664419200
8 1664419500
7 1664419800
9 1664420100
container=container2,namespace=namespace2,pod=pod2
1 1664420100
Build it from two JSON programs:
Header lines: .metric | to_entries | map(join("=")) | join(",")
Get metric object: .metric
Convert to an array of key-value pairs: to_entries
Map each key-value pair object to a string "key=value": map(join("="))
Join all pairs by comma: join(",")
Value lists: .values[] | [last,first] | join(" ")
Stream values: .values[]
Reverse each two-valued array: [last,first]
Join items by blank: join(" ")
An alternative for 2.2. and 2.3. could be "\(last) \(first)", i.e. values[] | "\(last) \(first)". Or [last,first] could be replaced with reverse: .values[] | reverse | join(" ").
Putting the two programs together:
(.metric | to_entries | map(join("=")) | join(",")),
(.values[] | [last,first] | join(" "))
And then execute with raw output enabled: jq -r (.metrics|to_entries…
Output:
container=container1,namespace=namespace1,pod=pod1
1 1664418600
2 1664418900
6 1664419200
8 1664419500
7 1664419800
9 1664420100
container=container2,namespace=namespace2,pod=pod2
1 1664420100
I'm new on this, those are my first steps. I guess I've started with a not simple case.
Let's see:
I have objects, with an ID (name) and a resource group (rgs). Each object may be part of several groups. And what a do need is to get the intersections of the groups.
It is important to say that the object may part of several groups, which are parent-child groups, and I just need to get the parent group. It is easy to identify the parenthoods as they share prefixes.
e.g. Group PROM_FD_ARCNA contains the child groups PROM_FD_ARCNA_TGM and PROM_FD_ARCNA_TGM_TGA.
And the child groups contains the objects itself. But, as long as I can get the information from object, it is over.
The parent groups are PROM_FD_ARCNA, PROM_JOB_ICMP and PROM_JOB_WIN. That is to say, I need to get those objects which belong to the intersections of those groups.
The JSON file which looks like:
[
{
"id_ci": "487006",
"name": "LABTNSARWID625",
"id_ci_class": "host",
"rgs": "PROM_FD_ARCNA, PROM_FD_ARCNA_TGM, PROM_FD_ARCNA_TGM_TGA"
},
{
"id_ci": "5706",
"name": "HCCQ2001",
"id_ci_class": "host",
"rgs": "PROM_JOB_ICMP"
},
{
"id_ci": "9106",
"name": "HCC02155",
"id_ci_class": "host",
"rgs": "PROM_FD_ARCNA, PROM_FD_ARCNA_TGA, PROM_JOB_ICMP"
},
{
"id_ci": "2306",
"name": "VM00006",
"id_ci_class": "host",
"rgs": "PROM_FD_ARCNA, PROM_FD_ARCNA_TGA, PROM_JOB_WIN, PROM_JOB_WIN_TGA"
}
]
If my explanation was not good, I need to get a JSON like this:
PROM_FD_ARCNA, PROM_JOB_ICMP
{
"HCC02155"
}
PROM_FD_ARCNA, PROM_JOB_WIN
{
"VM00006"
}
As those are the intersections.
So far, I tried this:
jq '[.[] | select(.id_ci_class == "host") | select (.rgs | startswith("PROM_FD_ARCNA")) | .rgs = "PROM_FD_ARCNA"]
| group_by(.rgs) | map({"rgs": .[0].rgs, "Hosts": map(.name)}) ' ./prom_jobs.json >> Step0A.json
jq '[.[] | select(.id_ci_class == "host") | select (.rgs | startswith("PROM_JOB_WIN")) | .rgs = "PROM_JOB_WIN"]
| group_by(.rgs) | map({"rgs": .[0].rgs, "Hosts": map(.name)}) ' ./prom_jobs.json >> Step0A.json
jq '[.[] | select(.id_ci_class == "host") | select (.rgs | startswith("PROM_JOB_ICMP")) | .rgs = "PROM_JOB_ICMP"]
| group_by(.rgs) | map({"rgs": .[0].rgs, "Hosts": map(.name)}) ' ./prom_jobs.json >> Step0A.json
And the result is:
[
{
"rgs": "PROM_FD_ARCNA",
"Hosts": [
"LABTNSARWID625",
"HCC02155",
"VM00006"
]
}
]
[
{
"rgs": "PROM_JOB_WIN",
"Hosts": [
"VM00006"
]
}
]
[
{
"rgs": "PROM_JOB_ICMP",
"Hosts": [
"HCCQ2001",
"HCC02155"
]
}
]
Of course, the full JSON is quite long and I need to process this as lightweight as possible. Don't know if I've started well or bad.
def to_set(s): reduce s as $_ ( {}; .[ $_ ] = true );
[ "PROM_FD_ARCNA", "PROM_JOB_ICMP", "PROM_JOB_WIN" ] as $roots |
map(
{
name,
has_rg: to_set( .rgs | split( ", " )[] )
}
) as $hosts |
[
range( 0; $roots | length ) as $i | $roots[ $i ] as $g1 |
range( $i+1; $roots | length ) as $j | $roots[ $j ] as $g2 |
{
root_rgs: [ $g1, $g2 ],
names: [
$hosts[] |
select( .has_rg[ $g1 ] and .has_rg[ $g2 ] ) |
.name
]
} |
select( .names | length > 0 )
]
produces
[
{
"root_rgs": [
"PROM_FD_ARCNA",
"PROM_JOB_ICMP"
],
"names": [
"HCC02155"
]
},
{
"root_rgs": [
"PROM_FD_ARCNA",
"PROM_JOB_WIN"
],
"names": [
"VM00006"
]
}
]
Demo on jqplay
I'm trying to generate a CSV of sort from json file, the files are as below
cat role1.json
{
"Tags": [
{
"Key": "Name",
"Value": "Role1Name"
},
{
"Key": "ID",
"Value": "Role1ID"
},
{
"Key": "Manager",
"Value": "Role1Manager"
},
{
"Key": "User",
"Value": "Role1User"
},
{
"Key": "Country",
"Value": "USA"
}
]
}
cat role2.json
{
"Tags": [
{
"Key": "Name",
"Value": "Role2Name"
},
{
"Key": "ID",
"Value": "Role2ID"
},
{
"Key": "City",
"Value": "NewYork"
},
{
"Key": "Creator",
"Value": "Role2Creator"
},
{
"Key": "User",
"Value": "Role2User"
}
]
}
cat role3.json
{
"Tags": [
{
"Key": "Name",
"Value": "Role3Name"
},
{
"Key": "ID",
"Value": "Role3ID"
},
{
"Key": "Creator",
"Value": "Role3Creator"
},
{
"Key": "ZIP",
"Value": 82378
},
{
"Key": "Manager",
"Value": "Role3Manager"
},
{
"Key": "User",
"Value": "Role3User"
}
]
}
I want to generate lines from each of these to be later used as CSV, something like:
Role1Name, Role1ID, null, Role1Manager, Role1User
Role2Name, Role2ID, Role2Creator, null, Role2User
Role3Name, Role3ID, Role3Creator, Role3Manager, Role3User
For the header line
Name, ID, Creator, Manager, User
I'm able to get all the "Value" but not able to print null for missing "Key"
$cat role1.json | jq -rc '[.Tags[] | select(.Key == ("Name","ID","Creator","Manager","User")) | .Value]'
["Role1Name","Role1ID","Role1Manager","Role1User"]
$cat role2.json | jq -rc '[.Tags[] | select(.Key == ("Name","ID","Creator","Manager","User")) | .Value]'
["Role2Name","Role2ID","Role2Creator","Role2User"]
$cat role3.json | jq -rc '[.Tags[] | select(.Key == ("Name","ID","Creator","Manager","User")) | .Value]'
["Role3Name","Role3ID","Role3Creator","Role3Manager","Role3User"]
Can someone share with me how this can be done using jq.
Also, how can we enforce the order.
Thanks!
The key (ha!) is
[ .[ $keys[] ] ]
Had you looked at other answers to questions relating to CSV, you might have noticed the first step taken is to get the list of keys. This is often done by collecting the keys of the input objects. (Example) In your case, you have a hard-coded list, so it's even simpler.
If you wanted actual CSV, you could use
jq -sr '
[ "Name", "ID", "Creator", "Manager", "User" ] as $keys |
(
$keys,
( .[].Tags | from_entries | [ .[ $keys[] ] ] )
) |
#csv
' role*.json
This produces
"Name","ID","Creator","Manager","User"
"Role1Name","Role1ID",,"Role1Manager","Role1User"
"Role2Name","Role2ID","Role2Creator",,"Role2User"
"Role3Name","Role3ID","Role3Creator","Role3Manager","Role3User"
jqplay
Without a header:
jq -r '.Tags | from_entries | [ .["Name","ID","Creator","Manager","User"] ] | #csv' role*.json
jqplay
To get the specific output you posted (which isn't CSV), you could use
jq -sr '
[ "Name", "ID", "Creator", "Manager", "User" ] as $keys |
(
$keys,
( .[].Tags | from_entries | [ .[ $keys[] ] | . // "null" ] )
) |
join(", ")
' role*.json
This produces
Name, ID, Creator, Manager, User
Role1Name, Role1ID, null, Role1Manager, Role1User
Role2Name, Role2ID, Role2Creator, null, Role2User
Role3Name, Role3ID, Role3Creator, Role3Manager, Role3User
jqplay
Without a header:
jq -r '.Tags | from_entries | [ .["Name","ID","Creator","Manager","User"] | . // "null" ] | join(", ")' role*.json
jqplay
Got an answer from another forum, might be useful for others
$jq -rc '.Tags | from_entries | [.Name, .ID, .Creator, .Manager, .User]' role*.json
["Role1Name","Role1ID",null,"Role1Manager","Role1User"]
["Role2Name","Role2ID","Role2Creator",null,"Role2User"]
["Role3Name","Role3ID","Role3Creator","Role3Manager","Role3User"]
Is there a way to know all the possible (acceptable) keywords, after a SELECT statement in SQL before the list of fields?
Edit
Reference DB is MySQL
PostgreSQL 9.4
http://www.postgresql.org/docs/9.4/static/sql-select.html
[ WITH [ RECURSIVE ] with_query [, ...] ]
SELECT [ ALL | DISTINCT [ ON ( expression [, ...] ) ] ]
[ * | expression [ [ AS ] output_name ] [, ...] ]
[ FROM from_item [, ...] ]
[ WHERE condition ]
[ GROUP BY expression [, ...] ]
[ HAVING condition [, ...] ]
[ WINDOW window_name AS ( window_definition ) [, ...] ]
[ { UNION | INTERSECT | EXCEPT } [ ALL | DISTINCT ] select ]
[ ORDER BY expression [ ASC | DESC | USING operator ] [ NULLS { FIRST | LAST } ] [, ...] ]
[ LIMIT { count | ALL } ]
[ OFFSET start [ ROW | ROWS ] ]
[ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY ]
[ FOR { UPDATE | NO KEY UPDATE | SHARE | KEY SHARE } [ OF table_name [, ...] ] [ NOWAIT ] [...] ]
where from_item can be one of:
[ ONLY ] table_name [ * ] [ [ AS ] alias [ ( column_alias [, ...] ) ] ]
[ LATERAL ] ( select ) [ AS ] alias [ ( column_alias [, ...] ) ]
with_query_name [ [ AS ] alias [ ( column_alias [, ...] ) ] ]
[ LATERAL ] function_name ( [ argument [, ...] ] )
[ WITH ORDINALITY ] [ [ AS ] alias [ ( column_alias [, ...] ) ] ]
[ LATERAL ] function_name ( [ argument [, ...] ] ) [ AS ] alias ( column_definition [, ...] )
[ LATERAL ] function_name ( [ argument [, ...] ] ) AS ( column_definition [, ...] )
[ LATERAL ] ROWS FROM( function_name ( [ argument [, ...] ] ) [ AS ( column_definition [, ...] ) ] [, ...] )
[ WITH ORDINALITY ] [ [ AS ] alias [ ( column_alias [, ...] ) ] ]
from_item [ NATURAL ] join_type from_item [ ON join_condition | USING ( join_column [, ...] ) ]
and with_query is:
with_query_name [ ( column_name [, ...] ) ] AS ( select | values | insert | update | delete )
TABLE [ ONLY ] table_name [ * ]
MySQL 5.6
https://dev.mysql.com/doc/refman/5.6/en/select.html
SELECT
[ALL | DISTINCT | DISTINCTROW ]
[HIGH_PRIORITY]
[STRAIGHT_JOIN]
[SQL_SMALL_RESULT] [SQL_BIG_RESULT] [SQL_BUFFER_RESULT]
[SQL_CACHE | SQL_NO_CACHE] [SQL_CALC_FOUND_ROWS]
select_expr [, select_expr ...]
[FROM table_references
[PARTITION partition_list]
[WHERE where_condition]
[GROUP BY {col_name | expr | position}
[ASC | DESC], ... [WITH ROLLUP]]
[HAVING where_condition]
[ORDER BY {col_name | expr | position}
[ASC | DESC], ...]
[LIMIT {[offset,] row_count | row_count OFFSET offset}]
[PROCEDURE procedure_name(argument_list)]
[INTO OUTFILE 'file_name'
[CHARACTER SET charset_name]
export_options
| INTO DUMPFILE 'file_name'
| INTO var_name [, var_name]]
[FOR UPDATE | LOCK IN SHARE MODE]]
SQL Server 2014
https://msdn.microsoft.com/en-us/library/ms189499.aspx
<SELECT statement> ::=
[ WITH { [ XMLNAMESPACES ,] [ <common_table_expression> [,...n] ] } ]
<query_expression>
[ ORDER BY { order_by_expression | column_position [ ASC | DESC ] }
[ ,...n ] ]
[ <FOR Clause>]
[ OPTION ( <query_hint> [ ,...n ] ) ]
<query_expression> ::=
{ <query_specification> | ( <query_expression> ) }
[ { UNION [ ALL ] | EXCEPT | INTERSECT }
<query_specification> | ( <query_expression> ) [...n ] ]
<query_specification> ::=
SELECT [ ALL | DISTINCT ]
[TOP ( expression ) [PERCENT] [ WITH TIES ] ]
< select_list >
[ INTO new_table ]
[ FROM { <table_source> } [ ,...n ] ]
[ WHERE <search_condition> ]
[ <GROUP BY> ]
[ HAVING < search_condition > ]
See the documentation.
For example: mysql 5.0 documentation
Reserved Keywords in MySql
Keywords are words that have significance in SQL. Certain keywords, such as SELECT, DELETE, or BIGINT, are reserved and require special treatment for use as identifiers such as table and column names. This may also be true for the names of built-in functions.
Nonreserved keywords are permitted as identifiers without quoting. Reserved words are permitted as identifiers if you quote them
Reserved Keywords in MSSQL server :
Microsoft SQL Server uses reserved keywords for defining, manipulating, and accessing databases. Reserved keywords are part of the grammar of the Transact-SQL language that is used by SQL Server to parse and understand Transact-SQL statements and batches.
Although it is syntactically possible to use SQL Server reserved keywords as identifiers and object names in Transact-SQL scripts, you can do this only by using delimited identifiers.
We have a cloud-based GUI at my work where I can choose some fields from different tables so it can print out the values afterwards. It is not possible for me to write a WITH clause before a SELECT statement as the program automatically write SELECT first. So I want to know If it is possible somehow.
Yes you can, according to MSDN documentation (https://msdn.microsoft.com/en-us/library/ms189499.aspx) :
<SELECT statement> ::=
[WITH <common_table_expression> [,...n]]
<query_expression>
[ ORDER BY { order_by_expression | column_position [ ASC | DESC ] }
[ ,...n ] ]
[ <FOR Clause>]
[ OPTION ( <query_hint> [ ,...n ] ) ]
<query_expression> ::=
{ <query_specification> | ( <query_expression> ) }
[ { UNION [ ALL ] | EXCEPT | INTERSECT }
<query_specification> | ( <query_expression> ) [...n ] ]
<query_specification> ::=
SELECT [ ALL | DISTINCT ]
[TOP ( expression ) [PERCENT] [ WITH TIES ] ]
< select_list >
[ INTO new_table ]
[ FROM { <table_source> } [ ,...n ] ]
[ WHERE <search_condition> ]
[ <GROUP BY> ]
[ HAVING < search_condition > ]
If you can't find any other way around it, this GUI might be persuaded to accept
SELECT * FROM (
-- write your entire select statement here
);