Kusto Query using a bracket with a wildcard - json

Can you help me identifying what type of wildcard I need to use to find a certain email address in my properties field?
I know that the email I'm looking for is in the slot number 2
How can I find the email address without knowing the slot number?
can I use a [*] instead of a [2]?
Here's my query:
resources
| where type == 'microsoft.insights/actiongroups'
| where properties["enabled"] in~ ('true')
| where properties['emailReceivers'][2]['emailAddress'] == "DevSecOps#pato.com"
| project id,name,resourceGroup,subscriptionId,properties,location
| order by tolower(tostring(name)) asc
I have the following data in my properties field:
{
"enabled": true,
"automationRunbookReceivers": [],
"azureFunctionReceivers": [],
"azureAppPushReceivers": [],
"logicAppReceivers": [],
"eventHubReceivers": [],
"webhookReceivers": [],
"armRoleReceivers": [],
"emailReceivers": [
{
"name": "TED",
"status": "Enabled",
"useCommonAlertSchema": true,
"emailAddress": "tedtechnicalengineeringdesign#pato.com"
},
{
"name": "SevenOfNine",
"status": "Enabled",
"useCommonAlertSchema": true,
"emailAddress": "sevenofnine#pato.com"
},
{
"name": "PEAT",
"status": "Enabled",
"useCommonAlertSchema": true,
"emailAddress": "DevSecOps#pato.com"
}
],
"voiceReceivers": [],
"groupShortName": "eng-mon",
"itsmReceivers": [],
"smsReceivers": []
}
I've tried using [*] instead of [2] but it didn't work.

Do you need to find a certain email address from properties? can you please explain a little more why you need wildcards? can this query work for you? basically expand the 'emailReceivers' list and find out where emailAddress contains the value you are searching for.
resources
| where type == 'microsoft.insights/actiongroups'
| where properties["enabled"] in~ ('true')
| mv-expand properties['emailReceivers'] | limit 100
| extend emailAddr = properties_emailReceivers['emailAddress']
| where emailAddr contains "DevSecOps#pato.com"
| project id,name,resourceGroup,subscriptionId,properties,location
| order by tolower(tostring(name)) asc

where properties.emailReceivers has_cs "DevSecOps#pato.com" is theoretically not 100% safe ("DevSecOps#pato.com" might appear in fields other than "emailAddress"), but in your case it might be enough and if you have a large data set it will also be fast.
If you need a 100% guarantee, then also add the following:
where dynamic_to_json(properties.emailReceivers) matches regex '"emailAddress":"DevSecOps#pato.com"'
It's not pretty, but Azure Resource Graph uses just a subset of the KQL supported by Azure Data Explorer.
let resources = datatable(id:string, name:string, resourceGroup:string, subscriptionId:string, location:string, type:string, properties:dynamic)
[
"my_id"
,"my_name"
,"my_resourceGroup"
,"my_subscriptionId"
,"my_location"
,"microsoft.insights/actiongroups"
,dynamic
(
{
"enabled": true,
"automationRunbookReceivers": [],
"azureFunctionReceivers": [],
"azureAppPushReceivers": [],
"logicAppReceivers": [],
"eventHubReceivers": [],
"webhookReceivers": [],
"armRoleReceivers": [],
"emailReceivers": [
{
"name": "TED",
"status": "Enabled",
"useCommonAlertSchema": true,
"emailAddress": "tedtechnicalengineeringdesign#pato.com"
},
{
"name": "SevenOfNine",
"status": "Enabled",
"useCommonAlertSchema": true,
"emailAddress": "sevenofnine#pato.com"
},
{
"name": "PEAT",
"status": "Enabled",
"useCommonAlertSchema": true,
"emailAddress": "DevSecOps#pato.com"
}
],
"voiceReceivers": [],
"groupShortName": "eng-mon",
"itsmReceivers": [],
"smsReceivers": []
}
)
];
resources
| where type == "microsoft.insights/actiongroups"
| where properties.enabled == true
| where properties.emailReceivers has_cs "DevSecOps#pato.com"
| where dynamic_to_json(properties.emailReceivers) matches regex '"emailAddress":"DevSecOps#pato.com"'
| project id,name,resourceGroup,subscriptionId,properties,location
| order by tolower(name) asc
id
name
resourceGroup
subscriptionId
properties
location
my_id
my_name
my_resourceGroup
my_subscriptionId
{"enabled":true,"automationRunbookReceivers":[],"azureFunctionReceivers":[],"azureAppPushReceivers":[],"logicAppReceivers":[],"eventHubReceivers":[],"webhookReceivers":[],"armRoleReceivers":[],"emailReceivers":[{"name":"TED","status":"Enabled","useCommonAlertSchema":true,"emailAddress":"tedtechnicalengineeringdesign#pato.com"},{"name":"SevenOfNine","status":"Enabled","useCommonAlertSchema":true,"emailAddress":"sevenofnine#pato.com"},{"name":"PEAT","status":"Enabled","useCommonAlertSchema":true,"emailAddress":"DevSecOps#pato.com"}],"voiceReceivers":[],"groupShortName":"eng-mon","itsmReceivers":[],"smsReceivers":[]}
my_location
Fiddle

I found a way to do it using the keyword "contains".
In that way you don't need to specify in which slot it should find it, it could be [0],[1],[2]...[n]
resources
| where type == 'microsoft.insights/actiongroups'
| where properties["enabled"] in~ ('true')
| where properties['emailReceivers'] contains "DevSecOps#pato.com"
| project id,name,resourceGroup,subscriptionId,properties,location
| order by tolower(tostring(name)) asc

Related

Group by and remove duplicates across arrays objects using JQ

Given the json, I need to group by key userName the object userClientDetailDTOList across all sites->buildings->floors and remove any duplicate mac addresses.
I have been able to do it using jq expression -
[.billingDetailPerSiteDTOList[].billingDetailPerBuildingDTOList[].billingDetailsPerFloorDTOList[].userClientDetailDTOList[] ] | group_by(.userName) | map((.[0]|del(.associatedMacs)) + { associatedMacs: (map(.associatedMacs[]) | unique) })
This groups by userName and also removes duplicate macs belonging to particular user. This results in a list as
[
{
"userName": "1",
"associatedMacs": [
"3:3:3:3:3:3",
"5:5:5:5:5:5"
]
},
{
"userName": "10",
"associatedMacs": [
"4:4:4:4:4:4",
"6:6:6:6:6:6"
]
},
{
"userName": "2",
"associatedMacs": [
"1:1:1:1:1:1",
"2:2:2:2:2:2"
]
},
{
"userName": "3",
"associatedMacs": [
"2:2:2:2:2:2"
]
}
]
Live example
Questions:
Can the expression be simplified?
How do I remove duplicate mac addresses across all users? The mac address 2:2:2:2:2:2 is repeated for users 2 and 3
The filter is practically as good as it can get. If you really wanted to, you could still change
del(.associatedMacs) to {userName} for a positive definition, and
(…) + {…} to {userName: …, associatedMacs: …} to avoid the addition,
resulting in
… | map({userName: (.[0].userName), associatedMacs: (map(.associatedMacs[]) | unique)})
Demo
As for the second question, if you treated the input as an INDEX on the IPs, you could mostly reuse the code from earlier (of course, the unique part wouldn't be necessary anymore)
[INDEX(…; .associatedMacs[])[]] | group_by(.userName) | map(…)
[
{
"userName": "1",
"associatedMacs": [
"3:3:3:3:3:3",
"5:5:5:5:5:5"
]
},
{
"userName": "10",
"associatedMacs": [
"4:4:4:4:4:4",
"6:6:6:6:6:6"
]
},
{
"userName": "2",
"associatedMacs": [
"1:1:1:1:1:1"
]
},
{
"userName": "3",
"associatedMacs": [
"2:2:2:2:2:2"
]
}
]
Demo

How to print a value using json several levels above?

Given a json such as:
{
"clusters": [
{
"domain": "crap1",
"name": "BB1",
"nodes": [
{
"gpu": null,
"node": "bb1-1",
"role": "worker"
},
{
"gpu": {
"P40": 2
},
"node": "bb1-2",
"role": "master"
}
],
"site": "B-place",
"hardware": "prod-2",
"timezone": "US/Eastern",
"type": "CCE",
"subtype": null
}
]
}
where there are actually many more clusters, I want to see if I can parse the json searching for node bb1-2, for example, and print out the cluster name it belongs to BB1?
I know I can search for that node with:
.clusters[] | .nodes[] | select(.node == "bb1-2")
but can't figure out how to code it to print out a value at a higher level?
In addition to the other approaches, a very general way to hold on to higher level context is to bind it to a variable.
jq '
.clusters[] |
. as $cluster |
.nodes[] |
select(.node == "bb1-2") |
{cluster_name:$cluster.name, node:.}
'
{
"cluster_name": "BB1",
"node": {
"gpu": {
"P40": 2
},
"node": "bb1-2",
"role": "master"
}
}
This makes sure you know both the cluster and the matching node itself, and avoids the confusion that arises if your select condition matches the same cluster more than once.
How about
.clusters[] | select(.nodes[].node == "bb1-2").name
Try it:
JQ play

How to select multiple parameters from JSON output, which meets a condition & further select individual value

I have a json output, from which I need to get id value and IPv4_address value where IPv4_address exists (this shouldn't be null). Have to use this ID value for another request along with random generated string.
Here is the breakdown of the requirement :
STEP 1 :
In the following example, for the ipv4_address:1.1.1.1 & ipv4_address:1.1.1.2, i need to get the id output which is "4e-0365-4e29-95ca-329165eecf8a" and "c9061b6674a8546cea" along with IP address.
Example of my output should look like (something similar):
1.1.1.1 4e-0365-4e29-95ca-329165eecf8a
1.1.1.2 c9061b6674a8546cea
I was trying to use jq but with this I'm not able to get the both values :
ID="$(echo "$test" \n | jq -r '.USER[] | select(.ipv4_address) | .ipv4_address')"
ID1="$(echo "$test" \n | jq -r '.USER[] | select(.ipv4_address) | .id')"
Sample output which is getting displayed with the above 2 commands :
ID value is : 1.1.1.1 1.1.1.2
ID1 value is : 4e-0365-4e29-95ca-329165eecf8a c9061b6674a8546cea
STEP 2: Profile creation: I need to use each $ID1 value in another request along with random generated string. Random string is generated as per the count of $ID1's (so here I will generate 2 random string)
And thus 2 profiles are created.
Ques: How can I get each ID from the $ID1 variable ? I tried something like ID1[0] but that seems to be wrong
STEP 3 :
Will use each ID and random string for another request, Once its done or if that step is failed, i need to provide the output to a file & output should look like :
My requirement for the final output is :
1.1.1.1 4e-0365-4e29-95ca-329165eecf8a <randomvalue-1> <profile-1> DONE
1.1.1.2 c9061b6674a8546cea <randomvalue-2> <profile-2> FAILED
where random value will be generated randomly and shall be used against the ID.
JSON output which needs to be parsed:
{
"errorcode": 0,
"message": "Done",
"operation": "get",
"resourceType": "USER",
"username": "root",
"tenant_name": "Owner",
"tenant_id": "05db6674ad458546cd2",
"resourceName": "",
"USER": [
{
"is_default": "false",
"session_timeout": "0",
"permission": "root",
"name": "ee",
"session_timeout_unit": "",
"tenant_id": "55bcb6674ad45854",
"id": "4e-0365-4e29-95ca-329165eecf8a",
"ipv4_address": "1.1.1.1",
"state": "Up",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"owner"
],
"root_user": ""
},
{
"is_default": "false",
"session_timeout": "0",
"permission": "read",
"name": "test",
"session_timeout_unit": "",
"tenant_id": "bc906674ad458546cd2",
"id": "12cd0-fb7f-4abf-b060-48e98b794b06",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"read"
],
"root_user": ""
},
{
"is_default": "true",
"session_timeout": "0",
"permission": "root",
"name": "root",
"session_timeout_unit": "",
"tenant_id": "c905db6d458546cd2",
"id": "c9061b6674a8546cea",
"ipv4_address": "1.1.1.2",
"state": "Not Reachable",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"owner"
],
"root_user": ""
},
{
"is_default": "false",
"session_timeout": "0",
"permission": "readonly",
"name": "a",
"session_timeout_unit": "",
"tenant_id": "c905674ad458546cd2",
"id": "bc8a-4fd6-bc09-8c39c131b54e",
"tenant_name": "Owner",
"encrypted": "false",
"groups": [
"read"
],
"root_user": ""
}
]
}
Not quite clear with the logic of marking it DONE and FAILED. But to answer your first question where you want to select the multiple fields, you can do something like this:
$ cat input.js | jq -r '.USER[] | select(.ipv4_address) | "\(.ipv4_address) \(.id)"' > result.js
This will output the result in a file named result.js. You can apply your custom logic of marking DONE and Failed on this file.
In the above command when you do select(.ipv4_address) It basically drops all the records for which ipv4_address value is null or it is not present.
if you want to select the records which have ipv4_address as null, then your select statement would become something like this
select(.ipv4_address == null)

JQ if then statement scope

I'd like to use JQ to grab only the sub-records that match an if-then statement. When I use
jq 'if .services[].banner == "FQMDAAICCg==" then .services[].port else empty end
it grabs all of the ports for the record. (there are multiple services under each record and I want to restrict my then statement to only the services scope where I actually found the if condition).
How do I just get the port, banner, etc. for the specific service underneath the record which hit my condition?
example:
{
"services": [
{
"tls_detected": false,
"banner_is_raw": true,
"transport_protocol": "tcp",
"banner": "PCFET0NUWVBFIEhU",
"certificate": null,
"timestamp": "2020-03-22T00:38:01.074Z",
"protocol": null,
"port": 4444
},
{
"tls_detected": false,
"banner_is_raw": true,
"transport_protocol": "tcp",
"banner": "SFRUUC8xLjEgMzA",
"certificate": null,
"timestamp": "2020-03-19T01:39:45.288Z",
"protocol": null,
"port": 8080
},
{
"tls_detected": false,
"banner_is_raw": true,
"transport_protocol": "tcp",
"banner": "FQMDAAICCg==",
"certificate": null,
"timestamp": "2020-03-19T01:39:45.288Z",
"protocol": null,
"port": 8085
},
{
"tls_detected": false,
"banner_is_raw": false,
"transport_protocol": "tcp",
"banner": "Q2FjaGUtQ29ud",
"certificate": null,
"timestamp": "2020-03-20T04:25:24Z",
"protocol": "http",
"port": 8080
}
],
"ip": "103.238.62.68",
"autonomous_system": {
"description": "CHAPTECH-AS-AP Chaptech Pty Ltd",
"asn": 133493,
"routed_prefix": "103.238.62.0/24",
"country_code": "AU",
"name": "CHAPTECH-AS-AP Chaptech Pty Ltd",
"path": [
11164,
3491,
63956,
7594,
7594,
7594,
7594,
133493
]
},
"location": {
"country_code": "AU",
"registered_country": "Australia",
"registered_country_code": "AU",
"continent": "Oceania",
"timezone": "Australia/Sydney",
"latitude": -33.494,
"longitude": 143.2104,
"country": "Australia"
}
}
Update:
Thanks to peak but I couldn't get the additional goals bit working below. I ended up using
jq 'select(.services[].banner == "FQMDAAICCg==") | {port: .services[].port, banner: .services[].banner, ip: .ip}' censys.json | jq 'if .banner == "FQMDAAICCg==" then .ip,.port else empty end'
which is ugly but did the trick and still allowed me to stream the data to the first filter.
Original question
How do I just get the port, banner, etc. for the specific service underneath the record which hit my condition?
To get just the "port" for the service matching the condition, you could modify your query:
.services[]
| if .banner == "FQMDAAICCg==" then .port else empty end
Equivalently:
.services[]
| select(.banner == "FQMDAAICCg==")
| .port
Additional goal
I want to end up in this example with '8085' + '103.238.62.68'
If you really want the two values in that format, you could write something along the following lines, invoking jq with the -r option:
.ip as $ip
| (.services[] | select(.banner == "FQMDAAICCg==") | .port) as $port
| "'\($port)' + '\($ip)'"
or more briefly but less readably:
"'\(.services[] | select(.banner == "FQMDAAICCg==") | .port)' + '\(.ip)'"

How do I get jq to return unique results when json has multiple identical entries?

jq '
.[]|select(.accountEnabled==true)|select(.assignedPlans[].service=="exchange" and .assignedPlans[].capabilityStatus=="Enabled").proxyAddresses[]'
Below is a sample of json, it's the output of "az ad user list" (getting the Active Directory userlist from Azure) anonymised and with irrelevant things removed. Above is a jq command that I want to use to extract email addresses, the desired output is "SMTP:russell.coker#example.com" printed once not 9 times. Yes, I know I could pipe this to the Unix command "sort -u" but I'd like to do other json queries on it.
[
{
"accountEnabled": true,
"assignedPlans": [
{
"capabilityStatus": "Enabled",
"service": "exchange"
},
{
"capabilityStatus": "Enabled",
"service": "exchange"
},
{
"capabilityStatus": "Enabled",
"service": "exchange"
}
],
"provisionedPlans": [
{
"capabilityStatus": "Enabled",
"provisioningStatus": "Success",
"service": "exchange"
},
{
"capabilityStatus": "Enabled",
"provisioningStatus": "Success",
"service": "exchange"
},
{
"capabilityStatus": "Enabled",
"provisioningStatus": "Success",
"service": "exchange"
},
{
"capabilityStatus": "Enabled",
"provisioningStatus": "Success",
"service": "exchange"
}
],
"proxyAddresses": [
"SMTP:russell.coker#example.com"
]
},
{
"accountEnabled": true,
"assignedPlans": [
{
"capabilityStatus": "Deleted",
"service": "exchange"
},
{
"capabilityStatus": "Deleted",
"service": "OfficeForms"
}
],
"provisionedPlans": [
{
"capabilityStatus": "Deleted",
"provisioningStatus": "Success",
"service": "SharePoint"
},
{
"capabilityStatus": "Deleted",
"provisioningStatus": "Success",
"service": "exchange"
},
{
"capabilityStatus": "Deleted",
"provisioningStatus": "Success",
"service": "exchange"
}
],
"proxyAddresses": [
"smtp:a#example.com",
"smtp:b#example.com",
"SMTP:c#example.com"
]
}
]
Perhaps the problem is that the given jq query is simply "wrong" in that it does not capture the OP's intent.
Even if the following query does not reflect the OP's intent, it is worth noting that, with the given JSON, it produces the single result that is wanted:
.[]
| select(.accountEnabled==true)
| select(any(.assignedPlans[];
.service=="exchange" and
.capabilityStatus=="Enabled"))
| .proxyAddresses[]
Likewise ....
Here's another query with different semantics but which, with the given JSON, also produces the single desired result. (It goes to show that a single example by itself is no substitute for requirements.)
.[]
| select(.accountEnabled==true)
| select(any(.assignedPlans[]; .service=="exchange"))
| select(any(.assignedPlans[]; .capabilityStatus=="Enabled"))
| .proxyAddresses[]
Above is a jq command that I want to use
The following response focuses on the above requirement.
unique/0 could be used if you don't mind the fact that it sorts its input. This filter expects an array as input, and so you could modify your query as follows:
[.[]
| select(.accountEnabled==true)
| select(.assignedPlans[].service=="exchange" and .assignedPlans[].capabilityStatus=="Enabled")
| .proxyAddresses[]]
| unique
This produces an array, so if you want a stream, simply tack on [] at the end.
A stream-oriented approach
Under some circumstances, it may be desirable to avoid the sort that unique/0 uses. Here is a stream-oriented solution using a generic filter, uniques/1, which involves no sorting and which has other potential advantages, though it is a bit tricky to define because it puts no restrictions on the stream.
def uniques(stream):
foreach stream as $s ({};
($s|type) as $t
| (if $t == "string" then $s else ($s|tostring) end) as $y
| if .[$t][$y]
then .emit = false
else .emit = true | (.item = $s) | (.[$t][$y] = true)
end;
if .emit then .item else empty end );
Using uniques/1, a small tweak to the previous solution is sufficient:
uniques(.[]
| select(.accountEnabled==true)
| select(.assignedPlans[].service=="exchange" and .assignedPlans[].capabilityStatus=="Enabled")
| .proxyAddresses[] )