I'm trying to setup some monitoring. As part of that I need to parse some gnarly json output to retrieve a node ID, which changes each time the node is rebooted or the service restarts. I always know the node name but not the "id". The JSON looks something like this:
{
"cluster_name":"cluster1",
"nodes":
{
"generatednodeid1":{"name":"node01"},
"generatednodeid2":{"name":"node2"}
}
}
Doing .nodes | keys gives me ["generatednodeid1","generatednodeid2"] as I'd expect.
I've tried .nodes[] | select(.name=="node2") but that only outputs {"name":"node2"}
What I really need to happen is if .name=="node2" then it gives me generatednodeid2
I've been beating my head against a wall. I can't for the life of me figure out what I'm missing. This seems so simple (probably is and I've looked at it too long). Any ideas?
Any ideas?
In this situation, the "to_entries" family of filters is helpful, e.g.:
.nodes
| to_entries[]
| select(.value.name == "node2")
| .key
Related
I'm trying to extract some information from a nested JSON in log analytics.
It is nested several levels deep though. And I come unstuck at the 3rd tier.
The scenario is to query on what user ID has had permissions removed in Azure. the ifnormation is all there is raw format, but I want to extract it to be more readable.
The data layout is :
AzureActivity
Properties_d
responseBody
properties
principalId
It's the principalID I want (getting a UPN from AAD comes later ;)
My query works to a point. But the _propertieslevel3 comes up blank (no error).
_resonsebody is fine. It is a dynamic JSON that contains the responsebody field from Properties_d.
AzureActivity
| where (OperationNameValue contains "ROLEASSIGNMENTS/DELETE" and ActivityStatusValue contains "SUCCESS")
| extend _responsebody = parse_json(Properties_d.responseBody)
| extend _propertieslevel3 = parse_json(_responsebody.properties)
| extend ModifiedUser = parse_json(_propertieslevel3.principalId)
as _propertieslevel3 comes back blank, so does modified user. I can only guess that there is a problem trying to nest this deep.
Any ideas?
TIA.
data sample of Properties_d
{"eventCategory":"Administrative",
"eventDataId":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"eventSubmissionTimestamp":"2022-03-09T16:53:26.4493278Z",
"resource":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"resourceProviderValue":"MICROSOFT.AUTHORIZATION",
"subscriptionId":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"activityStatusValue":"Success",
"entity":"/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxx/providers/Microsoft.Authorization/roleAssignments/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"message":"Microsoft.Authorization/roleAssignments/delete",
"hierarchy":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"caller":"xxxxxx#xxxxxxx",
"httpRequest":"{\"clientIpAddress\":\"3.3.3.3\"}",
"statusCode":"OK",
"serviceRequestId":"",
"activitySubstatusValue":"OK",
"responseBody":"{\"properties\":{\"roleDefinitionId\":\"/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxx/providers/Microsoft.Authorization/roleDefinitions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\",
\"principalId\":\"xxxxxxxxxxxxxxxxxxxxxxxxxxxx\",
\"principalType\":\"User\",
\"scope\":\"/subscriptions/xxxxxxxxxxxxxxxxxxxxxx\",
\"condition\":null,
\"conditionVersion\":null,
\"createdOn\":\"2022-03-09T11:28:48.4781104Z\",
\"updatedOn\":\"2022-03-09T11:28:48.4781104Z\",
\"createdBy\":\"xxxxxxxxxxxxxxxxxxxxxxxxx\",
\"updatedBy\":\"xxxxxxxxxxxxxxxxxxxxxxx\",
\"delegatedManagedIdentityResourceId\":null,
\"description\":null},
\"id\":\"/subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/providers/Microsoft.Authorization/roleAssignments/xxxxxxxxxxxxxxxxxxxxxx\",
\"type\":\"Microsoft.Authorization/roleAssignments\",
\"name\":\"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"}"}
Most likely, you need to apply parse_json() on the nested property bag too.
See: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/parsejsonfunction
got it :)
Not sure why I needed to make _propertieslevel3 be just be the same as response body, rather than being able to extract .properties
but it works.
Thanks.
AzureActivity
| where (OperationNameValue contains "ROLEASSIGNMENTS/WRITE" and ActivityStatusValue contains "Start")
| extend _responsebody = parse_json(Properties_d.responseBody)
| extend _propertieslevel3 = parse_json(tostring(parse_json(_responsebody)))
| extend _level4 = parse_json(tostring(parse_json(_propertieslevel3.properties)))
| extend ModifiedUser = parse_json(tostring(parse_json(_level4.principalId)))
I am very new to tcl. In my implementation, my input is a nested-json file. I need to store the values along with the keys in a txt file.
I converted the json file to dict and then by using jsonget from http://wiki.tcl.tk/13419, I could provide the keys and found out the values of each key iteratively very easily.
Now, my problem is, my json file might get updated later. Hence, I will not know which key has been added in the json, thus I will not be able to get the values too.
My code must be generic and must apply to all these dynamic json files and it must be able to give all these keys and along with their values.
For example
{"test":[{"a":1,"b":2}]} }
result:
a 1
b 2
{"test":[{"a":1,"b":2,"c":3}]}
result:
a 1
b 2
c 3
{"test":[{"a":1,"b":2,"c":[{"d":4,"e":5}]]}
result:
a 1
b 2
c d 4
e 5
All this, without requiring any change in the code. Is there a way to do this?
Neither your example, nor your wording ("requirement") are particularly helpful. I can only second-guess on your intentions: Do you want to know how to write a piece of Tcl that processes selected content of the JSON document in a generic (regular) manner, repeatedly, without knowledge about its structure or actual literal content (e.g., key labels)?
If yes, ...
... does this help?
proc foo {json} {
foreach k [dict keys [lindex [dict get $json test] 0]] {
puts "$k => [jsonget $json test 0 $k]"
}
}
foo [json::json2dict { {"test":[{"a":1,"b":2}]} }];
foo [json::json2dict { {"test":[{"a":1,"b":2,"c":3}]} }];
foo [json::json2dict { {"test":[{"a":1 }]}}];
It does not add much to what glenn has pointed you to.
If no, ...
... refine your example, wording.
General recommendation
For serious JSON-based development, consider rl_json or tdom.
I have a problem accessing json-objects of predictable structure but unknown depth in Powershell. So the json-objects contain information that can be connected by "and" and "or", but those connections can be used in several levels. As an exanple:
$ab=#"
{
"cond": "one",
"and": [
{"cond": "two"},
{"cond": "three"},
{"or": [{"cond": "four"},
{"cond": "five"}
]
}
]
}
"# | ConvertFrom-Json
I need to be able to read/test something like
$test="and.or"
$ab.$test.cond
where $test is a combination of several "and"s and "or"s like and.or.or.and .
The problem is that I can't figure out how my idea of $ab.$test.cond is to be written in Powershell to work. In theory I could test all possible combinations to a given depth by hand, but I'd prefer not to. Does anyhow have an idea how this could work? Thanks a lot!
(Powershell Version 5)
I think you should define a proper set of classes for your conditional engine/descriptors, either using PowerShell classes or using C# to create an assembly so you can use the types within PowerShell.
But for a quick and dirty PowerShell solution, you could do this:
"`$ab.$test.cond" | Invoke-Expression
# or
'$ab.{0}.cond' -f $test | Invoke-Expression
This has no error checking of course. Any other solution is likely going to be a separate recursive function if you want to get real checking and such, but it will be more fragile then using a well-defined set of objects.
I have a file with lines like this:
{"items":["blue","green"]}
{"items":["yellow","green"]}
{"items":["blue","pink"]}
How can I use jq to select and show only the JSON values that have "blue" in their "items" array?
So the output would be:
{"items":["blue","green"]}
{"items":["blue","pink"]}
Found out the answer
jq 'select(.items | index("blue"))'
On Jan 30, 2017, a builtin named IN was added for efficiently testing whether a JSON entity is contained in a stream. It can also be used for efficiently testing membership in an array. In the present case, the relevant usage would be:
select( .items as $items | "blue" | IN($items[]) )
If your jq does not have IN/1, then so long as your jq has first/1, you can use this equivalent definition:
def IN(s): . as $in | first(if (s == $in) then true else empty end) // false;
any/0
Using any/0 here is relatively inefficient, e.g. compared to using any/1:
select( any( .items[]; . == "blue" ))
(In practice, index/1 is usually fast enough, but its implementation currently (jq 1.5 and versions through at least July 2017) is suboptimal.)
While what you have certainly works, it would be more correct to use contains. I would avoid that use since it can lead to confusion. index("blue") is 0 and one wouldn't consider that a truthy value and might expect it to be excluded from the results.
Consider using this filter instead:
select(.items | contains(["blue"]))
This has the added benefit that it would work if you wanted items with more than one match by simply adding more to the array.
As Will pointed out in the comments, this isn't quite correct. Strings are compared using substring matching (contains is used recursively) here.
In retrospect, contains didn't work out as I thought it would. Using index works, but personally I wouldn't use it. There's something about figuring out if an item is in a collection by looking for it's index that feels wrong to me. Using contains makes more sense to me, but in light of this information, it wouldn't be ideal in this case.
Here's an alternative that should work correctly:
select([.items[] == "blue"] | any)
Or for a more scalable way if you wanted to be able to match more values:
select(.items as $values | ["blue", "yellow"] | map([$values[] == .] | any) | all)
I have needed to use 'regex' for the same situation of the objects. (In another context, of course). I write the code because I did not find a solution for my need in these pages. This can be useful for someone.
For example, to match the blue color using a regular expression:
jq 'select(.items[]|test("bl.*"))' yourfile.json
jqPlay
I've just started to read upon graph-teory and data structures.
I'm building an example application which should be able to find the xpath for the most common links. Imagine a Google serp, my application should be able to find the xpath for all links pointing to a result.
Imagine that theese xpaths were found:
/html/body/h2/a
/html/body/p/a
/html/body/p/strong/a
/html/body/p/strong/a
/html/body/p/strong/a
/html/body/div[#class=footer]/span[#id=copyright]/a
From these xpats, i've thought of a graph like this (i might be completely lost here):
html
|
body
h2 - p - div[#class=footer]
| | |
a (1) a - strong span[#id=copyright]
| |
a (3) a (1)
Is this the best approach to this problem?
What would be the best way (data structure) to store this in memory? The language does not mather. We can see that we have 3 links matching the path html -> body -> p -> strong -> a.
As I said, i'm totally new to this so please forgive me if I thought of this completely wrong.
EDIT: I may be looking for the trie data structure?
Don't worry about tries yet. Just construct a tree using standard graph representation (node = {value, count, parent} while immediately collapsing same branches and incrementing the counter. Then, sort all the leaves by count in descending order and traverse from each leaf upwards to get a path.