I need to grab the value of the first entry in a json array with Kusto KQL in Microsoft Defender ATP.
The data format looks like this (anonymized), and I want the value of "UserName":
[{"UserName":"xyz","DomainName":"xyz","Sid":"xyz"}]
How do I split or in any other way get the "UserName" value?
In WDATP/MSTAP, for the "LoggedOnUsers" type of arrays, you want "mv-expand" (multi-value expand) in conjunction with "parsejson".
"parsejson" will turn the string into JSON, and mv-expand will expand it into LoggedOnUsers.Username, LoggedOnUsers.DomainName, and LoggedOnUsers.Sid:
DeviceInfo
| mv-expand parsejson(LoggedOnUsers)
| project DeviceName, LoggedOnUsers.UserName, LoggedOnUsers.DomainName
Keep in mind that if the packed field has multiple entries (like DeviceNetworkInfo's IPAddresses field often does), the entire row will be expanded once per entry - so a row for a machine with 3 entries in "IPAddresses" will be duplicated 3 times, with each different expansion of IpAddresses:
DeviceNetworkInfo
| where Timestamp > ago(1h)
| mv-expand parsejson(IPAddresses)
| project DeviceName, IPAddresses.IPAddress
to access the first entry's UserName property you can do the following:
print d = dynamic([{"UserName":"xyz","DomainName":"xyz","Sid":"xyz"}])
| extend result = d[0].UserName
to get the UserName for all entries, you can use mv-expand/mv-apply:
print d = dynamic([{"UserName":"xyz","DomainName":"xyz","Sid":"xyz"}])
| mv-apply d on (
project d.UserName
)
thanks for the reply, but the proposed solution didn't work for me. However instead I found the following solution:
project substring(split(split(LoggedOnUsers,',',0),'"',4),2,9)
The output of this is: UserName
Related
I'm trying to select the SerialNumber of a specific AWS MFADevice for different profiles.
This command returns the list of MFADevices for a certain profile:
aws iam list-mfa-devices --profile xxx
and this is a sample JSON output:
{
"MFADevices": [
{
"UserName": "foobar#example.com",
"SerialNumber": "arn:aws:iam::000000000000:mfa/foo",
"EnableDate": "2022-12-06T16:23:41+00:00"
},
{
"UserName": "barfoo#example.com",
"SerialNumber": "arn:aws:iam::111111111111:mfa/bar_cli",
"EnableDate": "2022-12-12T09:13:10+00:00"
}
]
}
I would like to select the SerialNumber of the device containing the string cli. But in case there is only one device in the list (regardless of the presence or absence of the string cli), I'd like to get its SerialNumber.
I have this expression which already filters for the first condition, namely the desired string:
aws iam list-mfa-devices --profile xxx --query 'MFADevices[].SerialNumber | [?contains(#,`cli`)] | [0]'
However I still haven't been able to figure out how to add the if number_of_devices == 1 then return the serial of that single device.
I can get the number of MFADevices with this command:
aws iam list-mfa-devices --profile yyy --query 'length(MFADevices)'
And as a first step towards my final solution I wanted to initially get the SerialNumber only in the case the list has exactly one element, so, I thought of something like this:
aws iam list-mfa-devices --profile yyy --query 'MFADevices[].SerialNumber | [?length(MFADevices) ==`1`]'
but actually already at this stage I get the error below (left alone the fact that I still need to combine it with the cli part):
In function length(), invalid type for value: None, expected one of: ['string', 'array', 'object'], received: "null"
Does anybody know how to achieve what I want?
I know that I could just pipe the raw output to jq and do the filtering there, but I was wondering if there is a way to do it directly in the command using some JMESPath expression.
In order to do those kind of condition in JMESPath you will have to rely on logical or (||) and logical and (&&), because the language does not have a conditional keyword, per se.
So, in pseudo-code, instead of doing:
if length(MFADevices) == 1
MFADevices[0]
else
MFADevices[?someFilter]
You have to do, like in bash:
length(MFADevices) == 1 and MFADevices[0] or MFADevices[?someFilter]
So, in JMESPath:
length(MFADevices) == `1`
&& MFADevices[0].SerialNumber
|| (MFADevices[?contains(SerialNumber, `cli`)] | [0]).SerialNumber
Note: this assumes that, if there are more than one element but none contains cli, we should get null.
If you want the first element, even when there are multiple devices and the SerialNumber does not contains cli, then you can simplify it further and simply do a logical or, when the contains filter return nothing (as a null result will evaluates to false):
(MFADevices[?contains(SerialNumber, `cli`)] | [0]).SerialNumber
|| MFADevices[0].SerialNumber
With stedolan/jq you can filter for the substring and unconditonally add the first, then take the first of them:
.MFADevices | map(.SerialNumber) | first((.[] | select(contains("cli"))), first)
Demo
or
[.MFADevices[].SerialNumber] | map(select(contains("cli"))) + .[:1] | first
Demo
Output:
arn:aws:iam::111111111111:mfa/bar_cli
My dataframe looks like
|ID|Notes|
---------------
|1|'{"Country":"USA","Count":"1000"}'|
|2|{"Country":"USA","Count":"1000"}|
ID : int
Notes : string
When i use from_json to parse the column Notes, it gives all Null values.
I need help in parsing this column Notes into columns in pyspark
When you are using from_json() function, make sure that the column value is exactly a json/dictionary in String format. In the sample data you have given, the Notes column value with id=1 is not exactly in json format (it is a string but enclosed within additional single quotes). This is the reason it is returning NULL values. Implementing the following code on the input dataframe gives the following output.
df = df.withColumn("Notes",from_json(df.Notes,MapType(StringType(),StringType())))
You need to change your input data such that the entire Notes column is in same format which is json/dictionary as a string and nothing more because it is the main reason for the issue. The below is the correct format that helps you to fix your issue.
| ID | Notes |
---------------
| 1 | {"Country":"USA","Count":"1000"} |
| 2 | {"Country":"USA","Count":"1000"} |
To parse Notes column values as columns in pyspark, you can simply use function called json_tuple() (no need to use from_json()). It extracts the elements from a json column (string format) and creates the result as new columns.
df = df.select(col("id"),json_tuple(col("Notes"),"Country","Count")) \
.toDF("id","Country","Count")
df.show()
Output:
NOTE: json_tuple() also returns null if the column value is not in the correct format (make sure the column values are json/dictionary as a string without additional quotes).
I have a database of resumes in json format that I want to reshape so that each row corresponds to a person's employment history at a given company:
personid, company_name, start_date, end_date
However, running the following jq command
{personid:.personid, company_name: .experience[].company.name, sdate: .experience[].start_date, edate: .experience[].end_date}
produces the cartesian product of all the fields (3 jobs x 3 fields). For example, a person who has held 3 jobs at 3 different companies in the past looks like this after running the jq command above:
{"id":"abc123","companyname":"companyA","sdate":"2020-06","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2020-06","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2020-06","edate":"2017-07"}
{"id":"abc123","companyname":"companyA","sdate":"2016-10","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2016-10","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2016-10","edate":"2017-07"}
{"id":"abc123","companyname":"companyA","sdate":"2017-05","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2017-05","edate":null}
{"id":"abc123","companyname":"companyA","sdate":"2017-05","edate":"2017-07"}
There are 9 entries for CompanyB and CompanyC each but I truncated the output above for brevity.
I think I need to use the group_by() command, but I've been unsuccessful.
Thanks in advance.
Without seeing the original data, my guess is that you get the cartesian product because you are iterating three times (.experience[]) within the object construction. You might want to pull out the iteration, maybe save it in a variable, and reference that instead:
.experience[] as $experience | {
personid: .personid,
company_name: $experience.company.name,
sdate: $experience.start_date,
edate: $experience.end_date
}
Depending on the outer structure of your construction, also the other way around may be appropriate, ie. storing the .person field instead in a variable:
.personid as $id | .experience[] | {
personid: $id,
company_name: .company.name,
sdate: .start_date,
edate: .end_date
}
I am retrieving data from mysql db. All the data is one column. I need to separate this into several cols: The structure of this col is as follows:
{{product ID=001 |Country=Netherlands |Repository Link=http://googt.com |Other Relevant Information=test }} ==Description== this are the below codes: code 1 code2 ==Case Study== case study 1 txt case study 2 txt ==Benefits== ben 1 ben 2 === Requirements === (empty col) === Architecture === *arch1 *arch2
So I want cols like: Product ID, Country, Repository Link, Architecture etc.....
If you are planning on simply parsing out the output of your column, it will depend on the language of choice you are currently using.
However, in general the procedure for doing this is as follows.
1, pull output into string
2, find a delimiter(In you case it appears '|' will do)
3, you have to options here(again depending on language)
A, Split each segment into an array
1, Run array through looping structure to print out each section OR use array
to manipulate data individually(your choice)
B, In Simple String method, you can either create a new string, or replace all
instances of '|' with '\n'(new line char) so that you can display all data.
I recommend the array conversion as this will allow you to easily interact with the data in a simple manner.
This is often something done today with json and other such formats which are often stored in single fields for various reasons.
Here is an example done in php making use of explode()
$unparsed = "this | is | a | string that is | not: parsed";
$parsed = explode("|", $unparsed);
echo $parsed[2]; // would be a
echo $parsed[4]; // would be not: parsed
I am selecting nodes in a JSON input but can't find a way to include parent object detail for each array entry that I am querying. I am using pentaho data integration to query the data using JSON input form a mongodb input.
I have also tried to create a mongodb query to achieve the same but cannot seem to do this either.
Here are the two fields/paths that display the data:
$.size_break_costs[*].size
$.size_break_costs[*].quantity
Here is the json source format:
{
"_id" : ObjectId("4f1f74ecde074f383a00000f"),
"colour" : "RAVEN-SMOKE",
"name" : "Authority",
"size_break_costs" : [
{
"quantity" : NumberLong("80"),
"_id" : ObjectId("518ffc0697eee36ff3000002"),
"size" : "S"
},
{
"quantity" : NumberLong("14"),
"_id" : ObjectId("518ffc0697eee36ff3000003"),
"size" : "M"
},
{
"quantity" : NumberLong("55"),
"_id" : ObjectId("518ffc0697eee36ff3000004"),
"size" : "L"
}
],
"sku" : "SK3579"
}
I currently get the following results:
S,80
M,14
L,55
I would like to get the SKU and Name as well as my source will have multiple products (SKU/Description):
SK3579,Authority,S,80
SK3579,Authority,M,14
SK3579,Authority,L,55
When I try To include using $.sku, I the process errors.
The end result i'm after is a report of all products and the available quantities of their various sizes. Possibly there's an alternative mongodb query that provides this.
EDIT:
It seems the issue may be due to the fact that not all lines have the same structure. For example the above contains 3 sizes - S,M,L. Some products come in one size - PACK. Other come in multiple sizes - 28,30,32,33,34,36,38 etc.
The error produced is:
*The data structure is not the same inside the resource! We found 1 values for json path [$.sku], which is different that the number retourned for path [$.size_break_costs[].quantity] (7 values). We MUST have the same number of values for all paths.
I have tried the following mongodb query separately which gives the correct results, but the corresponding export of this doesn't work. No values are returned for the Size and Quantity.
Query:
db.product_details.find( {}, {sku: true, "size_break_costs.size": true, "size_break_costs.quantity": true}).pretty();
Export:
mongoexport --db brandscope_production --collection product_details --csv --out Test01.csv --fields sku,"size_break_costs.size","size_break_costs.quantity" --query '{}';
Shortly after I added my own bounty, I figured out the solution. My problem has the same basic structure, which is a parent identifier, and some number N child key/value pairs for ratings (quality, value, etc...).
First, you'll need a JSON Input step that gets the SKU, Name, and size_break_costs array, all as Strings. The important part is that size_break_costs is a String, and is basically just a stringified JSON array. Make sure that under the Content tab of the JSON Input, that "Ignore missing path" is checked, in case you get one with an empty array or the field is missing for some reason.
For your fields, use:
Name | Path | Type
ProductSKU | $.sku | String
ProductName | $.name | String
SizeBreakCosts | $.size_break_costs | String
I added a "Filter rows" block after this step, with the condition "SizeBreakCosts IS NOT NULL", which is then passed to a second JSON Input block. This second JSON block, you'll need to check "Source is defined in a field?", and set the value of "Get source from field" to "SizeBreakCosts", or whatever you named it in the first JSON Input block.
Again, make sure "Ignore missing path" is checked, as well as "Ignore empty file". From this block, we'll want to get two fields. We'll already have ProductSKU and ProductName with each row that's passed in, and this second JSON Input step will further split it into however many rows are in the SizeBreakCosts input JSON. For fields, use:
Name | Path | Type
Quantity | $.[*].quantity | Integer
Size | $.[*].size | String
As you can see, these paths use "$.[*].FieldName", because the JSON string we passed in has an array as the root item, so we're getting every item in that array, and parsing out its quantity and size.
Now every row should have the SKU and name from the parent object, and the quantity and size from each child object. Dumping this example to a text file, I got:
ProductSKU;ProductName;Size;Quantity
SK3579;Authority;S; 80
SK3579;Authority;M; 14
SK3579;Authority;L; 55