I'm getting a JSON output from the VirusTotal API, as seen below:
{
"scan_id": "id_goes_here",
"resource": "domain_goes_here",
"url": "http://example.com/",
"response_code": 1,
"scan_date": "2021-01-19 21:28:32",
"permalink": "https://www.virustotal.com/gui/url/id_goes_here/detection/u-id_goes_here",
"verbose_msg": "Scan finished, scan information embedded in this object",
"filescan_id": null,
"positives": 0,
"total": 83,
"scans": {
"CMC Threat Intelligence": {
"detected": false,
"result": "clean site"
},
"CLEAN MX": {
"detected": false,
"result": "clean site"
},
"DNS8": {
"detected": false,
"result": "clean site"
},
"MalwareDomainList": {
"detected": false,
"result": "clean site",
"detail": "http://www.malwaredomainlist.com/mdl.php?search=example.com"
...
I want to go through this JSON and find the entries that have detected as true and if so print the name. As well, print the reference (if there is one)
For example, let's say the entry for MalwareDomainList had detected set to true. So as below:
"MalwareDomainList": {
"detected": true,
"result": "malicious site",
"detail": "http://www.malwaredomainlist.com/mdl.php?search=example.com"
I want to print the name of the list, so in this case, MalwareDomainList and also print the value of detail. As you can see in the above output, not all lists have a detail field.
In python3, I am thinking I need to have something like:
scan_content = json.dumps(response.json(), indent =4 )
for scan in scan_content["scans"]:
if scan["detected"]:
print("Name of blacklist:", scan["name"])
But getting an error since name is not an actual key in the output. Not sure how to alter my above code to achieve the following:
If detected is set to true
Print the name of the list
If applicable, print value in the detail key
Trying the below solution as such:
I have the following:
json_data = json.dumps(response.json(), indent =4 )
scan_content = json_data["scans"]
positive = []
for elem in scan_content.keys():
if scan_content[elem]["detected"] == "true":
positive.append({"Name" : elem, "Detail" : scan_content[elem]["detail"]})
print(*positive)
I get the error
scan_content = json_data["scans"]
TypeError: string indices must be integers
Related
I'm trying to sort a JSON String in Power Automate by a nested field called "orderHint".
My JSON String looks like this:
[
{
"id": "5134",
"value": {
"isChecked": false,
"title": "This is another test",
"orderHint": "8585298133570680672PF"
},
"lastModifiedDateTime": "2022-12-23T11:06:28.4256622Z"
},
{
"id": "26576",
"value": {
"isChecked": true,
"title": "This is a test",
"orderHint": "8585498133570680672DE"
},
"lastModifiedDateTime": "2022-12-23T11:06:28.4256622Z"
}
]
When I'm trying to sort by "orderHint", I get an error:
"'The template language function 'sort' did not find the named sortField 'orderHint' on one or more objects in the array."
I'm using the following expression:
sort(variables('varArrayChecked'), 'value/orderHint')
Sorting by other fields works fine, e.g.:
sort(variables('varArrayChecked'), 'id')
Is there any way how I can sort by a nested field in a JSON String?
Thanks in advance!
You can use the Advanced Data Operations connector as it will do it for you in a single step.
The Flatten Object Array step is perfect for the payload you've provided.
You can see that it will take the data, flatten it and you have the ability to sort it on the way out (noting that the Array variable contains the exact JSON you provided in your question) ...
Note: Balance Output must be set to true in order for the sorting to occur.
Result
This is the resulting JSON order by orderHint ascending.
[
{
"id": "5134",
"lastModifiedDateTime": "2022-12-23T11:06:28",
"value/isChecked": false,
"value/orderHint": "8585298133570680672PF",
"value/title": "This is another test"
},
{
"id": "26576",
"lastModifiedDateTime": "2022-12-23T11:06:28",
"value/isChecked": true,
"value/orderHint": "8585498133570680672DE",
"value/title": "This is a test"
}
]
... and to show it in descending order (which is obvious, but simply change the sort order object value from Asc to Desc) ...
[
{
"id": "26576",
"lastModifiedDateTime": "2022-12-23T11:06:28",
"value/isChecked": true,
"value/orderHint": "8585498133570680672DE",
"value/title": "This is a test"
},
{
"id": "5134",
"lastModifiedDateTime": "2022-12-23T11:06:28",
"value/isChecked": false,
"value/orderHint": "8585298133570680672PF",
"value/title": "This is another test"
}
]
It there is a way to process the result of States.StringToJson intesic function directly ?
Currently in a step function, I try to handle the error from another synchronous step function call :
"OtherStepFunction": {
"Type": "Task",
"Resource": "arn:aws:states:::states:startExecution.sync:2",
"Parameters": {
"StateMachineArn": "otherstepFunctionCall",
"Input.$": "$"
},
"End": true,
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Comment": "OtherStepFunctionFailed",
"Next": "StatusStepFunctionFailed",
"ResultPath": "$.error"
}
]
},
All errors goes in a pass flow named StatusStepFunctionFailed, with the errors output in $.error path.
The $.error is composed of the error type and the cause as an escapedJson string.
"error": {
"Error": "States.TaskFailed",
"Cause": "{\"ExecutionArn\":\"otherfunctionarm:executionid\",\"Input\":\"foooooo\"}"
}
Is there any way to extract only the ExecutionARN from this input ? In my pass step, I convert the Cause path as a json, but i didn't find a way to select directly the ExectionARN part. The following :
"reason.$": "States.JsonMerge($.error.Cause).ExecutionArn"
return The value for the field 'reason.$' must be a valid JSONPath or a valid intrinsic function call (at /States/HandleResource/Iterator/States/StatusStepFunctionFailedHandleJSON/Parameters)
My current workaround is to use 2 pass flow, first convert the output and then formating.
I had a similar issue.
What I did was create a task to put the Cause into a new path parameter using StringToJSON. I put that task as the next from the error and then called the subsequent task from that one.
Using your variable names and values:
In the Catch, change the Next from StatusStepFunctionFailed to parseErrorCause
Then parseErrorCause is like this:
"parseErrorCause": {
"Type": "Pass",
"Parameters": {
"Result.$": "States.StringToJson($.error.Cause)"
},
"ResultPath": "$.parsedJSON",
"Next": "StatusStepFunctionFailed"
},
And StatusStepFunctionFailed accesses
"Variable": "$.parsedJSON.Result.Input",
to get foooooo
See attached screenshot, when processing the messages in a user's folder, when the message is an "eventMessageRequest" then the json is invalid: it contains the "type" key twice and the Json parsers can't deal with it.
screenshot of original json received from MS Graph
abbreviated json: note the "type" key used twice:
[
{
"type": "#microsoft.graph.eventMessageRequest",
"createdDateTime": "2020-07-01T18:21:32Z",
"lastModifiedDateTime": "2020-07-01T18:21:33Z",
"categories": [
],
"receivedDateTime": "2020-07-01T18:21:33Z",
"sentDateTime": "2020-07-01T18:21:29Z",
"hasAttachments": true,
"isDeliveryReceiptRequested": null,
"isReadReceiptRequested": false,
"isRead": true,
"isDraft": false,
"inferenceClassification": "focused",
"meetingMessageType": "meetingRequest",
"type": "singleInstance",
"isOutOfDate": false,
"isAllDay": false,
"isDelegated": false,
"responseRequested": true,
"allowNewTimeProposals": null,
"meetingRequestType": "informationalUpdate"
}
]
I have a JSON response that looks like this:
{
"results": [
{
"entityType": "PERSON",
"id": 679,
"graphId": "679.PERSON",
"details": [
{
"entityType": "PERSON",
"id": 679,
"graphId": "679.PERSON",
"parentId": 594,
"role": "Unspecified Person",
"relatedEntityType": "DOCUMENT",
"relatedId": 058,
"relatedGraphId": "058.DOCUMENT",
"relatedParentId": null
}
]
},
{
"entityType": "PERSON",
"id": 69678,
"graphId": "69678.PERSON",
"details": [
{
"entityType": "PERSON",
"id": 678,
"graphId": "678.PERSON",
"parentId": 594,
"role": "UNKNOWN",
"relatedEntityType": "DOCUMENT",
"relatedId": 145,
"relatedGraphId": "145.DOCUMENT",
"relatedParentId": null
}
]
}
The problem with this JSON response is that $.results[0] is not always the same, and it can have dozens of results. I know I can do individual JSON Assertion calls where I do the JSON with a wild card
$.results[*].details[0].entityType
$.results[*].details[0].relatedEntityType
etc
However I need to verify that both "PERSON" and "DOCUMENT" match correctly in the same path on one api call since the results come back in a different path each time.
Is there a way to do multiple calls in one JSON Assertion or am I using the wrong tool?
Thanks in advance for any help.
-Grav
I don't think JSON Assertion is flexible enough, consider switching to JSR223 Assertion where you have the full flexibility in terms of defining whatever pass/fail criteria you need.
Example code which checks that:
all attributes values which match $.results[*].details[0].entityType query are equal to PERSON
and all attributes values which match $.results[*].details[0].relatedEntityType are equal to DOCUMENT
would be:
def entityTypes = com.jayway.jsonpath.JsonPath.read(prev.getResponseDataAsString(), '$.results[*].details[0].entityType').collect().find { !it.equals('PERSON') }
def relatedEntityTypes = com.jayway.jsonpath.JsonPath.read(prev.getResponseDataAsString(), '$.results[*].details[0].relatedEntityType').collect().find { !it.equals('DOCUMENT') }
if (entityTypes.size() != 1) {
SampleResult.setSuccessful(false)
SampleResult.setResponseMessage('Entity type mismatch, one or more entries are not "PERSON" ' + entityTypes)
}
if (relatedEntityTypes.size() != 1) {
SampleResult.setSuccessful(false)
SampleResult.setResponseMessage('Entity type mismatch, one or more entries are not "DOCUMENT" ' + relatedEntityTypes)
}
More information:
SampleResult class JavaDoc
Groovy: Working with collections
Scripting JMeter Assertions in Groovy - A Tutorial
Here is a simplified json file of a terraform state file (let's call it dev.ftstate)
{
"version": 4,
"terraform_version": "0.12.9",
"serial": 2,
"lineage": "ba56cc3e-71fd-1488-e6fb-3136f4630e70",
"outputs": {},
"resources": [
{
"module": "module.rds.module.reports_cpu_warning",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds.module.reports_lag_warning",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds.module.cross_region_replica_lag_alert",
"mode": "managed",
"type": "datadog_monitor",
"name": "alert",
"each": "list",
"provider": "module.rds.provider.datadog",
"instances": []
},
{
"module": "module.rds",
"mode": "managed",
"type": "aws_db_instance",
"name": "master",
"provider": "provider.aws",
"instances": [
{
"schema_version": 0,
"attributes": {
"address": "dev-database.123456.us-east-8.rds.amazonaws.com",
"allocated_storage": 10,
"password": "",
"performance_insights_enabled": false,
"tags": {
"env": "development"
},
"timeouts": {
"create": "6h",
"delete": "6h",
"update": "6h"
},
"timezone": "",
"username": "admin",
"vpc_security_group_ids": [
"sg-1234"
]
},
"private": ""
}
]
}
]
}
There are many modules at the same level of module.rds inside the instances. I took out many of them to create the simplified version of the raw data. The key takeway: do not assume the array index will be constant in all cases.
I wanted to extract the password field in the above example.
My first attempt is to use equality check to extract the relevant modules
` jq '.resources[].module == "module.rds"' dev.tfstate`
but it actually just produced a list of boolean values. I don't see any mention of builtin functions like filter in jq's manual
then I tried to just access the field:
> jq '.resources[].module[].attributes[].password?' dev.tfstate
then it throws the following error
jq: error (at dev.tfstate:1116): Cannot iterate over string ("module.rds")
So what is the best way to extract the value? Hopefully it can only focus on the password attribute in module.rds module only.
Edit:
My purpose is to detect if a password is left inside a state file. I want to ensure the passwords are exclusively stored in AWS secret manager.
You can extract the module you want like this.
jq '.resources[] | select(.module == "module.rds")'
I'm not confident that I understand the requirements for the rest of the solution. So this might not only not be the best way of doing what you want; it might not do what you want at all!
If you know where password will be, you can do this.
jq '.resources[] | select(.module == "module.rds") | .instances[].attributes.password'
If you don't know exactly where password will be, this is a way of finding it.
jq '.resources[] | select(.module == "module.rds") | .. | .password? | values'
According to the manual under the heading "Recursive Descent," ..|.a? will "find all the values of object keys “a” in any object found “below” ."
values filters out the null results.
You could also get the password value out of the state file without jq by using Terraform outputs. Your module should define an output with the value you want to output and you should also output this at the root module.
Without seeing your Terraform code you'd want something like this:
modules/rds/main.tf
resource "aws_db_instance" "master" {
# ...
}
output "password" {
value = aws_db_instance.master.password
sensitive = true
}
example/main.tf
module "rds" {
source = "../modules/rds"
# ...
}
output "rds_password" {
value = module.rds.password
sensitive = true
}
The sensitive = true parameter means that Terraform won't print the output to stdout when running terraform apply but it's still held in plain text in the state file.
To then access this value without jq you can use the terraform output command which will retrieve the output from the state file and print it to stdout. From there you can use it however you want.