apigee - Json payload extract variables - json

I am confused why the extractVariables rule I am using is returning data as it is. See Below.
The json to Parse is:
{
"callNotificationSubscriptionList": {
"playAndCollectInteractionSubscription": [],
"recognitionInteractionSubscription": [],
"playAndRecordInteractionSubscription": [],
"callDirectionSubscription": [],
"callEventSubscription": [
{
"clientCorrelator": "112345",
"resourceURL": "http:someurl",
"callbackReference": {
"notifyURL": "someotherurlt",
"notificationFormat": "XML"
},
"filter": {
"data1": "data abc",
"data2": "data def",
"data3": "data xyz"
}
}
]
}
}
The rule:
<JSONPayload>
<Variable name="callNotSubL">
<!-- <JSONPath>$.callNotificationSubscriptionList</JSONPath> -->
<JSONPath>$.*</JSONPath>
</Variable>
</JSONpayload>
When I use the value that is commented out, I get no response variable data. If I set the "ignoreUnresolvedVariables" parm to "false", I am returned a failure, so it has no data. Thus, I tried "$.*" With this, I am returned:
[
{
"callbackReference": {
"notifyURL": "someotherurlt",
"notificationFormat": "XML"
},
"filter": {
"data1": "data abc",
"data2": "data def",
"data3": "data xyz"
}
}
]
could this be because the EntryNames are so long? I admit they are long, but they are well under the default values in the JSON Threat Potection Policy.
I did pump this json though a web based JSONPayload parser and $.callNotificationSubscriptionList worked fine as did $.callNotificationSubscriptionList.callEventSubscription[0] which is what I am really after. But, if I can't get the top level right, I can't get the sub-levels at all.

I solved this issue using #Santanu's comments:
It seems that the JSON to parse getting through the policy is not same as what you are expecting? Can you try to assign entire payload to a variable using the AssignVariable policy before the JSON path extraction policy, and check the value of that variable in the debug view? That would help understand what payload value is actually passing through when you are trying to apply the json path extraction policy.
the <Source> tag was "request" and this was a response extract. I removed the <Source> tag and all is well

Related

Access properties of an object via Dust.js after running JSON Parse filter

Is there any way to access properties of an object that was transformed into JSON through the jp (JSON parse) filter of Dust.js?
{
"response": {
"services": [
"{
\"prop1\":\"value1\",
\"prop2\":\"value2\",
\"prop3\":10
}"
]
}
}
For example, with the input above, I intend to receive the following output:
[
{
"prop1": "value1"
}
]
Note that the values inside the service array are strings, and because of that, before accessing the object's properties, I need to run JSON parse filter.
[
{#response.services}
{
"prop1": "{.|jp}"
}{#sep}, {/sep}
{/response.services}
]
What I've developed so far is the code above, and this code is returning the following output:
[
{
"prop1": "[object Object]"
}
]
In short, what I need to do is increment this {.|jp} into something where I can access the properties of the returned object, without adding new filters.
Thanks in advance to everyone who is willing to help!

Converting JSON Raw data into Powershell Format

I've got a thermal scanner that I'm trying to send triggers to. I'm able to achieve desired results with postman, but I need a standalone script to achieve this so I've been working in powershell. The manufacturer documentation merely specifies a POST as application/json and provide the raw format to be sent in the body as such:
{
"data": [
{
"output": "DO2",
"mode": "ON",
"duration": 5,
"delay": false
}
]
}
Like I said, I have no issues getting the desired result using postman, but I can't figure out how this needs to be formatted in powershell. Powershell isn't complaining about my code, but the device is not accepting the commands like it does from postman, probably because my code is garbage and the device isn't receiving the data correctly. The documentation says to input this data in the body so I have defined the body variable and simply call the variable when Invoke-Restmethod:
$body = ConvertTo-Json #(
'{"data":[
{
"output": "DO2"
"mode": "ON"
"duration": "5"
"delay": false
}
]
}'
)
any help would be appreciated and thanks in advance!
Use ConvertTo-Json on objects that need to be converted to json (JavaScript Object Notation), which basically is text describing objects in a structured way.
It looks like you already have the info for the body as valid json string, so in your case, use a Here-String for the json body:
$body = #'
{
"data": [
{
"output": "DO2",
"mode": "ON",
"duration": 5,
"delay": false
}
]
}
'#
and use that in your Invoke-Restmethod call

POSTMAN How to parse a nested json object in postman which has dynamic keys?

supposing the json body returned from a call contains some dynamic keys ie
{
"message": "search results matching criteria",
"permission": {
"261ef70e-0a95-4967-b078-81e657e32699": {
"device": {
"read:own": [
"*"
]
},
"account": {
"read:own": [
"*"
]
},
"user": {
"read:own": [
"*"
]
}
}
}
I can validate the json as follows easily enough although I am having a lot of trouble working out how to validate the objects BELOW the dynamic guid level of the response.
pm.test("response body to have correct items", function () {
pm.expect(jsonData.message).to.eq("search results matching criteria");
pm.expect(jsonData).to.have.property('permission');
pm.expect(jsonData.permission).to.have.property(pm.variables.get("otherUserId"));
});
Would ideally like to verify the device and account and user levels of the object.
Anyone with some tips?
I've tried a few ways to try and reference the otherUserId variable but nothing is working. It is either not resolving the variable therefore failing the test as its looking for a level in the json called otherUserId or it fails to run the test due to a syntax error.
This works:
pm.expect(jsonData.permission[pm.variables.get("otherUserId")]).to.have.property('device');

rename invalid keys from JSON

I have following flow in NIFI , JSON has (1000+) objects in it.
invokeHTTP->SPLIT JSON->putMongo
Flow works fine, till I receive some keys in json with "." in the name. e.g. "spark.databricks.acl.dfAclsEnabled".
my current solution is not optimal, I have jotted down bad keys, and using multiple replace text processor to replace "." with "_". I am not using REGEX, I am using string literal find/replace. So each time I am getting failure in putMongo processor, I am inserting new replaceText processor.
This is not maintainable. I am wondering if I can use JOLT for this? couple of info regarding input JSON.
1) no set structure, only thing that is confirmed is. everything will be in events array. But event object itself is free form.
2) maximum list size = 1000.
3) 3rd party JSON, so I cant ask for change in format.
Also, key with ".", can appear anywhere. So I am looking for JOLT spec that can cleanse at all level and then rename it.
{
"events": [
{
"cluster_id": "0717-035521-puny598",
"timestamp": 1531896847915,
"type": "EDITED",
"details": {
"previous_attributes": {
"cluster_name": "Kylo",
"spark_version": "4.1.x-scala2.11",
"spark_conf": {
"spark.databricks.acl.dfAclsEnabled": "true",
"spark.databricks.repl.allowedLanguages": "python,sql"
},
"node_type_id": "Standard_DS3_v2",
"driver_node_type_id": "Standard_DS3_v2",
"autotermination_minutes": 10,
"enable_elastic_disk": true,
"cluster_source": "UI"
},
"attributes": {
"cluster_name": "Kylo",
"spark_version": "4.1.x-scala2.11",
"node_type_id": "Standard_DS3_v2",
"driver_node_type_id": "Standard_DS3_v2",
"autotermination_minutes": 10,
"enable_elastic_disk": true,
"cluster_source": "UI"
},
"previous_cluster_size": {
"autoscale": {
"min_workers": 1,
"max_workers": 8
}
},
"cluster_size": {
"autoscale": {
"min_workers": 1,
"max_workers": 8
}
},
"user": ""
}
},
{
"cluster_id": "0717-035521-puny598",
"timestamp": 1535540053785,
"type": "TERMINATING",
"details": {
"reason": {
"code": "INACTIVITY",
"parameters": {
"inactivity_duration_min": "15"
}
}
}
},
{
"cluster_id": "0717-035521-puny598",
"timestamp": 1535537117300,
"type": "EXPANDED_DISK",
"details": {
"previous_disk_size": 29454626816,
"disk_size": 136828809216,
"free_space": 17151311872,
"instance_id": "6cea5c332af94d7f85aff23e5d8cea37"
}
}
]
}
I created a template using ReplaceText and RouteOnContent to perform this task. The loop is required because the regex only replaces the first . in the JSON key on each pass. You might be able to refine this to perform all substitutions in a single pass, but after fuzzing the regex with the look-ahead and look-behind groups for a few minutes, re-routing was faster. I verified this works with the JSON you provided, and also JSON with the keys and values on different lines (: on either):
...
"spark_conf": {
"spark.databricks.acl.dfAclsEnabled":
"true",
"spark.databricks.repl.allowedLanguages"
: "python,sql"
},
...
You could also use an ExecuteScript processor with Groovy to ingest the JSON, quickly filter all JSON keys that contain ., perform a collect operation to do the replacement, and re-insert the keys in the JSON data if you want a single processor to do this in a single pass.

How to specify JSON-formatted string in Cloudformation?

I have the following resource on my CloudFormation template to create a rule to run a Lambda function, from the AWS documentation:
"ScheduledRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"Description": "ScheduledRule",
"ScheduleExpression": "rate(5 minutes)",
"State": "ENABLED",
"Targets": [{
"Arn": { "Fn::GetAtt": ["myLambda", "Arn"] },
"Id": "TargetFunctionV1"
}]
}
}
I would like to specify the Input:
{
"Arn" : String,
"Id" : String,
"Input" : String,
"InputPath" : String
}
and Input is a JSON-formatted text string that is passed to the target. This value overrides the matched event.
I would like my JSON formatted text to be:
{
"mykey1": "Some Value"
}
I do not know how to specify it in the Input, when I put:
"ScheduledRule": {
"Type": "AWS::Events::Rule",
"Properties": {
"Description": "ScheduledRule",
"ScheduleExpression": "rate(5 minutes)",
"State": "ENABLED",
"Targets": [{
"Arn": { "Fn::GetAtt": ["myLambda", "Arn"] },
"Id": "TargetFunctionV1",
"Input": { "mykey1": "Some Value" }
}]
}
}
I will get error:
Value of property Input must be of type String
How should I specify it correctly?
I would use YAML as it is easier and more readable:
Input:
!Sub |
{
mykey1: "${myKey}"
}
Found out the answer myself:
"Input": "{ \"test\" : \"value11\", \"test2\" : \"value22\"}"
Hope it helps someone else.
Update:
You basically use the result of JSON.Stringify() to get the string into "Input" field. Use online JSON.Stringify() like https://onlinetexttools.com/json-stringify-text
I wanted to expand on #Pau's answer. Basically if you use the | to say that the whole block below is to be treated as a raw string.
If you need to replace any variable in the JSON then you can use Sub, but if you don't have any variables, then you don't need Sub. An example would be:
Input:|
{
"jsonVar":"jsonVal",
"jsonVar2" : "jsonVal2"
}
You can later do JSON.parse(<input-variable>) to get the JSON object.
NOTE: Don't put commas at the end of the variables in JSON, if there is no next variable. Example :
Input:|
{
"jsonVar":"jsonVal",
"jsonVar2" : "jsonVal2",
}
This will cause JSON parsing errors.
If you are writing your CloudFormation scripts in yaml and finding it difficult to use a JSON string (such as a policy doc) the easiest way is to convert your JSON into yaml using an online converter
ApiGatewayRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
Description: API Gateway for some API
EndpointConfiguration:
Types:
- PRIVATE
Name: MyAPIGateway
Policy: <Policy Doc >
Lets say the policy doc is as follows.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "arn:aws:execute-api:ap-southeast-2:something/*",
"Condition": {
"ForAnyValue:StringEquals": {
"aws:sourceVpce": "vpce-abcd"
}
}
}
]
}
Using the converter you can convert the JSON into an identical yaml and use as follows.
ApiGatewayRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
Description: API Gateway for some API
EndpointConfiguration:
Types:
- PRIVATE
Name: MyAPIGateway
Policy:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal: "*"
Action: execute-api:Invoke
Resource: arn:aws:execute-api:ap-southeast-2:something/*
Condition:
ForAnyValue:StringEquals:
aws:sourceVpce: vpce-abcd
Here's my similar YAML code for the "input" line. It took me all day to figure out the syntax so I'm hoping this might help someone in the future.
Input: "{\"action\":[\"configure\"],\"mode\":[\"ec2\"],\"optionalConfigurationSource\":[\"ssm\"],\"optionalConfigurationLocation\":[\"AmazonCloudWatch-Baseline-Windows\"],\"optionalRestart\":[\"yes\"]}"