How to add a field to an existing JSON in Velocity? - json

I have a JSON coming from a request body and I'm trying to use VTL to map it by adding an additional field to it. The request body looks like this:
{
"name": "John",
"age": 20,
"address": {
"street": "Albert's Street",
"City": "Test City"
}
}
In AWS, I'm trying to map this request body to have an additional field which is coming from a parameter in the URL path to become this:
{
"name": "John",
"age": 20,
"address": {
"street": "Albert's Street",
"City": "Test City"
},
"operation": "$input.params('path.operation')"
}
I have tried looping through with something like this but it doesn't work very well:
#set($allParams = $input.path('$'))
{
#foreach($type in $allParams.keySet())
#set($params = $allParams.get($type))
"$type" : {
#foreach($paramName in $params.keySet())
"$paramName" : "$util.escapeJavaScript($params.get($paramName))"
#if($foreach.hasNext),#end
#end
}
#if($foreach.hasNext),#end
#end
}
Moreover, this only works for those with 2 levels in the JSON. It doesn't work for those at the first level of the JSON or if I happen to have more than 2 levels in the JSON payload.
All I need is simply appending one more field into the existing request body of a JSON payload. How can I do that in Velocity?

You can add a operation property to the input JSON like this:
#set ($input.path('$').operation = 'example')
$input.json('$')
The above results in the following for your example:
{
"name": "John",
"age": 20,
"address": {
"street": "Albert's Street",
"City": "Test City"
},
"operation": "example"
}
Of course, you can use a value from params instead of 'example'.
By the way, consider running the param through $util.escapeJavaScript for added security.

Related

ADF Data Flow Remove empty JSON objects and arrays when building JSON files

I am using a Data Flow in Azure Data Factory to transform rows of CSV files into JSON documents of a standard layout.
I can't figure out how to get rid of empty JSON objects and arrays when there is no data to populate.
For example if I have a CSV like below:
firstName,lastName,Email,Address1,City,State,Zip
Bob,Smith,someemail#email.com,123 st,Somecity,TX,12345
I need to turn it into this:
{
"firstName": "Bob",
"lastName": "Smith",
"contactData": [
{
"contactType": "postalAddress",
"contactData": {
"postalAddress1": "123 st",
"postalCity": "Somecity",
"postalState": "TX",
"postalCode": "12345"
}
},
{
"contactType": "email",
"contactData": {
"emailAddress": "someemail#email.com"
}
}
]
}
I am using derived columns to build the subcolumns and arrays. I have been able to produce the JSON above.
The problem I run into is that if an email or address is null, I want to remove the object from the array.
If both are null, I want to remove the entire contactData object.
Example:
firstName,lastName,Email,Address1,City,State,Zip
Bob,Smith,,,,,
I need to turn it into this:
{
"firstName": "Bob",
"lastName": "Smith"
}
If I set all of the child objects to NULL with IF statements I can produce something like this:
{
"firstName": "Bob",
"lastName": "Smith",
"contactData": [
{
"contactData": {}
},
{
"contactData": {}
}
]
}
but I can't get rid of the entire section.
The Sink will get rid of the empty string objects, but not the nested JSON objects and arrays.
Is there any way to do this in ADF Data Flows?
You can split the data and then apply union to get the desired result.
Since I don't have the previous data transformations, I have taken the following data as my source data.
[{
"firstName": "Bob",
"lastName": "Smith",
"contactData": [
{
"contactType": "postalAddress",
"contactData": {
"postalAddress1": "123 st",
"postalCity": "Somecity",
"postalState": "TX",
"postalCode": "12345"
}
},
{
"contactType": "email",
"contactData": {
"emailAddress": "someemail#email.com"
}
}
]
},
{
"firstName": "b1",
"lastName": "s1",
"contactData": [
{
"contactData": {}
},
{
"contactData": {}
}
]
},
{
"firstName": "Bob1",
"lastName": "Smith1",
"contactData": [
{
"contactType": "postalAddress",
"contactData": {
"postalAddress1": "123 st1",
"postalCity": "Somecity1",
"postalState": "TX1",
"postalCode": "123456"
}
},
{
"contactType": "email",
"contactData": {
"emailAddress": "someemail1#email.com"
}
}
]
},
{
"firstName": "b2",
"lastName": "s2",
"contactData": [
{
"contactData": {}
},
{
"contactData": {}
}
]
}]
Now, I have taken a sample derived column to find the length of contactType. The rows without any contactType would have the same length (convert to string and find length).
tp : length(toString(contactData.contactType))
So, split the data based on whether the contactType (converted to string) length is equal to 2 or not. The split condition would be as shown below:
length(toString(contactData.contactType))!=2
This will split the data as required. The noContact stream would have the following data:
Now select only required columns. I have used rule based selection to select only columns where name is not contactData.
condition : name!='contactData'
column name : $$
Now apply union transformation by Name on hasContact and select1 stream.
I have configured the sink dataset as shown in the image below:
In sink, output to a single JSON file with selected name (under settings tab). The data preview in sink would be as:
Once the file is written, the null fields would not be written, so the data would be as per requirement. The following is an image for reference.

How to avoid extra nesting in json string

I tried to create json string according to syntax from wikipedia. I created json string with the following code:
var data = [];
data.push(
{
"firstName": "John",
"lastName": "Smith",
"isAlive": true,
"age": 27,
});
var addressdata = [];
addressdata.push(
{
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
});
data.push(
{
"address" : addressdata
}
);
The string is correct json string. However, the json structure contains some unnecessary nesting, as shown in the Figures 1 and 2 below. More precisely, there are surplus braces for address block, and the string is also enclosured with brackets instead of braces. So, what am I doing wrong? How can I avoid this unnecessary nesting and get structure as shown in Fig. 3?
Fig. 1
Fig. 2
Fig. 3
The string is generated with jsonData : data, in Ajax request.
You are using a list for the data variable. That's why you get brackets in the beginning and end of the JSON body. To overcome this problem you can declare the whole JSON body in the data variable like:
data = {
"firstname": "test",
(...)
"address": [{
"streetAddress": "test"
(...)
}]
}

Json parsing and mapping keys

I'm trying to map json to send it to another applications which expects the data in it's own formats, I'm using the AWS Lambda which when an event is triggered GETs below json which needs to be parsed and mapped according to what application expects. but the key stack is so large eg "rateCode" in "ratePlan" in "Details", there are almost 20000 rate codes like "abc", "xyz",... it is not a great idea to map like
if "rateCode" == "abc":
application_two_dict["rate_code"] = 123
so there are many more keys which keys has large set of values. what is the best way to map those keys. Also this needs to be happened in two way like when we get data from application two we need the parse the json and map the keys other way around which application one understands and vice versa.
{
"customer": {
"firstName": "john",
"lastName": "doe",
"email": "john.doe#test.com",
"mailingAddress": {
"address1": "123 N 1st st",
"address2": "789",
"countryCode": "USA",
"stateCode": "AZ",
"city": "Phoenix",
"postalCode": "34567"
},
"telephoneNumber": {
"telephoneNumber": "1235456789"
}
},
"paymentAccount": {
"firstName": "john",
"lastName": "doe",
"paymentAccountType": "VA",
"expirationDate": "2021-05-31",
"billingAddress": {
"address1": "1234 N 1st st",
"address2": "435",
"city": "Phoenix",
"countryCode": "USA",
"postalCode": "213445",
"stateCode": "AZ"
}
},
"Details": {
"123": [{
"quantity": 1,
"ratePlan": {
"rateCode": "abc",
"DetailsList": [{
"CategoryCode": "1234",
}]
}
}
}
I still don't have the exact format of app2 json
example json
for example
app1 json
{
"Details": {
"123": [{
"quantity": 1,
"ratePlan": {
"rateCode": "abc",
"DetailsList": [{
"CategoryCode": "1234",
}]
}
}
}
}
app 2 json
{
user_details_code : 123,
quantity : [1],
rate_plan : {
rate_code: "xyz",
category_code : "US_SAN"
}
}
I would try the following ways:
- use two static map with rateCode as keys
{ "abc": "123", ...} and { "123": "abc", ...} and use them to get values from the other app rateCode value.
use a database to fetch rateCode for app2 based on app1 value. Dynamo has a very low latency and can be very effective.
Maybe you could describe more precisely the json structure of the two apps.

parsing nested json data - access directly to a member

I have json data like
data = {
"id":1,
"name":"abc",
"address": {
"items":[
"streetName":"cde",
"streetId":"SID"
]
}
}
How can i access directly to the streetName Value ?
Your json is actually invalid. If you have control over the json generation, first change it to this:
data = {
"id": 1,
"name": "abc",
"address": {
"items": [{
"streetName": "cde",
"streetId": "SID"
}]
}
}
Notice the additional braces around streetName and streetId. Then, to access streetName, do this:
var streetName = data.address.items[0].streetName;

Appending a key value pair to a json object

This is the json object I am working with
{
"name": "John Smith",
"age": 32,
"employed": true,
"address": {
"street": "701 First Ave.",
"city": "Sunnyvale, CA 95125",
"country": "United States"
},
"children": [
{
"name": "Richard",
"age": 7
},
{
"name": "Susan",
"age": 4
},
{
"name": "James",
"age": 3
}
]
}
I want this as another key-value pair :
"collegeId": {
"eventno": "6062",
"eventdesc": "abc"
};
I tried concat but that gave me the result with || symbol and I cdnt iterate. I used spilt but that removes only commas.
concattedjson = JSON.stringify(JSON.parse(json1).concat(JSON.parse(json2)));
How do I add a key pair value to an existing json object ?
I am working in javascript.
This is the easiest way and it's working to me.
var testJson = {
"name": "John Smith",
"age": 32,
"employed": true,
"address": {
"street": "701 First Ave.",
"city": "Sunnyvale, CA 95125",
"country": "United States"
},
"children": [
{
"name": "Richard",
"age": 7
},
{
"name": "Susan",
"age": 4
},
{
"name": "James",
"age": 3
}
]
};
testJson.collegeId = {"eventno": "6062","eventdesc": "abc"};
Just convert the JSON string to an object using JSON.parse() and then add the property. If you need it back into a string, do JSON.stringify().
BTW, there's no such thing as a JSON object. There are objects, and there are JSON strings that represent those objects.
You need to make an object at reference "collegeId", and then for that object, make two more key value pairs there like this:
var concattedjson = JSON.parse(json1);
concattedjson["collegeId"] = {};
concattedjson["collegeId"]["eventno"] = "6062";
concattedjson["collegeId"]["eventdesc"] = "abc";
Assuming that concattedjson is your json object. If you only have a string representation you will need to parse it first before you extend it.
Edit
demo for those who think this will not work.
const newTestJson = JSON.parse(JSON.stringify(testJson));
newTestJson.collegeId = {"eventno": "6062","eventdesc": "abc"};
testJson = newTestJson;