Json parsing and mapping keys - json

I'm trying to map json to send it to another applications which expects the data in it's own formats, I'm using the AWS Lambda which when an event is triggered GETs below json which needs to be parsed and mapped according to what application expects. but the key stack is so large eg "rateCode" in "ratePlan" in "Details", there are almost 20000 rate codes like "abc", "xyz",... it is not a great idea to map like
if "rateCode" == "abc":
application_two_dict["rate_code"] = 123
so there are many more keys which keys has large set of values. what is the best way to map those keys. Also this needs to be happened in two way like when we get data from application two we need the parse the json and map the keys other way around which application one understands and vice versa.
{
"customer": {
"firstName": "john",
"lastName": "doe",
"email": "john.doe#test.com",
"mailingAddress": {
"address1": "123 N 1st st",
"address2": "789",
"countryCode": "USA",
"stateCode": "AZ",
"city": "Phoenix",
"postalCode": "34567"
},
"telephoneNumber": {
"telephoneNumber": "1235456789"
}
},
"paymentAccount": {
"firstName": "john",
"lastName": "doe",
"paymentAccountType": "VA",
"expirationDate": "2021-05-31",
"billingAddress": {
"address1": "1234 N 1st st",
"address2": "435",
"city": "Phoenix",
"countryCode": "USA",
"postalCode": "213445",
"stateCode": "AZ"
}
},
"Details": {
"123": [{
"quantity": 1,
"ratePlan": {
"rateCode": "abc",
"DetailsList": [{
"CategoryCode": "1234",
}]
}
}
}
I still don't have the exact format of app2 json
example json
for example
app1 json
{
"Details": {
"123": [{
"quantity": 1,
"ratePlan": {
"rateCode": "abc",
"DetailsList": [{
"CategoryCode": "1234",
}]
}
}
}
}
app 2 json
{
user_details_code : 123,
quantity : [1],
rate_plan : {
rate_code: "xyz",
category_code : "US_SAN"
}
}

I would try the following ways:
- use two static map with rateCode as keys
{ "abc": "123", ...} and { "123": "abc", ...} and use them to get values from the other app rateCode value.
use a database to fetch rateCode for app2 based on app1 value. Dynamo has a very low latency and can be very effective.
Maybe you could describe more precisely the json structure of the two apps.

Related

ADF Data Flow Remove empty JSON objects and arrays when building JSON files

I am using a Data Flow in Azure Data Factory to transform rows of CSV files into JSON documents of a standard layout.
I can't figure out how to get rid of empty JSON objects and arrays when there is no data to populate.
For example if I have a CSV like below:
firstName,lastName,Email,Address1,City,State,Zip
Bob,Smith,someemail#email.com,123 st,Somecity,TX,12345
I need to turn it into this:
{
"firstName": "Bob",
"lastName": "Smith",
"contactData": [
{
"contactType": "postalAddress",
"contactData": {
"postalAddress1": "123 st",
"postalCity": "Somecity",
"postalState": "TX",
"postalCode": "12345"
}
},
{
"contactType": "email",
"contactData": {
"emailAddress": "someemail#email.com"
}
}
]
}
I am using derived columns to build the subcolumns and arrays. I have been able to produce the JSON above.
The problem I run into is that if an email or address is null, I want to remove the object from the array.
If both are null, I want to remove the entire contactData object.
Example:
firstName,lastName,Email,Address1,City,State,Zip
Bob,Smith,,,,,
I need to turn it into this:
{
"firstName": "Bob",
"lastName": "Smith"
}
If I set all of the child objects to NULL with IF statements I can produce something like this:
{
"firstName": "Bob",
"lastName": "Smith",
"contactData": [
{
"contactData": {}
},
{
"contactData": {}
}
]
}
but I can't get rid of the entire section.
The Sink will get rid of the empty string objects, but not the nested JSON objects and arrays.
Is there any way to do this in ADF Data Flows?
You can split the data and then apply union to get the desired result.
Since I don't have the previous data transformations, I have taken the following data as my source data.
[{
"firstName": "Bob",
"lastName": "Smith",
"contactData": [
{
"contactType": "postalAddress",
"contactData": {
"postalAddress1": "123 st",
"postalCity": "Somecity",
"postalState": "TX",
"postalCode": "12345"
}
},
{
"contactType": "email",
"contactData": {
"emailAddress": "someemail#email.com"
}
}
]
},
{
"firstName": "b1",
"lastName": "s1",
"contactData": [
{
"contactData": {}
},
{
"contactData": {}
}
]
},
{
"firstName": "Bob1",
"lastName": "Smith1",
"contactData": [
{
"contactType": "postalAddress",
"contactData": {
"postalAddress1": "123 st1",
"postalCity": "Somecity1",
"postalState": "TX1",
"postalCode": "123456"
}
},
{
"contactType": "email",
"contactData": {
"emailAddress": "someemail1#email.com"
}
}
]
},
{
"firstName": "b2",
"lastName": "s2",
"contactData": [
{
"contactData": {}
},
{
"contactData": {}
}
]
}]
Now, I have taken a sample derived column to find the length of contactType. The rows without any contactType would have the same length (convert to string and find length).
tp : length(toString(contactData.contactType))
So, split the data based on whether the contactType (converted to string) length is equal to 2 or not. The split condition would be as shown below:
length(toString(contactData.contactType))!=2
This will split the data as required. The noContact stream would have the following data:
Now select only required columns. I have used rule based selection to select only columns where name is not contactData.
condition : name!='contactData'
column name : $$
Now apply union transformation by Name on hasContact and select1 stream.
I have configured the sink dataset as shown in the image below:
In sink, output to a single JSON file with selected name (under settings tab). The data preview in sink would be as:
Once the file is written, the null fields would not be written, so the data would be as per requirement. The following is an image for reference.

How to add a field to an existing JSON in Velocity?

I have a JSON coming from a request body and I'm trying to use VTL to map it by adding an additional field to it. The request body looks like this:
{
"name": "John",
"age": 20,
"address": {
"street": "Albert's Street",
"City": "Test City"
}
}
In AWS, I'm trying to map this request body to have an additional field which is coming from a parameter in the URL path to become this:
{
"name": "John",
"age": 20,
"address": {
"street": "Albert's Street",
"City": "Test City"
},
"operation": "$input.params('path.operation')"
}
I have tried looping through with something like this but it doesn't work very well:
#set($allParams = $input.path('$'))
{
#foreach($type in $allParams.keySet())
#set($params = $allParams.get($type))
"$type" : {
#foreach($paramName in $params.keySet())
"$paramName" : "$util.escapeJavaScript($params.get($paramName))"
#if($foreach.hasNext),#end
#end
}
#if($foreach.hasNext),#end
#end
}
Moreover, this only works for those with 2 levels in the JSON. It doesn't work for those at the first level of the JSON or if I happen to have more than 2 levels in the JSON payload.
All I need is simply appending one more field into the existing request body of a JSON payload. How can I do that in Velocity?
You can add a operation property to the input JSON like this:
#set ($input.path('$').operation = 'example')
$input.json('$')
The above results in the following for your example:
{
"name": "John",
"age": 20,
"address": {
"street": "Albert's Street",
"City": "Test City"
},
"operation": "example"
}
Of course, you can use a value from params instead of 'example'.
By the way, consider running the param through $util.escapeJavaScript for added security.

How do I pass dropdown values via JSON record response to react UI?

I need to get feedback on the preferred method of architecting a JSON response to not only provide data elements but also control the UI elements as well (specifically select boxes)
I am currently designing a suite of REST services to support a new application. All data will be served from the REST services and the frontend will be Spring-Boot/REACT.js.
We want all business logic to be managed at the data layer (Spring-Boot MicroServices). So all of the "select options" (dropdowns) will be driven by data provided by the Java Service via the RESTful services.
I am just looking for feedback on either of these options (or if you have a more preferred option, please feel free.
[
{
"id": 1,
"first_name": "Curry",
"last_name": "Aburrow",
"email": "caburrow0#gnu.org",
"gender": "Male",
"gender_options": {
"0":"Male",
"1":"Female"
}
"department": "Sales"
"department_options": {
"0":"Sales",
"1":"Legal",
"2":"HR",
"3":"Finance",
"4":"Operations"
}
},
{
"id": 2,
"first_name": "Joachim",
"last_name": "Lotte",
"email": "jlotte1#webnode.com",
"gender": "Male",
"gender_options": {
"0":"Male",
"1":"Female"
}
"department": "Finance"
"department_options": {
"0":"Sales",
"1":"Legal",
"2":"HR",
"3":"Finance",
"4":"Operations"
}
}
]
OR THIS
[
{
"id": 1,
"first_name": "Curry",
"last_name": "Aburrow",
"email": "caburrow0#gnu.org",
"gender": {
"selected":"Male",
"options": {
"0":"Male",
"1":"Female"
}
"department": {
"selected":"Sales",
"options": {
"0":"Sales",
"1":"Legal",
"2":"HR",
"3":"Finance",
"4":"Operations"
}
},
{
"id": 2,
"first_name": "Joachim",
"last_name": "Lotte",
"email": "jlotte1#webnode.com",
"gender": {
"selected":"Female",
"options": {
"0":"Male",
"1":"Female"
}
"department": {
"selected":"Finance",
"options": {
"0":"Sales",
"1":"Legal",
"2":"HR",
"3":"Finance",
"4":"Operations"
}
}
]
Option 1 is basic json providing the select option array in it's own element.
Option 2 is nested array.
Neither of which is overly complex, just looking for the more performant method.
I feel that department options and gender options should not be a part of the Object in the first place. Those should come from a separate json which can be either static or is configurable by some CMS. So you should just be fine with this:
[{
"id": 1,
"first_name": "Curry",
"last_name": "Aburrow",
"email": "caburrow0#gnu.org",
"gender": "Male",
"department": "Sales"
},
{
"id": 2,
"first_name": "Joachim",
"last_name": "Lotte",
"email": "jlotte1#webnode.com",
"gender": "Male",
"department": "Finance"
}
]
Also, your gender options and department options aren't really arrays. They're Objects. You can either make them string arrays or better, make them object arrays so that they could also have something like a display and value field that could then be used. For eg:
For gender options:
[{
display: "Male",
value: "MALE"
},
{
display: "Female",
value: "FEMALE"
}
]
For department options:
[{
id: 1,
display: "Sales",
value: "SALES"
},
{
id: 2,
display: "Legal",
value: "LEGAL"
},
{
id: 3,
display: "HR",
value: "HR"
},
{
id: 4,
display: "Finance",
value: "FINANCE"
},
{
id: 5,
display: "Operations",
value: "OPERATIONS"
}
]

Groovy - Parse JSON where only certain values exists in response

I am trying to parse a JSON response that has repeating objects with JsonSlurper to compare to a JDBC query. However, I only want to compare objects where a certain values exist within that object.
If I had a response that looks like this, how would I only parse the objects where the country equals USA or Canada, therefore ignoring anything else?
{
"info": [{
"name": "John Smith",
"phone": "2125557878",
"country": {
"value": "USA"
}
},
{
"name": "Jane Smith",
"phone": "2125551212",
"country": {
"value": "USA"
}
},
{
"name": "Bob Jones",
"phone": "4165558714",
"country": {
"value": "Canada"
}
},
{
"name": "George Tucker",
"phone": "4454547171",
"country": {
"value": "UK"
}
},
{
"name": "Jean Normand",
"phone": "4454547171",
"country": {
"value": "France"
}
}]
}
This is what I have in groovy:
def jsonResponse = context.expand('${RESTRequest#Response}')
def parsedJson = new JsonSlurper().parseText(jsonResponse)
def info = parsedJson.info
def jsonDataObjects = []
info.each { json ->
jsonDataObjects.add(Model.buildJSONData(json))
}
I am building a collection of the elements that I need to compare to a database. How do I only add to that collection where the info.country.value = USA or Canada?
I tried using .findAll like this just to test if I could get it to filter by just one of the countries:
def info = parsedJson.info.country.findAll{it.value == "USA"}
But, when I do that, only the value field is kept. I lose the name and phone from the parse.
Thanks in advance for any assistance.
Did you try
def info = parsedJson.info.findAll{it.country.value == "USA"}
?

Appending a key value pair to a json object

This is the json object I am working with
{
"name": "John Smith",
"age": 32,
"employed": true,
"address": {
"street": "701 First Ave.",
"city": "Sunnyvale, CA 95125",
"country": "United States"
},
"children": [
{
"name": "Richard",
"age": 7
},
{
"name": "Susan",
"age": 4
},
{
"name": "James",
"age": 3
}
]
}
I want this as another key-value pair :
"collegeId": {
"eventno": "6062",
"eventdesc": "abc"
};
I tried concat but that gave me the result with || symbol and I cdnt iterate. I used spilt but that removes only commas.
concattedjson = JSON.stringify(JSON.parse(json1).concat(JSON.parse(json2)));
How do I add a key pair value to an existing json object ?
I am working in javascript.
This is the easiest way and it's working to me.
var testJson = {
"name": "John Smith",
"age": 32,
"employed": true,
"address": {
"street": "701 First Ave.",
"city": "Sunnyvale, CA 95125",
"country": "United States"
},
"children": [
{
"name": "Richard",
"age": 7
},
{
"name": "Susan",
"age": 4
},
{
"name": "James",
"age": 3
}
]
};
testJson.collegeId = {"eventno": "6062","eventdesc": "abc"};
Just convert the JSON string to an object using JSON.parse() and then add the property. If you need it back into a string, do JSON.stringify().
BTW, there's no such thing as a JSON object. There are objects, and there are JSON strings that represent those objects.
You need to make an object at reference "collegeId", and then for that object, make two more key value pairs there like this:
var concattedjson = JSON.parse(json1);
concattedjson["collegeId"] = {};
concattedjson["collegeId"]["eventno"] = "6062";
concattedjson["collegeId"]["eventdesc"] = "abc";
Assuming that concattedjson is your json object. If you only have a string representation you will need to parse it first before you extend it.
Edit
demo for those who think this will not work.
const newTestJson = JSON.parse(JSON.stringify(testJson));
newTestJson.collegeId = {"eventno": "6062","eventdesc": "abc"};
testJson = newTestJson;