ADF pipeline get dynamic count of input parameters - json

I have a pipeline where I want to pass a JSON to from 3rd party application.
simple JSON Example, but the keys could be different for each call
{
"name": "Here is a name",
"guaid": "123456-123456-123456-111112",
"owner": "my.email#example.com",
"description": "here comes my description"
}
Passing the JSON is not the problem but I do not want to configure each parameter in the JSON as an pipeline parameter inside my ADF because it is not needed.
I do not modify the JSON inside my pipeline, I just have to surround it with some other parameters, so I need it as a whole.
I cloud of course define a parameter for each key in the JSON and concatenate it again but why do the effort if it is not needed.
I also can not modify how the JSON will be passed to my pipeline, so it is not possible to pass the whole JSON inside one parameter like this:
{
"inputParam": "{\"name\": \"Here is a name\",\"guaid\": \"123456-123456-123456-111112\",\"owner\": \"my.email#example.com\",\"description\":\"here comes my description\"}"
}
so is it possible to get all input without "knowing" it and use it in my activity?
My Pipeline is just simple, I only need a set variable and web activities, so I want to avoid a complex solution.
JSON Outbutt should look like this
{
"processingMode": "full",
"version": "1.0.0",
"content": [
{
"type": "Application",
"id": "akjhajf-ffsfsfs-sf-sf-sf",
"data": {
"name": "Here is a name",
"guaid": "123456-123456-123456-111112",
"owner": "my.email#example.com",
"description": "here comes my description"
}
}
]
}
I achieve this by adding this in a set variable like this with the not allowed method i mentioned above
{
"processingMode": "full",
"version": "1.0.0",
"content": [
{
"type": "Application",
"id": "#{guid()}",
"data": #{pipeline().parameters.inputParam}
}
]
}
I then just use this result JSON and call a external Webservice via Web activity

Related

Add data to a json file using Talend

I have the following JSON:
[
{
"date": "29/11/2021",
"Name": "jack",
},
{
"date": "30/11/2021",
"Name": "Adam",
},
"date": "27/11/2021",
"Name": "james",
}
]
Using Talend, I wanna add 2 lines to have something like:
[
{
"company": "AMA",
"service": "BI",
"date": "29/11/2021",
"Name": "jack",
},
{
"company": "AMA",
"service": "BI",
"date": "30/11/2021",
"Name": "Adam",
},
"company": "AMA",
"service": "BI",
"date": "27/11/2021",
"Name": "james",
}
]
Currently, I use 3 components (tJSONDocOpen, tFixedFlowInput, tJSONDocOutput) but I can't have the right configuration of components in order to get the job done !
If you are not comfortable with json .
Just do these steps :
In the metaData just create a FileJson like this then paste it in your job as a tFileInputJson
Your job design and mapping would be
In your tFileOutputJson don't forget to change in the name of the data block "Data" with ""
What you need to do there according to the Talend practices is read your JSON. Then extract each object of it, add your properties and finally rebuild your JSON in a file.
An efficient way to do this is using tMap componenent like this.
The first tFileInputJSON will have to specify what properties it has to read from the JSON by setting your 2 objects in the mapping field.
Then the tMap will simply add 2 columns to your main stream, here is an example with hard coded string values. Depending on you needs, this component will also offer you the possibility to assign dynamic data to your 2 new columns, it's a powerful tool for manipulating the structure of a data stream.
You will find more infos about this component in the official documentation : https://help.talend.com/r/en-US/7.3/tmap/tmap; especially the "tMap scenarios" part.
Note
Instead of using the tMap, if you are comfortable with Java, you can use a tjavaRow instead. Using this, you can setup your 2 new columns with whatever java code you want to put as long as you have defined the output schema of the component.
output_row.Name = input_row.Name;
output_row.date = input_row.date;
output_row.company = "AMA";
output_row.service = "BI";

Executing a specific http method 'depending' on the condition given in the JSON file

I was thinking about the possibility of executing a specific http method (POST or PUT) in POSTMAN without specifying it.
I mean; imagine if there was a field in a JSON file called: METHOD within 2 possible states: 'I' corresponding to INSERT OR POST and the another one: 'U' related to UPDATE or PUT
Something like this: (please, do note the field called "method"):
[
{
"sku": "95LB645R34ER",
"method": 'I'
"payload": {
"price": "147000",
"tax_percentage": "US-21",
"store_code": "B2BUSD",
"markup_top": "1.62",
"status": "1",
"group_prices": [
{
"group": "CLASS A",
"price": "700038.79",
"website": "B2BUSD"
}
]
}
},
{
"sku": "95TYS34344ER",
"method": 'U'
"payload": {
"price": "69978",
"tax_percentage": "US-21",
"store_code": "B2BUSD",
"markup_top": "9.99",
"status": "1",
"group_prices": [
{
"group": "CLASS B",
"price": "88888.79",
"website": "B2BUSD"
}
]
}
}
]
I would like to run that JSON using the Collection Runner but i have no idea how to do the trick. I mean, everytime i generate a collection i have to specify the HTTP METHOD otherwise it wont know what to do.
I want the program to adjust that by looking at the JSON file, if "method":'I' then, perform a POST or if "method":'U' execute a PUT method. Do you get me?
I've been reading the documentation but i did not find something like that or maybe i did not understand. I'm not an expert on POSTMAN :(
Can you help me?
EDIT:
Alright, i did this:
In the request UI, use the {{METHOD}} syntax where you would see the HTTP method. This is an editable field as it allows you to add custom HTTP methods.
In the file, use the METHOD key and any HTTP verb as the value. Ensure that it's part of each object in the datafile as you will need it for each iteration.

Sending a request without specifying the fields in the body

I'm trying to send several PUT request to a specific endpoint.
I am using a CSV file within the columns and values, the name of the different columns reference the different variables inside the body, do you get me?
This is the endpoint:
{{URL_API}}/products/{{sku}} --sku is the id of the product, i created that variable because i use it to pass the reference
This is the body that i use:
{
"price":"{{price}}",
"tax_percentage":"{{tax_percentage}}",
"store_code":"{{store_code}}",
"markup_top":"{{markup_top}}",
"status":"{{status}}",
"group_prices": [
{
"group":"{{class_a}}",
"price":"{{price_a}}",
"website":"{{website_a}}"
}
]
}
I don't want to use the body anymore.. sometimes some products have more tan 1 group of prices, i mean:
"group_prices": [
{
"group":"{{class_a}}",
"price":"{{price_a}}",
"website":"{{website_a}}"
},
{
"group":"{{class_b}}",
"price":"{{price_b}}",
"website":"{{website_b}}"
}
Is it possible to create something like this in the CSV file?
sku,requestBody
99RE345GT, {JSON Payload}
How should i declare the {JSON Payload}?
Can you help me?
EDIT:
This is the CSV file i used:
sku,price,tax_percentage,store_code,markup_top,status,class_a,price_a,website_a
95LB645R34ER,147000,US-21,B2BUSD,1.62,1,CLASS A,700038.79,B2BUSD
I want to pass the JSON within the CSV file, i mean
sku,requestBody
95LB645R34ER,{"price":"147000","tax_percentage":"US-21","store_code":"B2BUSD","markup_top":"1.62","status":"1","group_prices":
[{ "group":"CLASS A","price":"700038.79","website":"B2BUSD"}]}
Is it okay?Should i specify anything on the request body or not? I read the documentation posted in POSTMAN website but i did not find an example like this.
As you're using JSON data as a payload, I would use a JSON file rather than a CSV file in the Collection Runner. Use this as a template and save it as data.json, it's in the correct format accepted in the Runner.
[
{
"sku": "95LB645R34ER",
"payload": {
"price": "147000",
"tax_percentage": "US-21",
"store_code": "B2BUSD",
"markup_top": "1.62",
"status": "1",
"group_prices": [
{
"group": "CLASS A",
"price": "700038.79",
"website": "B2BUSD"
}
]
}
},
{
"sku": "MADEUPSKU",
"payload": {
"price": "99999",
"tax_percentage": "UK-99",
"store_code": "BLAH",
"markup_top": "9.99",
"status": "5",
"group_prices": [
{
"group": "CLASS B",
"price": "88888.79",
"website": "BLAH"
}
]
}
}
]
In the pre-request Script of the request, add this code. It's creating a new local variable from the data under the payload key in the data file. The data needs to be transformed into a string so it's using JSON.stringify() to do this.
pm.variables.set("JSONpayload", JSON.stringify(pm.iterationData.get('payload'), null, 2));
In the Request Body, add this:
{{JSONpayload}}
In the Collection Runner, select the Collection you would like to run and then select the data.json file that you previously created. Run the Collection.

How to remove one property from json input message using replace() expression in azure logic app?

{
"metadata": {
"id": "2",
"uri": "3",
"type": "2"
},
"Number": "2323600002913",
"Date": "04/21/2009",
"postingDate": "00/00/0000",
"ata": {
"results": [
{
"metadata": {
"id": "r",
"uri": "e2",
"type": "s2"
},
"item": "000010",
"data":"ad"
}
]
}
}
want to remove metadata property from above json message and output should be like below
{
"Number": "2323600002913",
"Date": "04/21/2009",
"postingDate": "00/00/0000",
"ata": {
"results": [
{
"item": "000010",
"data":"ad"
}
]
}
}
I tried with removeProperty() which is working for root level metadata but inside metadata not removed.
how to use replace() in this case or anything else to only remove metadata.
The simplest way is use inline code, cause even with removeProperty() expression to remove the metadata under results, it will return the results array data not the whole json data. Then you will have to combine them, it's not a convenient way.
And with inline code you could refer to my below picture. The variable json is the value from triggerbody, then just delete the node or key and return the json variable. And with this way, even you want to delete many metadata in the array, you could add a for loop to delete it, just think of it as plain js code.
Update:if you want to get value from variable,cause no support expression to get value from variable so use the below expression.
var json =wworkflowContext.actions.Initialize_variable.inputs.variables[0].value;
And about how to loop the array in the json refer to my below pic.

How to do deep sets and gets in Go's map[string]interface{}?

If I have some arbitrary JSON how can I do deep sets and gets on the nested properties using a slice of map keys and/or slice indexes?
For example, in the following excerpt from the JSON API example:
{
"data": [{
"type": "posts",
"id": "1",
"title": "JSON API paints my bikeshed!",
"links": {
"self": "http://example.com/posts/1",
"author": {
"self": "http://example.com/posts/1/links/author",
"related": "http://example.com/posts/1/author",
"linkage": { "type": "people", "id": "9" }
}
}
}]
}
I'd like to get the string "9" located at data.0.links.author.linkage.id using something like:
[]interface{}{"data",0,"links","author","linkage","id"}
I know the ideal way to do this is to create nested structs that map to the JSON object which I do for production code, but sometimes I need to do some quick testing which would be nice to do in Go as well.
You have stretchr/objx that provide a similar approach.
Example use:
document, _ := objx.FromJSON(json)
document.Get("path.to.field[0].you.want").Str()
However, unless you really don't know at all the structure of your JSON input ahead of time, this isn't the preferred way to go in golang…