I'm new to node-red and want to parse content received from wikipedia api. I send requests to the query endpoint:
https://en.wikipedia.org/w/api.php?action=query&titles={{{query}}}&prop=revisions&rvprop=parsetree&format=json&rvsection=0
The response looks similar to this:
{
...,
"query": {
"normalized": [ ... ],
"pages": {
"123456789": {
"pageid": 123456789,
"ns": 0,
"title": "title",
"revisions": [{
"parsetree": "...."
}]
}
}
}
}
I need to parse the content of parsetree, but am unable to get the first json object of pages dynamically.
Of course I can do something like: msg.payload.query.pages.123456789.revisions[0].parsetree
But I have a lot of titles i like to query and to process.
Is there an other way to get the content of parsetree?
You can always get hold of the list of keys in an object using the Object.keys(obj) method (doc)
So something like this should work
var pages = Object.keys(msg.payload.query.pages);
for (var i=0; i<pages.length; i++) {
var parsetree = msg.payload.query.pages[pages[i]].revisions[0].parsetree;
...
}
Related
I am very new to Postman and have a method in my API taking in 2 params: AccountNumber and GroupId to check if there's a customer Id associated with the AccountNumber. When I run the Post request, if it returns 2 or more customerIds, I'm supposed to return the response to the user to pick which customerId to use. How do I do that?
Right now, in the body of the request, I have:
{
"AccountNumber":"0001",
"GroupId": "1"
}
The output is an array of customerId. How do I test with only one customerId picked from the POST response?
Thank you for your help, I really appreciate it!
It's unlikely to be a Postman issue.
You are consuming the API through Postman. You are not able to control the output of the API itself from Postman.
What you can do, though, is write a test case for the current request. There is a 'Tests' tab, where the 'Params' tab is. At the right, there are snippets on how to create the test with response body. Let me give you a simple test case to get the first element:
Let's say I have the following response body
[
{
"name": "Blue",
"id": "5f1830d217afbb52acac9868"
},
{
"name": "Orange",
"id": "5f1e7afeb334ae70b4f05d1d"
},
{
"name": "Red",
"id": "5f1e7b10b334ae70b4f05d1e"
},
{
"name": "Green",
"id": "5f1e7b10b334ae70b4f05d1f"
},
{
"name": "Yellow",
"id": "5f1e7b10b334ae70b4f05d20"
}
]
The test case would look like this:
pm.test("Example Test", function () {
var jsonData = pm.response.json();
pm.expect(jsonData[0]).to.eql({
name: "Blue",
id: "5f1830d217afbb52acac9868"
});
});
I have a Datastream[ObjectNode] which I read as deserialized json from a kafka topic. One of the element of this ObjectNode is actually an array of events. This array has varying length. The incoming json stream looks like this :
{
"eventType": "Impression",
"deviceId": "359849094258487",
"payload": {
"vertical_name": "",
"promo_layout_type": "aa",
"Customer_Id": "1011851",
"ecommerce": {
"promoView": {
"promotions": [{
"name": "/-category_icons_all",
"id": "300275",
"position": "slot_5_1",
"creative": "Central/Gift Card/00000001B890D1739913DDA956AB5C79775991EC"
}, {
"name": "/-category_icons_all",
"id": "300276",
"position": "slot_6_1",
"creative": "Lifestyle/Gift Card/00000001B890D1739913DDA956AB5C79775991EC"
}, {
"name": "/-category_icons_all",
"id": "413002",
"position": "slot_7_1",
"creative": "Uber/Deals/00000001B890D1739913DDA956AB5C79775991EC"
}]
}
}
}
}
I want to be able to explode the promotions array so that each element inside becomes an individual message which can be written to a sink kafka topic. Does Flink provide the explode feature in DataStream and/or Table API?
I have tried to do a RichFlatMap on this stream to be able to collect individual rows but this also just returns me a DataStream[Seq[GenericRecord]] as below:
class PromoMapper(schema: Schema) extends RichFlatMapFunction[node.ObjectNode,Seq[GenericRecord]] {
override def flatMap(value: ObjectNode, out: Collector[Seq[GenericRecord]]): Unit = {
val promos = value.get("payload").get("ecommerce").get("promoView").get("promotions").asInstanceOf[Seq[node.ObjectNode]]
val record = for{promo <- promos} yield {
val processedRecord: GenericData.Record = new GenericData.Record(schema)
promo.fieldNames().asScala.foreach(f => processedRecord.put(f,promo.get(f)))
processedRecord
}
out.collect(record)
}
}
Please help.
Using a flatmap is the right idea (not sure why you bothered with a RichFlatMap, but that's a detail).
Seems like you should be calling out.collect(processedRecord) for each element inside the for loop, rather than once on the Seq produced by that loop.
I'm trying to generate a JSON file with mustache with the following template:
{
"name": "{{customer_info.first_name}}",
"email": "{{contact_info.email}}",
"campaign": {
"campaignId": "{{contact_info.campaign.campaignId}}"
},
"tags": [
{{#contact_info.tags}}
{
"tagId": "{{tagId}}"
},
{{/contact_info.tags}}
]
}
As an output example I get:
{
"name": "Antonio",
"email": "myemail#gmail.com",
"campaign": {
"campaignId": "pfft"
},
"tags": [
{
"tagId": "6prrtAP"
},
{
"tagId": "64rrrE9"
},
]
}
Which unluckily is a BAD FORMATTED JSON, because there is a not wanted "," after the last element in the array.
Can any of you help me in solving this issue and remove the comma ?
Thanks a lot
Try using SelectTransform npm package. It has Mustache like syntax without all the side-effects that Mustache creates and the package size is also not as heavy as Handlebars.js
import ST from "stjs";
const data = {
name: 'Jakub',
friends: [
{
name: 'Michal'
}
]
};
const template = {
newName: '{{ name }}',
friends: {
'{{ #each friends }}': {
subName: '{{ name }}'
}
}
};
console.log(ST.select(data).transformWith(template).root());
// Result:
/**
* {
* "newName": "Jakub",
* "friends": [
* {
* "subName": "Michal"
* }
* ]
* }
*/
I would do this:
var md = {};
var tagsCount = 2;
var currTagIndex = 0;
md['show_comma'] = function(){
currTagIndex++;
return currTagIndex <= tagsCount;
}
Then in Mustache template:
{{#show_comma}}
,
{{/show_comma}}
I've been experiencing some similar problem and I found out that Handlebars is a lot similar to mustache and way more powerful.
You could check that out and try using this template to solve your problem, without adding anything to your current model.
{
"name": "{{customer_info.first_name}}",
"email": "{{contact_info.email}}",
"campaign": {
"campaignId": "{{contact_info.campaign.campaignId}}"
},
"tags": [
{{#each contact_info.tags}}
{
"tagId": "{{tagId}}"
}{{#unless #last}},{{/unless}}
{{/each}}
]
}
Don't generate JSON from textual templates. You'll constantly face problems like this. Superfluous commas, meta characters in strings (what if customer_info.first_name contains double quotes), failing to properly nest structures etc.
Generate your data as native structures in your programming language, and encode it as JSON using library provided by your programming language.
However, if you absolutely need, try to generate as much JSON data as possible (ideally, self-contained JSON fragment) outside template, and interpolate that inside template. For example:
let contact_info = {"tags": [ "6prrtAP", "64rrrE9" ]}
let tags = contact_info.tags.map((tag) => ({"tagId": tag})); // [{tagId: "6prrtAP"}, {tagId: "64rrrE9"}]
let tagsJSON = JSON.stringify(tags); // "[{\"tagId\":\"6prrtAP\"},{\"tagId\":\"64rrrE9\"}]"
Then, pass tagsJSON to your template:
{
"name": "{{customer_info.first_name}}",
"email": "{{contact_info.email}}",
"campaign": {
"campaignId": "{{contact_info.campaign.campaignId}}"
},
"tags": {{tagsJSON}}
}
That way, tagsJSON always contains valid JSON-encoded data, so it might be safely interpolated as a value in JSON dictionary/object. Even if tag list is empty, even if tag IDs suddenly start to contain characters that need escaping etc. All corner cases are already handled for you.
This looks like a good answer:
contact_info['tags'][ contact_info['tags'].length - 1 ].last = true;
and the template would be
{{#contact_info.tags}}
{
"tagId": "{{tagId}}"
} {{^last}}, {{/last}}
{{/contact_info.tags}}
Source: https://stackoverflow.com/a/7591866
I am quite new to programming and especially Microsoft.Graph
I am having problems handling the response to:
https://graph.microsoft.com/v1.0/me/drive/root/children
the response looks like this (just much longer):
{
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('xyz%40hotmail.com')/drive/root/children",
"value": [
{
"createdBy": {
"user": {
"displayName": "xyz",
"id": "cf58e4781082"
}
},
"createdDateTime": "2009-01-08T08:52:07.063Z",
"cTag": "adDpFREJDR4RTQMTgxMDgyITEyOC42MzYxODM0MTU0Mjc3MDAwMDA",
"eTag": "aRURCQ0Y1OEU0A4MiExMjguMA",
"id": "EDBCF58E471082!128",
"lastModifiedBy": {
"user": {
"displayName": "xyz",
"id": "edbcf58e48082"
}
}, ............. etc...
The response that I received is correct, in JSON format (I believe ><), but I cannot figure out how to parse it into an array containing the folders name.
Please help!
Have considered using the Microsoft Graph client library? It will deserialize the JSON. Your call will look like this:
// Return all children files and folders off the drive root.
var driveItems = await graphClient.Me.Drive
.Root
.Children
.Request()
.GetAsync();
foreach (var item in driveItems)
{
// Get your item information
}
Here's some samples to help you get started:
https://github.com/microsoftgraph?utf8=%E2%9C%93&q=csharp
You can use the JavaScriptSerializer to do this. Assuming
//json contains the JSON Response
var jsonOutput = new System.Web.Script.Serialization.JavaScriptSerializer();
jsonOutput.DeserializeObject(json);
This has been discussed earlier. See this thread: Easiest way to parse JSON response
Refer this link for JavaScriptSerializer: https://msdn.microsoft.com/en-us/library/system.web.script.serialization.javascriptserializer(v=vs.110).aspx
I pushed a json file ( as shown below ) to ES using the following code :
with open('test.json','rb') as payload:
headers = {'content-type': 'application/json'}
r = requests.post('http://localhost:9200/test_nest_json/1',data=payload, verify=False, headers=headers)
{
"data": [
{
"keyword": "abc",
"lists": [
{
"item_val": "some_val"
}
],
"another_key": "some_key"
},
{
"keyword": "xyz",
"lists": [
{
"item_val":"another_val"
}
],
"another_key": "pqr"
}
]
}
I tried updating the mappings and used the term query but still it results in displaying all the indices. I am not able to query only one keyword like "data.keyword" = "abc" using term query.
Looks like you are having a problem with nested object
https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html
The reason for this cross-object matching, as discussed in Arrays of
Inner Objects, is that our beautifully structured JSON document is
flattened into a simple key-value format in the index
So the effective document stored looks like this:
{
"data.keyword": [ abc, xyz ],
"data.another_key": [ some_key, pqr ],
}
Which means the query you posted will match any document, as long as at least one of the nested object contains the xyz keyword. I recommend reading the link above for clarification.
This is what worked for me :
es.indices.refresh(index="test-index")
with open('abc.json','rb') as payload:
json_data = json.load(payload);
leng = len(json_data["data"])
for i in range (leng):
doc = json.dumps(json_data["data"][i]);
res = es.index(index="sample-index", doc_type='pdf',id=str(uuid.uuid4()), body=doc)
I am parsing the json and extracting the array items one by one and push it to ElasticSearch.
{
"keyword": "abc",
"lists": [
{
"item_val": "some_val"
}
],
"another_key": "some_key"
},
Still looking for an optimised solution.