Background
Recently, I have been working with the Elasticsearch Node.js API to bulk-index a large JSON file. I have successfully parsed the JSON file. Now, I should be able to pass the index-ready array into the Elasticsearch bulk command. However, using console log, it appears as though the array shouldn't be causing any problems.
Bugged Code
The following code is supposed to take an API URL (with a JSON response) and parse it using the Node.js HTTP library. Then using the Elasticsearch Node.js API, it should bulk-index every entry in the JSON array into my Elasticsearch index.
var APIUrl = /* The url to the JSON file on the API providers server. */
var bulk = [];
/*
Used to ready JSON file for indexing
*/
var makebulk = function(ParsedJSONFile, callback) {
var JSONArray = path.to.array; /* The array was nested... */
var action = { index: { _index: 'my_index', _type: 'my_type' } };
for(const item of items) {
var doc = { "id": `${item.id}`, "name": `${item.name}` };
bulk.push(action, doc);
}
callback(bulk);
}
/*
Used to index the output of the makebulk function
*/
var indexall = function(madebulk, callback) {
client.bulk({
maxRetries: 5,
index: "my_index",
type: "my_type",
body: makebulk
}, function(err, resp) {
if (err) {
console.log(err);
} else {
callback(resp.items);
}
});
}
/*
Gets the specified URL, parses the JSON object,
extracts the needed data and indexes into the
specified Elasticsearch index
*/
http.get(APIUrl, function(res) {
var body = '';
res.on('data', function(chunk) {
body += chunk;
});
res.on('end', function() {
var APIURLResponse = JSON.parse(body);
makebulk(APIURLResponse, function(resp) {
console.log("Bulk content prepared");
indexall(resp, function(res) {
console.log(res);
});
console.log("Response: ", resp);
});
});
}).on('error', function(err) {
console.log("Got an error: ", err);
});
When I run node bulk_index.js on my web server, I receive the following error: TypeError: Bulk body should either be an Array of commands/string, or a String. However, this doesn't make any sense because the console.log(res) command (From the indexall function under http.get client request) outputs the following:
Bulk content prepared
Response: [ { index: { _index: 'my_index', _type: 'my_type', _id: '1' } },
{ id: '5', name: 'The Name' }, ... },
... 120690 more items ]
The above console output appears to show the array in the correct format.
Question
What does TypeError: Bulk body should either be an Array of commands/string, or a String indicate is wrong with the array I am passing into the client.bulk function?
Notes
My server is currently running Elasticsearch 6.2.4 and Java Development Kit version 10.0.1. Everything works as far as the Elaticsearch server and even my Elaticsearch mappings (I didn't provide the client.indices.putMapping code, however I can if it is needed). I have spent multiple hours reading over every scrap of documentation I could find regarding this TypeError. I couldn't find much in regards to the error being thrown, so I am not sure where else to look for information regarding this error.
Seems a typo in your code.
var indexall = function(**madebulk**, callback) {
client.bulk({
maxRetries: 5,
index: "my_index",
type: "my_type",
body: **makebulk**
Check the madebulk & makebulk.
Related
Basically, I am setting up a web server via Node.js and Express (I am a beginner at this) to retrieve data by reading a JSON file.
For example, this is my data.json file:
[{
"color": "black",
"category": "hue",
"type": "primary"
},
{
"color": "red",
"category": "hue",
"type": "primary"
}
]
I am trying to retrieve all of the colors by implementing this code for it to display on localhost:
router.get('/colors', function (req, res) {
fs.readFile(__dirname + '/data.json', 'utf8', function (err, data) {
data = JSON.parse(data);
res.json(data); //this displays all of the contents of data.json
})
});
router.get('/colors:name', function (req, res) {
fs.readFile(__dirname + '/data.json', 'utf8', function (err, data) {
data = JSON.parse(data);
for (var i = 0; i < data.length; i++) {
res.json(data[i][1]); //trying to display the values of color
}
})
});
How do I go about doing this?
What you are trying to do is actually pretty simple once you break it into smaller problems. Here is one way to break it down:
Load your JSON data into memory for use by your API.
Define an API route which extracts only the colours from your JSON data and sends them to the client as a JSON.
var data = [];
try {
data = JSON.parse(fs.readFileSync('/path/to/json'));
} catch (e) {
// Handle JSON parse error or file not exists error etc
data = [{
"color": "black",
"category": "hue",
"type": "primary"
},
{
"color": "red",
"category": "hue",
"type": "primary"
}
]
}
router.get('/colors', function (req, res, next) {
var colors = data.map(function (item) {
return item.color
}); // This will look look like: ["black","red"]
res.json(colors); // Send your array as a JSON array to the client calling this API
})
Some improvements in this method:
The file is read only once synchronously when the application is started and the data is cached in memory for future use.
Using Array.prototype.map Docs to extract an array of colors from the object.
Note:
You can structure the array of colors however you like and send it down as a JSON in that structure.
Examples:
var colors = data.map(function(item){return {color:item.color};}); // [{"color":"black"},{"color":"red"}]
var colors = {colors: data.map(function(item){return item.color;})} // { "colors" : ["black" ,"red"] }
Some gotchas in your code:
You are using res.json in a for loop which is incorrect as the response should only be sent once. Ideally, you would build the JS object in the structure you need by iterating over your data and send the completed object once with res.json (which I'm guessing internally JSON.stringifys the object and sends it as a response after setting the correct headers)
Reading files is an expensive operation. If you can afford to read it once and cache that data in memory, it would be efficient (Provided your data is not prohibitively large - in which case using files to store info might be inefficient to begin with)
in express, you can do in this way
router.get('/colors/:name', (req, res) => {
const key = req.params.name
const content = fs.readFileSync(__dirname + '/data.json', 'utf8')
const data = JSON.parse(content)
const values = data.reduce((values, value) => {
values.push(value[key])
return values
}, [])
// values => ['black', 'red']
res.send(values)
});
and then curl http://localhost/colors/color,
you can get ['black', 'red']
What you're looking to do is:
res.json(data[i]['color']);
If you don't really want to use the keys in the json you may want to use the Object.values function.
...
data = JSON.parse(data)
var values = []
for (var i = 0; i < data.length; i++) {
values.push(Object.values(data[i])[0]) // 0 - color, 1 - category, 2 - type
}
res.json(values) // ["black","red"]
...
You should never use fs.readFileSync in production. Any sync function will block the event loop until the execution is complete hence delaying everything afterwords (use with caution if deemed necessary). A few days back I had the worst experience myself and learnt that in a hard way.
In express you can define a route with param or query and use that to map the contents inside fs.readFile callback function.
/**
* get color by name
*
* #param {String} name name of the color
* #return {Array} array of the color data matching param
*/
router.get('/colors/:name', (req, res) => {
const color = req.params.name
const filename = __dirname + '/data.json';
fs.readFile('/etc/passwd', 'utf8', (err, data) => {
if(err){
return res.send([]); // handle any error returned by readFile function here
}
try{
data = JSON.parse(data); // parse the JSON string to array
let filtered = []; // initialise empty array
if(data.length > 0){ // we got an ARRAY of objects, right? make your check here for the array or else any map, filter, reduce, forEach function will break the app
filtered = data.filter((obj) => {
return obj.color === color; // return the object if the condition is true
});
}
return res.send(filtered); // send the response
}
catch(e){
return res.send([]); // handle any error returned from JSON.parse function here
}
});
});
To summarise, use fs.readFile asynchronous function so that the event loop is not clogged up. Inside the callback parse through the content and then return the response. return is really important or else you might end up getting Error: Can't set headers after they are sent
DISCLAIMER This code above is untested but should work. This is just to demonstrate the idea.
I think you can’t access JSON without key. You can use Foreach loop for(var name : object){} check about foreach it may help you
I am new to NodeJS and trying to figure out how to load some xml data from a REST service and convert it to JSON so I can afterwards load it into my view.
I'm trying to do so querying with Hapi API and loading it into an xml parser then convert it to JSON.
Doing the following seems loading correctly the xml object and when printing it it actually shows me some JSON. Does it mean that I don't need to convert to JSON anymore?
const server = new Hapi.Server();
...
server.route({
method: 'GET',
path: '/',
handler: function (request, reply) {
Request.get('http://ws.seloger.com/search.xml?&idtt=2&idtypebien=2,1&ci=750056&pxmax=400000&tri=initial&naturebien=1,2&surfacemin=65search.xml?',
function (error, response, body) {
if (error) {
throw error;
}
var parse = require('xml-parser');
var inspect = require('util').inspect;
var obj = parse(body);
console.log(inspect(obj, { colors: true, depth: 4 }));
Note that the JSON displayed is also not what I am looking for, ie. showing the details with attributes, children, etc.:
{ declaration: { attributes: { version: '1.0', encoding: 'UTF-8' } },
root:
{ name: 'recherche',
attributes: {},
children:
[ { name: 'resume',
attributes: {},
children: [],
content: '....' },
However looking for something more like this (might be just a different representation)
So I figured out that my issue seemed to have been related to the parser itself. Using this one I get what I need:
var parseString = require('xml2js').parseString;
parseString(body, function (err, jsonData) {
reply.view('index', { result: body });
});
I am creating a website that reads externally hosted json files and then uses node.js to populate the sites content.
Just to demonstrate what I'm after, this is a really simplified version of what I'm trying to do in node.js
var ids = [111, 222, 333];
ids.forEach(function(id){
var json = getJSONsomehow('http://www.website.com/'+id+'.json');
buildPageContent(json);
});
Is what I want to do possible?
(Marked as a duplicate of "How do I return the response from an asynchronous call?" see my comment below for my rebuttal)
You are trying to get it synchronously. What you should aim for instead, is not a function used like this:
var json = getJSONsomehow('http://www.website.com/'+id+'.json');
but more like this:
getJSONsomehow('http://www.website.com/'+id+'.json', function (err, json) {
if (err) {
// error
} else {
// your json can be used here
}
});
or like this:
getJSONsomehow('http://www.website.com/'+id+'.json')
.then(function (json) {
// you can use your json here
})
.catch(function (err) {
// error
});
You can use the request module to get your data with something like this:
var request = require('request');
var url = 'http://www.website.com/'+id+'.json';
request.get({url: url, json: true}, (err, res, data) => {
if (err) {
// handle error
} else if (res.statusCode === 200) {
// you can use data here - already parsed as json
} else {
// response other than 200 OK
}
});
For a working example see this answer.
For more info see: https://www.npmjs.com/package/request
I think problem is in async request. Function will return result before request finished.
AJAX_req.open( "GET", url, true );
Third parameter specified async request.
You should add handler and do all you want after request finished.
For example:
function AJAX_JSON_Req( url ) {
var AJAX_req = new XMLHttpRequest.XMLHttpRequest();
AJAX_req.open( "GET", url, true );
AJAX_req.setRequestHeader("Content-type", "application/json");
AJAX_req.onreadystatechange = function() {
if (AJAX_req.readyState == 4 && AJAX_req.status == 200) {
console.log(AJAX_req.responseText);
}
};
}
I'm using NodeJS and an npm package called oauth to communicate with Twitter's search API. For some reason however, twitter is returning to me an empty array of statuses without any error... What is even more confusing is the fact that using a tool like Postman with the exact same request and keys returns the list of tweets? It makes no sense! Here is my request:
URL: https://api.twitter.com/1.1/search/tweets.json?count=100&q=hello&since_id=577103514154893312&max_id=577103544903462913
Here is my code:
var twitter_auth = new OAuth(
"https://api.twitter.com/oauth/request_token",
"https://api.twitter.com/oauth/access_token",
config.consumer_key,
config.consumer_secret,
"1.0A",
null,
"HMAC-SHA1"
);
var request = twitter_auth.get(
"https://api.twitter.com/1.1/search/tweets.json" + url,
config.access_token,
config.access_token_secret
);
var chunk = "", message = "", that = this;
request.on("response", function(response){
response.setEncoding("utf8");
response.on("data", function(data){
chunk += data;
try {
message = JSON.parse(chunk);
} catch(e) {
return;
}
console.log(message);
if(message.statuses)
{
for(var i = 0; i < message.statuses.length; i++)
{
var tweet = message.statuses[i];
that.termData[term.name].push(tweet);
}
if(message.search_metadata.next_results)
{
that.openRequests.push(that.createNewSearch(message.search_metadata.next_results, term));
}
else
{
that.termCompleted(term);
}
}
else if(message)
{
console.log("Response does not appear to be valid.");
}
});
response.on("end", function(){
console.log("Search API End");
});
response.on("error", function(err){
console.log("Search API Error", err);
});
});
request.end();
The console.log(message) is returning this:
{
statuses: [],
search_metadata: {
completed_in: 0.007,
max_id: 577103544903462900,
max_id_str: '577103544903462913',
query: 'hello',
refresh_url: '?since_id=577103544903462913&q=hello&include_entities=1',
count: 100,
since_id: 577103514154893300,
since_id_str: '577103514154893312'
}
}
Any ideas what is going on? Why is the statuses array empty in my code but full of tweets in Postman?
This issue was described at twittercommunity.com.
Accordingly answer of user rchoi(Twitter Staff):
"Regarding web vs. API search, we're aware that the two return different results at the moment. We made upgrades to the web search. There is no timeline for those
changes to be brought to other parts of our system."
Try to use
https://dev.twitter.com/rest/reference/get/statuses/mentions_timeline
https://dev.twitter.com/rest/reference/get/statuses/user_timeline
if you get empty result with api search functionality.
Please follow this link
https://twittercommunity.com/t/search-tweets-api-returned-empty-statuses-result-for-some-queries/12257/6
I am using worlight JSONstore. I am new to it. I tried searching that read all docs but didn't get much idea.
I have one login page from that I get some json data I want to store that data using jsonstore. and get that afterwards.
I made jsonstore adapter.
Json-Store-Impl.js
function getJsonStores(custData) {
var data = custData;
return data;
//custdata is json
}
function addJsonStore(param1) {
var input = {
method : 'put',
returnedContentType : 'json',
path : 'userInputRequired'
};
return WL.Server.invokeHttp(input);
}
function updateJsonStore(param1) {
var input = {
method : 'post',
returnedContentType : 'json',
path : 'userInputRequired'
};
return WL.Server.invokeHttp(input);
}
function deleteJsonStore(param1) {
var input = {
method : 'delete',
returnedContentType : 'json',
path : 'userInputRequired'
};
return WL.Server.invokeHttp(input);
}
after that I Create a local JSON store.
famlCollection.js
;(function () {
WL.JSONStore.init({
faml : {
searchFields: {"response.mci.txnid":"string","response.mci.scrnseqnbr":"string","response.loginUser":"string","request.fldWebServerId":"string","response.fldRsaImageHeight":"string","request.fldRequestId":"string","request.fldTxnId":"string","response.fldDeviceTokenFSO":"string","response.fldRsaCollectionRequired":"string","response.datlastsuccesslogin":"string","response.fldRsaUserPhrase":"string","response.fldRsaAuthTxnId":"string","response.rc.returncode":"string","response.datcurrentlogin":"string","response.mci.deviceid":"string","response.customername":"string","request.fldDeviceId":"string","response.fldRsaUserStatus":"string","request.fldScrnSeqNbr":"string","response.fldRsaImageWidth":"string","request.fldLangId":"string","response.fldTptCustomer":"string","response.encflag":"string","response.rc.errorcode":"string","response.fldRsaImagePath":"string","response.mci.appid":"string","response.mci.requestid":"string","response.rc.errormessage":"string","response.mci.appserverid":"string","response.fldRsaCollectionType":"string","request.fldAppId":"string","response.fldRsaImageId":"string","request.fldLoginUserId":"string","response.mci.sessionid":"string","response.mci.langid":"string","response.mci.remoteaddress":"string","request.fldAppServerId":"string","response.mci.webserverid":"string","response.fldRsaImageText":"string","response.fldRsaEnrollRequired":"string","response.fldRsaActivityFlag":"string"},
adapter : {
name: 'JsonStore',
replace: 'updateJsonStore',
remove: 'deleteJsonStore',
add: 'addJsonStore',
load: {
procedure: 'getJsonStores',
params: [],
key: 'faml'
},
accept: function (data) {
return (data.status === 200);
}
}
}
}, {
password : 'PleaseChangeThisPassword'
})
.then(function () {
WL.Logger.debug(['Take a look at the JSONStore documentation and getting started module for more details and code samples.',
'At this point there is no data inside your collection ("faml"), but JSONStore is ready to be used.',
'You can use WL.JSONStore.get("faml").load() to load data from the adapter.',
'These are some common JSONStore methods: load, add, replace, remove, count, push, find, findById, findAll.',
'Most operations are asynchronous, wait until the last operation finished before calling the next one.',
'JSONStore is currently supported for production only in Android and iOS environments.',
'Search Fields are not dynamic, call WL.JSONStore.destroy() and then initialize the collection with the new fields.'].join('\n'));
})
.fail(function (errObj) {
WL.Logger.ctx({pretty: true}).debug(errObj);
});
}());
When I clicked on login button I call getJsonStores like this -
getJsonStores = function(){
custData = responseData();
var invocationData = {
adapter : "JsonStore",
procedure : "getJsonStores",
parameters : [custData],
compressResponse : true
};
//WL.Logger.debug('invoke msg '+invocationData, '');
WL.Client.invokeProcedure(invocationData, {
onSuccess : sucess,
onFailure : AdapterFail,
timeout: timeout
});
};
I followed these steps
Is this right way? and how can I check jsonstore working locally or not? and how can I store my jsondata in JSONStore? Where should I initialize the wlCommonInit function in project?
plz Help me out.
Open main.js and find the wlCommonInit function, add the JSONStore init code.
WL.JSONStore.init(...)
You already have an adapter that returns the data you want to add to JSONStore, call it any time after init has finished.
WL.Client.invokeProcedure(...)
Inside the onSuccess callback, a function that gets executed when you successfully get data from the adapter, start using the JSONStore API. One high level way to write the code would be, if the collection is empty (the count API returns 0), then add all documents to the collection.
WL.JSONStore.get(collectionName).count()
.then(function (countResult) {
if(countResult === 0) {
//collection is empty, add data
WL.JSONStore.get(collectionName).add([{name: 'carlos'}, {name: 'mike'}])
.then(function () {
//data stored succesfully
});
}
});
Instead of adding [{name: 'carlos'}, {name: 'mike'}] you probably want to add the data returned from the adapter.
Later in your application, you can use the find API to get data back:
WL.JSONStore.get(collectionName).findAll()
.then(function (findResults) {
//...
});
There is also a find API that takes queries (e.g. {name: 'carlos'}), look at the getting started module here and the documentation here.
It's worth mentioning that the JSONStore API is asynchronous, you must wait for the callbacks in order to perform the next operation.