I am working on sending JSON data from my HTTP server to my client. I have been successful in sending smaller sized JSON responses to the client, but once I have to wait and collect all the data before sending, my code does not function properly. I am using the .on('data', function (){}) method to collect the data and build it back up and the .on('end', function(){}) to try sending the data, but my code never enters either of these methods.
My code for sending the larger sized JSON data is below:
exports.sendJson = function (req, resp, data) {
resp.writeHead(200, "Valid EndPoints", { "Content-Type": "application/json" });
var payload = '';
resp.on('data', function(data){
payload += data;
})
.on('end', function(){
resp.write(JSON.stringify(payload));
});
resp.end();
};
My code that works fine for smaller sized JSON data is as follows:
exports.sendJson = function (req, resp, data) {
resp.writeHead(200, { "Content-Type": "application/json" });
if(data) {
resp.write(JSON.stringify(data));
}
resp.end();
};
Thanks in advance. I'm actually pretty happy I got this far.
A HTTP server response is a Writable; it does not emit data events.
If the thing you pass in to sendJson's data parameter is an object, you just do exactly what you're doing in the second example. There is no need to buffer anything. (If the thing you pass to sendJson is a stream, just pipe it to the response.)
(When parsing received JSON, you must buffer the data you get from a Readable -- the HTTP response on the client.)
Related
I have a Logic App that gets the contents from a SharePoint (.xlsx) and posts the body to an endpoint to get processed. now the content I see is a base64-encoded file, what I wanted to do was to post this data as is.
when I try to post it using postman it gets accepted successfully but when it is posted form the Logic app I get
BadRequest. Http request failed: the content was not a valid JSON.
but I can see that the body that was meant to be sent is of the type, which is a valid Json
{
"$content-type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"$content": "AA....VeryLong...B1241BACDFA=="
}
also tried this expression
decodeBase64(triggerBody()?[body('getFile')])
but I get a different error
InvalidTemplate. Unable to process template language expressions in action 'HTTP' inputs at line '1' and column '2565': 'The template language expression 'decodeBase64(triggerBody()?[body('getFile')])' cannot be evaluated because property '{
"$content-type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"$content": "UEsDBBQABgAIAAAAIQDuooLHjAEAAJkGAAATAAgCW0Nvb...
What I want to achieve is simple really I want to post to my end point the Json as is or the contents of the base64Encoded string.
If you decode the content with base64 you will find the content is garbled. This is because the content is in ooxml format then encoded with base64. And in logic app you could not decode the ooxml.
First solution, you could use Azure Function to write a method to read the document then return the content. Then in logic app call the function to get the content.
Second solution, change your file to a directly readable file(like .txt file), this way I tried and you could parse it Json.
Can you send me your postman request save it in collect and then export as file.
I think what you can do in postman can be done in logic app too.
Or send me your code I can modify it.Make sure you have post as body and it matches the parameter of Post at web api level.
var Webrequestdata = {
"webserviceurl": "http://examplecom/u/b/b/e.ee.msg",
"username": "123"
};
$.ajax({
cache: false,
type: "POST",
url: 'http://example.com/res/api/Outlookapi',
data: JSON.stringify(Webrequestdata),
contentType: "application/json",
success: function (result) {
console.log("email sent successfully");
},
error: function (response) { alert('Error: ' + response.responseText); }
});
after looking at your statement:
Logic App gets the contents from a SharePoint (.xlsx) and posts the
body to an endpoint to get processed. now the content I see is a
base64-encoded file
i assume your endpoint is still looking for a content type -
vnd.openxmlformats-officedocument.spreadsheetml.sheet
but you are passing a Base64 String
1) check if your "content-type" header is correctly placed change it to application/json if possible
2) Make sure you are processing the base64 string correctly.
In my case, I extracted the base64 string from an image file using Javascript. I got special keys at the start of base64 string like ''data:image/png;base64','ACTUAL BASE 64 STRING...' so i removed the special keys before the actual base64 string with some regular expression.
This sample Json request might help in attaching the base64 Content
Make sure you are converting your request to JSON using JSON.stringify
//var finalString = srcData.replace('data:image/png;base64,','');
var finalString = srcData.replace(/^,+?(\,)/, '');
finalString = srcData.substring(srcData.indexOf(',')+1, srcData.length);
var data = JSON.stringify({
"requests": [
{
"image": {
"content": finalString
},
"features":
[
{
"type": "DOCUMENT_TEXT_DETECTION",
"maxResults": 1
}
]
}
]
});
This is the XMLHttpRequest i used:
var xhr = new XMLHttpRequest();
xhr.withCredentials = true;
xhr.addEventListener("readystatechange", function () {
if (this.readyState === 4) {
console.log(this.responseText);
var obj = JSON.parse(this.responseText);
try
{
//do something
}catch(err)
{
//do something
}
}
});
try {
xhr.open("POST", "https://vision.googleapis.com/v1/images:annotate?key=blablabla");
xhr.setRequestHeader("Content-Type", "application/json");
xhr.setRequestHeader("cache-control", "no-cache");
xhr.setRequestHeader("Postman-Token", "111111111");
xhr.send(data);
}
catch(err) {
//do something
}
Until recently I've been fetching XML data using the node request module, and then running that XML through an XML to JSON converter. I discovered by accident that if I set json: true as an option (even knowing the endpoint returns XML, not JSON), I was actually getting back JSON:
var request = require('request');
var options = { gzip: true, json: true, headers: { 'User-Agent': 'stackoverflow question (https://stackoverflow.com/q/52609246/4070848)' } };
options.uri = 'https://api.met.no/weatherapi/locationforecast/1.9/?lat=40.597&lon=-74.26';
request(options, function (error, response, body) {
console.log(`body for ${options.uri}: ${JSON.stringify(body)}`);
});
The above call returns JSON, whereas the raw URL is actually sending XML. Sure enough, with json: false the returned data is XML:
var request = require('request');
var options = { gzip: true, json: true, headers: { 'User-Agent': 'stackoverflow question (https://stackoverflow.com/q/52609246/4070848)' } };
options.uri = 'https://api.met.no/weatherapi/locationforecast/1.9/?lat=40.597&lon=-74.26';
options.json = false; // <<--- the only difference in the request
request(options, function (error, response, body) {
console.log(`body for ${options.uri}: ${body}`);
});
So I thought "that's handy", until I tried the same trick with a different URL that also returns XML, and in this case the returned data is still XML despite using the same request options:
var request = require('request');
var options = { gzip: true, json: true, headers: { 'User-Agent': 'stackoverflow question (https://stackoverflow.com/q/52609246/4070848)' } };
options.uri = 'https://graphical.weather.gov/xml/SOAP_server/ndfdXMLclient.php?whichClient=NDFDgen&lat=40.597&lon=-74.26&product=time-series&temp=tempSubmit=Submit';
request(options, function (error, response, body) {
console.log(`body for ${options.uri}: ${body}`);
});
What is the difference here? How do I get the latter request to return the data in JSON format (so that I can avoid the step of converting XML to JSON myself)? Maybe the endpoint in the first example can detect that JSON is requested and it does in fact return JSON rather than XML?
EDIT weirdly, the first request is now returning XML rather than JSON even with json: true. So maybe this behaviour was down to what was being sent from the endpoint, and they've changed this even since I posted a few hours ago
So now that the behavior is unrepeatable, the answer is less useful for your particular problem, but I think it's worth pointing out that when you set json:true on the request module, it does a few things under the hood for you:
Sets the Accept header to 'application/json'
Parses the response body using JSON.parse()
Request types with a body also get the body automatically serialized as JSON
Request types with a body also get the Content-Type header added as 'application/json'
So perhaps they did change it, but there are plenty of web services I've seen that will detect the content-type to send based on the Accept header and respond appropriately for some set of types that make sense (usually XML or JSON, but sometimes CSV, TXT, HTML, etc).
To handle XML query, I usually do something like this using the request module:
import parser from "xml2json";
const resp = await rp({
method: "POST",
url: 'some url',
form: {xml_query}, // set XML query to xml_query field
});
const parsedData = parser.toJson(resp, {
object: true, // returns a Javascript object instead of a JSON string
coerce: true, // makes type coercion.
});
I have the following function which returns a promise to get data. I am using the request npm module.
let getData = function (user) {
return new Promise(function (resolve, reject) {
let url = 'https://someurl.com/' + user;
request(url, function (error, res, body) {
if (error) reject(error);
try{
resolve(cleanData(JSON.parse(body).items))
}catch(e){
console.log(body)
console.log(url)
console.log(e);
}
})
})
}
When I resolve my promise, I sometimes get something like this:
SyntaxError: Unexpected token u in JSON at position 0
That happens when my body is returned as undefined. But other times I also get something like this:
SyntaxError: Unexpected token f in JSON at position 11203
And other times (most of the times), it goes through perfectly.
I've been able to double check this while debugging and it seems that there are times when I get an incomplete body. I know for a fact that the body from the source url is not incomplete. I checked this by going to the url directly with my browser and making sure the json was complete and valid.
What is going on? It is my understanding that the callback on the request function is only called when the response is ready to be consumed.
No - the callback is triggered as soon as there is a response - see
HTTP request API documentation. You will need to implement a response listener as:
res.on('data', (chunk) => {
// do the data accumulation here
});
res.on('end', () => {
// do the resolve here
});
In request module if you expect binary data from response, then set encoding: null in request options.
Excerpt from:
encoding - encoding to be used on setEncoding of response data. If null, the body is returned as a Buffer. Anything else (including the default value of undefined) will be passed as the encoding parameter to toString() (meaning this is effectively utf8 by default). (Note: if you expect binary data, you should set encoding: null.)
https://www.npmjs.com/package/request
So I have a very large JSON data which I am sending to a node server via AngularJS. This is the procedure I'm using to send data :-
var send_data="data="+encodeURIComponent(JSON.stringify($scope.records,null,4));
$http({
method : 'POST',
url : 'http://localhost:8888/updateDetails',
data : send_data,
responseType : "json",
headers: {
"Content-Type": 'application/x-www-form-urlencoded'
}
}).success(function(data, status, headers, config){
console.log(data);
});
I successfully managed to send a pretty printed JSON to the Node Server via the above code. However when I write this to a file using :-
jsonfile.writeFile(file, JSON.parse(req['body']['data']), function (err) {
console.error(err);
});
After some testing I figured out that the error is in the JSON.parse statement. Any way to pretty print JSON to a file?
Use JSON.stringify(data[, replacer[, indent]]):
jsonfile.writeFile(file, JSON.stringify(JSON.parse(req.body.data), 0, 4), function (err) {
console.error(err);
});
You also may or may not need to parse the response body though -- I believe responseType: "json" in the builder will automatically parse it for you:
jsonfile.writeFile(file, JSON.stringify(req.body.data, 0, 4), function (err) {
console.error(err);
});
Here's a complete working isolated example of how it works:
var fs = require('fs');
var req = JSON.parse('{"body": {"data": "Some text."}}');
fs.writeFile('test.json', JSON.stringify(req.body, 0, 4), function (err) {
});
Assuming you're handling POST requests via an HTTP listener or framework like express? Are you using a body parser to parse the incoming JSON?
Let's try:
console.log(req.body) to make sure your server recognizes the request.
Is it a JSON string? Is it valid? Or is it a javascript object (i.e. already been parsed)?
If it's been parsed, then my second suggestion will work. If it hasn't been parsed, parse it as I did in the first example.
I have a middleware function using Node's Express4 to log each request & response for debugging. I use the res.json call in the request handler to send back JSON to the client for all but static files. So I do not want to log the response for static files, but only the JSON responses. I have the following code:
function logRequests(req, res, next) {
// do logging (will show user name before authentication)
logger.reqLog('IN '+req.method+' '+req.url, req);
var oldEnd = res.end,
oldWrite = res.write,
chunks = [];
res.write = function(chunk) {
chunks.push(chunk);
oldWrite.apply(res, arguments);
};
res.end = function(chunk, encoding) {
if(chunk) {
chunks.push(chunk);
}
oldEnd.apply(res, arguments);
// the content-type prints "undefined" in some cases
// even though the browser shows it returned as "application/json"
console.log('type='+res.get('content-type'));
if(res.get('content-type') === 'application/json') {
var body = Buffer.concat(chunks).toString('utf8');
logger.info(body, req);
}
logger.reqLog('OUT '+req.method+' '+req.path, req);
};
next(); // make sure we go to the next routes and don't stop here
}
So why do some requests show the correct content type in the middleware meaning they also print the response fine and others do not? All of them look good in the REST client when inspecting the returned headers.
EDIT: Some more info discovered tonight while trying to figure this out - if I append any character as a dummy request parameter, it logs the response type correctly:
http://localhost:8081/node/ionmed/api/logout?0 WORKS
where
http://localhost:8081/node/ionmed/api/logout DOES NOT
Also, I can always get a response type logging in the middleware function if I replace the .json() call with .end() so this:
res.json({ item: 'logout', success: true });
becomes:
res.set('content-type', 'application/json');
res.end({ item: 'logout', success: true });