HttpClient reads incomplete or malformed string - json

Im trying to figure out which part of my app (Xamarin Forms and proxy written in PHP) is buggy. Firstly I thought that my proxy (written in PHP) is working incorrectly with long set of data (ie. json containing 1.300.000 characters) and returns malformed response, but every single request with Postman gives me correct JSON, which is successfully decoded with third-party tools. So I assume, proxy is working well.
The problem is (I guess) with decoding response in my Xamarin Forms (2.0.0-beta.22) app. I'm using HttpClient to read response with this code:
response.EnsureSuccessStatusCode();
var entries = new List<HistoryEntry>();
var content = await response.Content.ReadAsStringAsync();
_loggerService.Error(content);
response is just GetAsync response from HttpClient. The problem is: content is randomly incomplete/malformed. Saying this I mean last character is missing (}) or JSON keys/values have additional " character, which breaks everything. Unfortunately, I can make exactly the same requests many times and once it works, once not. I found out that this behavior happens only with large set of data (as I mentioned before, long JSON string).
Is there any possibility that ReadAsStringAsync does not wait for full response or in any way alters my response string? How can I find the reason of wrongly downloaded data?
EDIT 21.05.2019:
Just copied valid JSON (available here: https://github.com/jabools/xamarin/blob/master/json.txt) and returned it from Lumen app by response()->json(json_decode(..., true)) and still the same result. Hope someone will be able to reproduce this and help me with this issue :( More informations in comments.
Used this code in C#:
Device.BeginInvokeOnMainThread(async () =>
{
for (int i = 0; i < 10; i++)
{
var response = await client.GetAsync("<URL_TO_PHP>");
//var response = await client.GetAsync("https://jsonplaceholder.typicode.com/photos");
var content = await response.Content.ReadAsAsync<object>();
Debug.WriteLine("Deserialized: " + i);
}
});

Related

How can I prevent API returning response as gibberish data instead of JSON?

I am using RapidAPI for integration of https://priceline-com-provider.p.rapidapi.com/v1/hotels/search external API with my asp.net micro-service. I need to get a JSON response from the API however it is returning gibberish data. I have tried many things to no success.
To make sure it is good the request, I used Postman to test and it worked out fine.
What can I do to solve this?
In my case, the response was Accept-Encoding: deflate/gzip and therefore I had to insert a HttpClientHandler into the HttpClient with the correct settings:
using (var HttpClientHandler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
})
{
using (var client = new HttpClient(HttpClientHandler))
{
}
}
Once I did, the await response.Content.ReadAsStringAsync() returned deserializable JSON data.
This answer led me to the solution: https://stackoverflow.com/a/69222690/2504659

InDesign Socket HTTP response is missing sections

I am automating Adobe InDesign to create documents using JSON data gathered from a web API with a SQL Server backend. I am using the Sockets object to make an HTTP 1.0 call to our server. Sometimes the response received is missing about 1700 characters from various points within the JSON string, yet when I call the same API endpoint using curl or Postman I get a complete and valid response.
The response should be about 150k characters long, and I'm using conn.read(99999999) to read it. In addition, the appearance of the end of the string looks correct, so I don't believe it's any kind of truncation problem.
The problem only seems to occur when I request a UTF-8 encoding. If I request ASCII I get a complete and valid response, but missing various Unicode characters. If I request BINARY I get a complete and valid response but the JavaScript/ExtendScript seems to be handling any multi-byte Unicode characters received as individual bytes, rather than as the Unicode characters we want to display.
Here is an illustration of the behavior I'm seeing, using bogus data...
"Expected" response...
[{"Id":1, "name":"Random Name", "Text":"A bunch of text", "AnotherId": 1}]
"Actual" response...
[{"Id":1, "name":"Random Name", "Text":"A bunc": 1}]
The problem first manifested itself as a JSON2 parsing error, for obvious reasons, but the root of it seems to be the fact that parts of the data are going missing in-transit.
So far we've only seen this problem when making the call using the InDesign Sockets object, and not every response exhibits this behavior.
Any help or insights you could offer would be appreciated.
Here is the function I'm using to call for data...
function httpRequest(url, encoding) {
try {
var response = "";
var hostName = getHostFromUrl(url);
var pathAndQuery = getPathAndQueryFromUrl(url);
var httpGet = "GET ";
httpGet += pathAndQuery;
httpGet += " HTTP/1.0\r\nHost: ";
httpGet += hostName;
httpGet += "\r\n";
var conn = new Socket;
conn.timeout = 30;
//conn.encoding = encoding || "UTF-8";
//conn.charset = "UTF-16";
if (conn.open(hostName + ":80", encoding || "UTF-8")) {
// send a HTTP GET request
conn.writeln(httpGet);
// and read the server's response
response = conn.read(99999999);
conn.close();
}
return parseHttpResponse(response);
}
catch (e) {
$.writeln(e);
$.global.alert("There was a problem making an HTTP Request: " + e);
return null;
}
}
It turns out my handling of the HTTP response was too simplistic and needed extra logic to handle Unicode characters properly.
The solution, in my case, was to use the GetURL method made available by Kris Coppieter here.

JSON.parse returning undefined object

Blizzard just shut down their old API, and made a change so you need an apikey. I changed the URL to the new api, and added the API key. I know that the URL is valid.
var toonJSON = UrlFetchApp.fetch("eu.api.battle.net/wow/character/"+toonRealm+"/"+toonName+"?fields=items,statistics,progression,talents,audit&apikey="+apiKey, {muteHttpExceptions: true})
var toon = JSON.parse(toonJSON.getContentText())
JSON.pase returns just an empty object
return toon.toSorce() // retuned ({})
I used alot of time to see if i could find the problem. have come up empty. Think it has something to do with the "responce headers".
Responce headers: http://pastebin.com/t30giRK1 (i got them from dev.battle.net (blizzards api site)
JSON: http://pastebin.com/CPam4syG
I think it is the code you're using.
I was able to Parse it by opening the raw url of your pastebin JSON http://pastebin.com/raw/CPam4syG
And using the following code
var text = document.getElementsByTagName('pre')[0].innerHTML;
var parse = JSON.parse(text);
So to conclude I think it is the UrlFetchApp.fetch that's returning {}
So i found the problems:
I needed https:// in the URL since i found after some hours that i had an SSL error
If you just use toString instead of getContentText it works. Thow why getContentText do not work, i am not sure of.
was same problem, this works for me (dont forget to paste your key)
var toonJSON = UrlFetchApp.fetch("https://eu.api.battle.net/wow/character/"+toonRealm+"/"+toonName+"?fields=items%2Cstatistics%2Cprogression%2Caudit&locale=en_GB&apikey= ... ")

Json put/get/post in Restful webservices not working

I am trying to pass parameters to a server and extract the report in csv format. So the code i have has PUT/GET/POST in the order. I could get GET and POST work, but when i add PUT there is no error just blank screen.
String output1 = null;
URL url = new URL("http://<servername>/biprws/raylight/v1/documents/12345/parameters");
HttpURLConnection conn1 = (HttpURLConnection) url.openConnection();
conn1.setRequestMethod("PUT");
conn1.setRequestProperty("Accept", "application/json");
conn1.setRequestProperty("Content-Type", "application/json; charset=utf-8");
conn1.setDoInput(true);
conn1.setDoOutput(true);
String body = "<parameters><parameter><id>0</id><answer><values><value>EN</value></values></answer></parameter></parameters>";
int len1 = body.length();
conn1.setRequestProperty("Content-Length", Integer.toString(len1));
conn1.connect();
OutputStreamWriter out1 = new OutputStreamWriter(conn1.getOutputStream());
out1.write(body, 0, len1);
out1.flush();
What i am trying to do is pass parameter EN to the report and refresh it, take the output in csv using GET. POST is used for login to the server. I could make GET and POST work and get the output in CSV but not refreshed one.
Appreciate very much any help here.
Thanks,
Ak
What is the response code from the server when using PUT?
A PUT may not actually return a body to display on the screen; often times a PUT will only return a 200 or 204 response code. 204 would clearly mean that the server took the data and applied it, but is not sending you anything back, 200/201 may include a response, but maybe not. It depends on the folks who implemented the API.
https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html (section 9.6)
Should a RESTful 'PUT' operation return something

BackgroundTransferRequest DownloadLocation to get the Response content

I am uploading a file to the server using BackgroundTransferRequest object. I need to access the response header sent from the server. Based on the research, I understand that there is no direct access to the Response of the request and one possible work around is to provide a download location to which the response content will be written.
However, when I try to do that, there is no data written to that file and its 0 bytes. My request server is actually sending a response having ONLY the headers (without any content).
So my question is, does the response have to have body in order to get the response written to the DownloadLocation?
My code sample:
var request = new BackgroundTransferRequest(targetUri)
{
DownloadLocation = new Uri(downloadTo, UriKind.Relative),
UploadLocation = new Uri(uploadFrom, UriKind.Relative),
Method = "POST"
};