c# httpcontent "400 - Bad Request" if string contains newline character - json

I am running into some kind of trouble communicating with a restful service from my .Net WPF application (.Net 4.5), in particular when sending a "PUT" request with some json data.
FYI: The restful service is running under Python Flask.
The method I use the following method to send request to the restful service:
HttpClient http = new HttpClient();
http.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", encodedCredentials);
http.Timeout = TimeSpan.FromSeconds(1);
// Add an Accept header for JSON format.
http.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpContent content = new StringContent(jDataString, Encoding.UTF8, "application/json");
When I submit usual string, all works just fine. But I am having trouble as soon as the string contains line breaks.
Using:
mytring.Replace("\r", "").Replace("\n", "")
works, my string is then accepted by the restful service.
Unfortunately, this is not acceptable, because I want to be able to retrieve line breaks.
I hence tried approaches like:
mytring.Replace("\n", "\\n").Replace("\r", "\\r")
or even with a inside-character to make sure I recognize the pattern:
mytring.Replace("\n", "\\+n").Replace("\r", "\\+r")
In both cases, my parsed string looks fine, but isn't accepted by the restful service.
Below two examples - the first version is accepted, not the second and third...
"XML_FIELD": "<IDs><Id Type=\"System.Int32\" Value=\"7\" /></IDs>"
"XML_FIELD": "<IDs>\r\n<Id Type=\"System.Int32\" Value=\"20\" />\r\n</IDs>"
"XML_FIELD": "<IDs>\+r\+n<Id Type=\"System.Int32\" Value=\"20\" />\+r\+n</IDs>"
Thanks in advance for the help!!
Regards!

Ok, got it...
The issue was coming from "\r\n" character which were coming directly from my DB...
Anyway, change to perform is for SERIALIZATION is
mySerializedString.Replace("\r\n", "\n")
.Replace("\n", "\\n")
.Replace("\r", "\\r")
.Replace("\'", "\\'")
.Replace("\"", "\\\"")
.Replace("\t", "\\t")
.Replace("\b", "\\b")
.Replace("\f", "\\f");
And to de-serialize do the inverse:
myDeSerializedString.Replace("\\n", "\n")
.Replace("\\r", "\r")
.Replace("\\'", "\'")
.Replace("\\\"", "\"")
.Replace("\\t", "\t")
.Replace("\\b", "\b")
.Replace("\\f", "\f");
NOTE: in the process, we loose "\r\n" characters (which are replaced by "\n").

Related

Spring WebClient not processing JSON content

I have an app that uses WebClient to fetch JSON data from ComicVine as follows:
WebClient client = WebClient.builder()
.baseUrl(url)
.defaultHeaders(
headers -> {
headers.add(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE);
headers.add(HttpHeaders.USER_AGENT, "ComiXed/0.7");
})
.build();
Mono<ComicVineIssuesQueryResponse> request =
client
.get()
.uri(url)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(ComicVineIssuesQueryResponse.class);
ComicVineIssuesQueryResponse response = request.block();
For a time this worked. But then, all of a sudden, it's throwing the following root exception when it executes:
Caused by: org.springframework.web.reactive.function.UnsupportedMediaTypeException: Content type 'application/json' not supported for bodyType=org.comixed.scrapers.comicvine.model.ComicVineIssuesQueryResponse
at org.springframework.web.reactive.function.BodyExtractors.lambda$readWithMessageReaders$12(BodyExtractors.java:201)
I'm not sure why it all of a sudden won't process JSON data. My unit test, which is explicitly returning JSON data and setting the content type properly:
private MockWebServer comicVineServer;
this.comicVineServer.enqueue(
new MockResponse()
.setBody(TEST_GOOD_BODY)
.addHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE));
Any ideas why this is the case? It's happening across multiple classes that use this same setup for WebClient and for testing.
After doing some digging, I added the following code to get the JSON as a String and then use ObjectMapper to convert it to the target type:
Mono<String> request =
client
.get()
.uri(url)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(String.class);
String value = request.block();
ObjectMapper mapper = new ObjectMapper();
ComicVineIssuesQueryResponse response = mapper.readValue(value, ComicVineIssuesQueryResponse.class);
This quickly exposed the underlying problem, which was that two instance variables in the response were annotated with the same JSON field name. Once I fixed that, things started working correctly again.
you can parse the json content to string without calling block method.
option 1) Jackson2Tokenizer
option 2) put your code which is calling "objectMapper.readValue(..) .." inside map operator.

How do You Encode WIQL JSON?

In Postman I'm able to submit a query to our Azure DevOps 2019 Server no problem:
POST https://<AZDOSERVER>/<COLLECTION>/<PROJECT>/<TEAM>/_apis/wit/wiql?api-version=5.0
{"query": "Select [System.Id] From WorkItems WHERE [System.AreaPath] UNDER '<AREANAME>'"}
But when I do it in Python it seems to run into an encoding issue with the single quotes around my AREANAME. Here's my code:
url = "https://<AZDOSERVER>/<COLLECTION>/<PROJECT>/<TEAM>/_apis/wit/wiql?api-version=5.0"
json = '{"query": "Select [System.Id] From WorkItems WHERE [System.AreaPath] UNDER '
+ "'<AREANAME>'" + '" }'
headers = {'Accept': 'application/json; api-version=5.0'}
response = request.post(url, json=json, auth=self.basicauth, headers=headers)
I get a 400 error with the following message:
b'{"count":1,"value":{"Message":"Error converting value \\"{\\"query\\":
\\"Select [System.Id] From WorkItems WHERE [System.AreaPath] UNDER \'<AREANAME>\'\\" }\\"
to type \'Microsoft.TeamFoundation.WorkItemTracking.WebApi.Models.Wiql\'. Path \'\',
line 1, position 92.\\r\\n"}}'
How should I pass JSON properly into the python request.post() method so that it's sent wtih proper encoding? I tried capturing my outbound request via Fiddler but it doesn't see python traffic. I'm installing Wireshark too but that's going to take a while. I also stepped thru the request.post method to try and understand how it's build the body of the request. It seems to be handling the single quotes properly when it covers to a byte array.
Figured it out: request.post() takes a dictionary object for the json parameter instead of a json string.

InDesign Socket HTTP response is missing sections

I am automating Adobe InDesign to create documents using JSON data gathered from a web API with a SQL Server backend. I am using the Sockets object to make an HTTP 1.0 call to our server. Sometimes the response received is missing about 1700 characters from various points within the JSON string, yet when I call the same API endpoint using curl or Postman I get a complete and valid response.
The response should be about 150k characters long, and I'm using conn.read(99999999) to read it. In addition, the appearance of the end of the string looks correct, so I don't believe it's any kind of truncation problem.
The problem only seems to occur when I request a UTF-8 encoding. If I request ASCII I get a complete and valid response, but missing various Unicode characters. If I request BINARY I get a complete and valid response but the JavaScript/ExtendScript seems to be handling any multi-byte Unicode characters received as individual bytes, rather than as the Unicode characters we want to display.
Here is an illustration of the behavior I'm seeing, using bogus data...
"Expected" response...
[{"Id":1, "name":"Random Name", "Text":"A bunch of text", "AnotherId": 1}]
"Actual" response...
[{"Id":1, "name":"Random Name", "Text":"A bunc": 1}]
The problem first manifested itself as a JSON2 parsing error, for obvious reasons, but the root of it seems to be the fact that parts of the data are going missing in-transit.
So far we've only seen this problem when making the call using the InDesign Sockets object, and not every response exhibits this behavior.
Any help or insights you could offer would be appreciated.
Here is the function I'm using to call for data...
function httpRequest(url, encoding) {
try {
var response = "";
var hostName = getHostFromUrl(url);
var pathAndQuery = getPathAndQueryFromUrl(url);
var httpGet = "GET ";
httpGet += pathAndQuery;
httpGet += " HTTP/1.0\r\nHost: ";
httpGet += hostName;
httpGet += "\r\n";
var conn = new Socket;
conn.timeout = 30;
//conn.encoding = encoding || "UTF-8";
//conn.charset = "UTF-16";
if (conn.open(hostName + ":80", encoding || "UTF-8")) {
// send a HTTP GET request
conn.writeln(httpGet);
// and read the server's response
response = conn.read(99999999);
conn.close();
}
return parseHttpResponse(response);
}
catch (e) {
$.writeln(e);
$.global.alert("There was a problem making an HTTP Request: " + e);
return null;
}
}
It turns out my handling of the HTTP response was too simplistic and needed extra logic to handle Unicode characters properly.
The solution, in my case, was to use the GetURL method made available by Kris Coppieter here.

Json put/get/post in Restful webservices not working

I am trying to pass parameters to a server and extract the report in csv format. So the code i have has PUT/GET/POST in the order. I could get GET and POST work, but when i add PUT there is no error just blank screen.
String output1 = null;
URL url = new URL("http://<servername>/biprws/raylight/v1/documents/12345/parameters");
HttpURLConnection conn1 = (HttpURLConnection) url.openConnection();
conn1.setRequestMethod("PUT");
conn1.setRequestProperty("Accept", "application/json");
conn1.setRequestProperty("Content-Type", "application/json; charset=utf-8");
conn1.setDoInput(true);
conn1.setDoOutput(true);
String body = "<parameters><parameter><id>0</id><answer><values><value>EN</value></values></answer></parameter></parameters>";
int len1 = body.length();
conn1.setRequestProperty("Content-Length", Integer.toString(len1));
conn1.connect();
OutputStreamWriter out1 = new OutputStreamWriter(conn1.getOutputStream());
out1.write(body, 0, len1);
out1.flush();
What i am trying to do is pass parameter EN to the report and refresh it, take the output in csv using GET. POST is used for login to the server. I could make GET and POST work and get the output in CSV but not refreshed one.
Appreciate very much any help here.
Thanks,
Ak
What is the response code from the server when using PUT?
A PUT may not actually return a body to display on the screen; often times a PUT will only return a 200 or 204 response code. 204 would clearly mean that the server took the data and applied it, but is not sending you anything back, 200/201 may include a response, but maybe not. It depends on the folks who implemented the API.
https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html (section 9.6)
Should a RESTful 'PUT' operation return something

Arduino Http Post with JSON

I'm trying to post a json data via Arduino.When ı'm trying to this code.ı will send a json data with QueryString.If ı try this code the server answer me with Wrong QueryString format.Which mean is ı'm connected to server and server got my data.
if (client.connect(server, 80)) {
Serial.println("connected");
// Make a HTTP request:
client.println("POST /URL?query=jsondata HTTP/1.1");
client.println("Host: **.**.**.**");
client.println("Connection: close\r\nContent-Type: application/x-www-form-urlencoded");
client.print("Content-Length: ");
client.println(PostData.length());
client.println();
client.println(PostData);
}
But My Main Plan is send my json data with querystring.If ı Try this code ;
client.println("POST /URL?query={request:{Header:{Username:kullaniciAdi,Password:123456},Item:{Serial:ABC123QWE,Data:100, DateOn:23/11/1986 15:45:24}}} HTTP/1.1");
I get a HTTP Error 400. The request is badly formed.
Anyone Has a any idea?
Yes, your URI contains spaces and may contain other characters to confuse the format of the post request. You need to encode these characters.
As far as I can tell, the Arduino standard libraries do not include any form of urlEncode method, which is common in other languages and libraries, so you will either have to create your own or look for one.
Your resulting code would be something like:
String request = "/URL?query={request:{Header:{Username:kullaniciAdi,Password:123456},Item:{Serial:ABC123QWE,Data:100, DateOn:23/11/1986 15:45:24}}}";
String encRequest = uriEncode(request); // need to write your own method for this...
String post = "POST " + encRequest + " HTTP/1.1");
client.println( post);
Some discussion on creating a uriEncode function is on the Arduino Forum and there also appears to be a working method on hardwarefun.com