Sending a json serialized object containing a escaped string using HttpWebRequest on mono - json

The requirement serialize a class and send it to the server.
Dev environment:
MonoDevelop 3.0.6
Runtime:
Mono 2.10.9 (tarball)
GTK 2.24.10
GTK# (2.12.0.0)
Operating System:
Mac OS X 10.7.4
The class contains a string with an escaped double quote in it.
class CustomClass
{
public string foo = "hi!\"";
}
The issue is that when I serialize it, encode it and create a URI object, the backslash used to escape double quote in the variable foo is converted into a forward slash, thus breaking the json.
Below are values of the different variables of the URI instance
Uri:
http://myserver/hello_world/0/{"foo":"hi!/""}
AbsoluteUri:
http://myserver/hello_world/0/%7B%22foo%22%3A%22hi%21/%22%22%7D
OriginalString:
http://myserver/hello_world/0/%7B%22foo%22%3A%22hi%21%5C%22%22%7D
The HttpWebRequest send the value "http://myserver/hello_world/0/{"foo":"hi!/""}" to the server, but for my requirement it should use the OriginalString to get a valid response from the server.
If i test the code on .NET the OriginalString is being sent to the server by the HttpWebRequest class, but there is additional code (hack) which doesn't work on mono mac
string paq = requestUri.PathAndQuery;
FieldInfo flagsFieldInfo = typeof(Uri).GetField("m_Flags", BindingFlags.Instance | BindingFlags.NonPublic);
ulong flags = (ulong)flagsFieldInfo.GetValue(requestUri);
flags &= ~((ulong)0x30); // Flags.PathNotCanonical|Flags.QueryNotCanonical
flagsFieldInfo.SetValue(requestUri, flags);
The code :
object messageObject = new CustomClass();
//try 1
//string jsonString = Uri.EscapeUriString(JsonConvert.SerializeObject(message2));
//try 2
//string jsonString = Uri.EscapeDataString(JsonConvert.SerializeObject(message2));
//try 3
string jsonString = System.Web.HttpUtility.UrlEncode(JsonConvert.SerializeObject(messageObject));
Uri uri = new Uri(string.Format("http://myserver/hello_world/0/{0}", jsonString));
//try 4:
//HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri.AbsoluteUri);
//try 5
//HttpWebRequest request =(HttpWebRequest)WebRequest.Create(string.Format("http://myserver/hello_world/0/{0}",jsonString));
//try 6
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri.OriginalString);
try
{
using (HttpWebResponse resp = request.GetResponse() as HttpWebResponse)
{
var reader = new StreamReader(resp.GetResponseStream());
string jsonResponse = reader.ReadToEnd();
}
}
catch (WebException wex)
{
Debug.WriteLine(wex.ToString());
}
As you can see I have tried using Uri.EscapeUriString, Uri.EscapeDataString and System.Web.HttpUtility.UrlEncode and using the AbsouluteUri and OriginalString to create the WebRequest instance and also creating it using a string.
I have also tried using the following code in the config.
<uri>
<schemeSettings>
<clear/>
<add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes"/>
</schemeSettings>
</uri>
But none of the methods work and the encoding is lost when the request in created.
Any ideas to get this to work will be appreciated.
Update:
I have tried a lot of things including testing with some 3rd party open source code, but still faced the same issue when sending in an encoded url.
I modified the code to use WebClient instead of WebRequest but still no luck.
So the solution that worked for me on both MonoMac an MonoTouch(simulator, i haven't tested it on the device) was to create the request using TcpClient.
using (TcpClient tc = new TcpClient()) {
tc.Connect ("myserver", 80);
using (NetworkStream ns = tc.GetStream()) {
using (System.IO.StreamWriter sw = new System.IO.StreamWriter(ns)) {
using (System.IO.StreamReader sr = new System.IO.StreamReader(ns)) {
sw.Write("GET /hello_world/0/%7B%22foo%22:%22hi!%5C%22%22%7D HTTP/1.1 Host:myserver \r\n\r\n");
sw.Flush ();
string line;
while ((line=sr.ReadLine())!=null)
Console.Write (line);
}
}
}
}

I have tried a lot of things including testing with some 3rd party open source code, but still faced the same issue when sending in an encoded url.
I modified the code to use WebClient instead of WebRequest but still no luck.
So the solution that worked for me on both MonoMac an MonoTouch(simulator, i haven't tested it on the device) was to create the request using TcpClient.
using (TcpClient tc = new TcpClient()) {
tc.Connect ("myserver", 80);
using (NetworkStream ns = tc.GetStream()) {
using (System.IO.StreamWriter sw = new System.IO.StreamWriter(ns)) {
using (System.IO.StreamReader sr = new System.IO.StreamReader(ns)) {
sw.Write("GET /hello_world/0/%7B%22foo%22:%22hi!%5C%22%22%7D HTTP/1.1 Host:myserver \r\n\r\n");
sw.Flush ();
string line;
while ((line=sr.ReadLine())!=null)
Console.Write (line);
}
}
}
}

Related

JSON - WepAPI - Unexpected character encountered while parsing value

ANY help will be greatly appreciated
I have a Generic class that facilitates WebAPI calls, Its been in place for quite sometime and has had no issue. Today I'm getting an error and not sure where to track the problem. the exact error is
{"Unexpected character encountered while parsing value: [. Path 'PayLoad', line 1, position 12."}
what I'm getting back as the result of the call is
"{\"PayLoad\":[\"file_upload_null20180629155922²AAGUWVP2XUezeM3CiEnSOw.pdf\"],\"Success\":true,\"Message\":\"1 File(s) Uploaded\",\"Exceptions\":[]}"
Which looks right and is what I expect back from the service call
Here is the method that I'm calling that suddenly quit working, and its failing on the last line
public static TR WebApiPost(string serveraddress, string endpoint, object data)
{
HttpResponseMessage msg;
var clienthandler = new HttpClientHandler
{
UseDefaultCredentials = false,
Credentials = new NetworkCredential(user, password, domain)
};
using (var client = new HttpClient(clienthandler) { BaseAddress = new Uri(serveraddress) })
{
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
msg = client.PostAsync(endpoint, new StringContent(new JavaScriptSerializer().Serialize(data), Encoding.UTF8, "application/json")).Result;
}
var result = msg.Content.ReadAsStringAsync().Result;
return JsonConvert.DeserializeObject<TR>(result);
}
AND finally the line that actually makes the call (which should not matter)
returned = CallHelper<ResultStatus<string>>.WebApiPost(serviceurl, sendFileUrl, model);
It's not clear where your web service is getting the value of PayLoad from, so it is very possible that the value has a Byte Order Mark (BOM) at its beginning. This is especially the case if you are returning the content of what was originally a Unicode encoded file.
Be aware that a BOM is NOT visible when you are viewing a string in the debugger.
On your web service, make sure that you are not returning a BOM in the value of PayLoad. Check for this byte sequence at the beginning of the string:
0xEF,0xBB,0xBF
For more information on Byte Order Mark:
https://en.wikipedia.org/wiki/Byte_order_mark

.Net Core send/receive JSON over TCP

I want to send and receive JSON over TCP.
QUESTION: I have to send and receive JSON in my TCP client-server. How can I achieve it?
I use TcpListener and TcpClient to connect and I have this code:
NetworkStream stream = client.GetStream();
var serializer = new JsonSerializer();
var sr = new StreamReader(stream, new UTF8Encoding(), false);
var jsonTextReader = new JsonTextReader(sr);
var data = serializer.Deserialize(jsonTextReader).ToString();
Console.WriteLine("Received: {0}", data);
StringBuilder sb = new StringBuilder();
StringWriter sw = new StringWriter(sb);
JsonWriter writer = new JsonTextWriter(sw);
writer.WriteValue('1');
byte[] buffer = Encoding.ASCII.GetBytes(writer.ToString());
stream.Write(buffer, 0, buffer.Length);
Can I do it better? The client has to receive JSON(I use Newtonsoft.Json) and I don't know if it is even good code. Maybe you write me some good practices? Or maybe some tips.
EDIT.
Now I wrote something like this:
public static T DeserializeFromStream<T>(Stream stream)
{
using (var sr = new StreamReader(stream))
using (var jsonTextReader = new JsonTextReader(sr))
{
return new JsonSerializer().Deserialize<T>(jsonTextReader);
}
}
And it doesn't work because Java client send me array like: [{"name" : "logo", "session" : "i3fnj34njn780"}] So how can I fix this problem? I want call it this way: Method ar = DeserializeFromStream<Method>(client.GetStream()); Trim and Replace doesn't work for me here.
byte[] buffer = Encoding.ASCII.GetBytes(writer.ToString());
This is incorrect. Per RFC4627:
Encoding
JSON text SHALL be encoded in Unicode. The default encoding is
UTF-8.
You are not sending JSON.
StringBuilder sb = new StringBuilder();
StringWriter sw = new StringWriter(sb);
JsonWriter writer = new JsonTextWriter(sw);
This is unnecessarily writing a SB. Write directly into the network stream:
using (NetworkStream stream = client.GetStream()) {
using (TextWriter tw = new StreamWriter(stream, Encoding.UTF8)) {
using (JsonWriter writer = new JsonTextWriter(tw)) {
...
}
}
}
Also, use using.
Plus, everything #CodeCaster says. This should be a proper Web API, not some rogue TCP server. Not only the obvious issue of having more than one request type (ie. routing), but you have to consider proxies (none will allow some arbitrary port), server authentication (you must tunnel through HTTPS and validate the server cert in the Android APP), make allowance for web caching, HTTP headers etc etc. And you need proper error states and error codes for your 'protocol', which HTTP provides out-of-the-box. And a good job would be to model a proper REST API, and likely a good data model on top of JSON, like JSON-API.

WCF GET returning XML instead of JSON on server

I have a WCF service with a webHttpBinding defined. The interface has a single method:
[OperationContract(Action = "*")]
[WebGet(UriTemplate = "/",RequestFormat = WebMessageFormat.Json, ResponseFormat = WebMessageFormat.Json)]
Stream GetServerInfo();
Which returns a stream with encoded JSON:
WebOperationContext.Current.OutgoingResponse.Headers.Add("Access-Control-Allow-Origin", "*");
ServerData r = ServerData.Instance;
r.upTime = (DateTime.Now - r._startupTime).ToString(#"dd\.hh\:mm\:ss");
using (Process proc = Process.GetCurrentProcess())
{
r.usedMemory = ((double)proc.PrivateMemorySize64) / 1024 / 1024;
}
r.activeSessions = getServiceData().Count();
string jsCode = "displayData" + "("+
new JavaScriptSerializer().Serialize(r)
+")";
WebOperationContext.Current.OutgoingResponse.ContentType = "application/javascript";
Console.WriteLine(jsCode);
return new MemoryStream(Encoding.UTF8.GetBytes(jsCode));
When used on my devel PC it works fine. I get sth like:
displayData({"_startupTime":"\/Date(1435867525056)\/","serverVersion":"1.0.0.8","startUpTime":"2. 7. 2015 22:05:25","acceptedConnections":0,"upTime":"00.00:00:00","usedMemory":21.265625,"activeSessions":0})
However, after deploy to a remote server I get only following response and I want to get a JSON:
<GetServerInfoResponse xmlns="http://tempuri.org/"><GetServerInfoResult>ZGlzcGxheURhdGEoeyJfc3RhcnR1cFRpbWUiOiJcL0RhdGUoMTQzNTg2ODUwMjc5NClcLyIsInNlcnZlclZlcnNpb24iOiIxLjAuMC44Iiwic3RhcnRVcFRpbWUiOiI3LzIvMjAxNSAxOjIxOjQyIFBNIiwiYWNjZXB0ZWRDb25uZWN0aW9ucyI6MCwidXBUaW1lIjoiMDAuMDA6MDA6MDAiLCJ1c2VkTWVtb3J5IjoyNy40NzY1NjI1LCJhY3RpdmVTZXNzaW9ucyI6MH0p</GetServerInfoResult></GetServerInfoResponse>
Note I call for the request locally directly on the server. But on remote call over network the response is the same. If I put a log output of the string to console I can see the output string is correct.
The config files are identical (except for addresses).
--edit
With the try-fail method I found out the string inside the XML response is the JSON string encoded in Base64.
Can somebody please help me whats wrong?
After some research I did not find any solution. However, after restarting OS (Win Server 2012) and rebuilding it just works like a charm.

WebRequest Caching Windows Phone 7

From what I understand, the HttpWebRequest class always cache the downloaded data. Now I don't mind this, but after a throughly reparsing the same URL through HttpWebRequest during the app duration, I've noticed that the data becomes corrupted (as in the downloaded JSON data becomes unparsable). After rebooting the Phone Emulator, it all goes smoonthy until it happens again.
Now I am just wondering if it possible to turn off the caching in HttpWebRequest.
Here is some of the code I am using to make a httpwebrequest call:
var request = (HttpWebRequest)WebRequest.Create(string.Format(uri));
request.BeginGetResponse(a =>
{
var response = request.EndGetResponse(a);
var responseStream = response.GetResponseStream();
using (var sr = new StreamReader(responseStream))
{
string json = sr.ReadToEnd();
Deployment.Current.Dispatcher.BeginInvoke(() =>
{
//Newtonsoft.Json.Linq;
JObject artistObject = JObject.Parse(json);
//...etc
});
}
}, null);
}
A common technique to get around this caching is to add an parameter to the query string that is incremented on successive calls. This thread discusses the silverlight behaviour in more detail, and covers some server handling you can look at too if you have that control.
WebClient Caching Problem
With that said, have you been able to produce a simple repro of the corruption you're experiencing? It might be worth getting that looked into.

Posting a File and Associated Data to a RESTful WebService preferably as JSON

In an application I am developing RESTful API and we want the client to send data as JSON. Part of this application requires the client to upload a file (usually an image) as well as information about the image.
I'm having a hard time tracking down how this happens in a single request. Is it possible to Base64 the file data into a JSON string? Am I going to need to perform 2 posts to the server? Should I not be using JSON for this?
As a side note, we're using Grails on the backend and these services are accessed by native mobile clients (iPhone, Android, etc), if any of that makes a difference.
I asked a similar question here:
How do I upload a file with metadata using a REST web service?
You basically have three choices:
Base64 encode the file, at the expense of increasing the data size by around 33%, and add processing overhead in both the server and the client for encoding/decoding.
Send the file first in a multipart/form-data POST, and return an ID to the client. The client then sends the metadata with the ID, and the server re-associates the file and the metadata.
Send the metadata first, and return an ID to the client. The client then sends the file with the ID, and the server re-associates the file and the metadata.
You can send the file and data over in one request using the multipart/form-data content type:
In many applications, it is possible for a user to be presented with
a form. The user will fill out the form, including information that
is typed, generated by user input, or included from files that the
user has selected. When the form is filled out, the data from the
form is sent from the user to the receiving application.
The definition of MultiPart/Form-Data is derived from one of those
applications...
From http://www.faqs.org/rfcs/rfc2388.html:
"multipart/form-data" contains a series of parts. Each part is
expected to contain a content-disposition header [RFC 2183] where the
disposition type is "form-data", and where the disposition contains
an (additional) parameter of "name", where the value of that
parameter is the original field name in the form. For example, a part
might contain a header:
Content-Disposition: form-data; name="user"
with the value corresponding to the entry of the "user" field.
You can include file information or field information within each section between boundaries. I've successfully implemented a RESTful service that required the user to submit both data and a form, and multipart/form-data worked perfectly. The service was built using Java/Spring, and the client was using C#, so unfortunately I don't have any Grails examples to give you concerning how to set up the service. You don't need to use JSON in this case since each "form-data" section provides you a place to specify the name of the parameter and its value.
The good thing about using multipart/form-data is that you're using HTTP-defined headers, so you're sticking with the REST philosophy of using existing HTTP tools to create your service.
I know that this thread is quite old, however, I am missing here one option. If you have metadata (in any format) that you want to send along with the data to upload, you can make a single multipart/related request.
The Multipart/Related media type is intended for compound objects consisting of several inter-related body parts.
You can check RFC 2387 specification for more in-depth details.
Basically each part of such a request can have content with different type and all parts are somehow related (e.g. an image and it metadata). The parts are identified by a boundary string, and the final boundary string is followed by two hyphens.
Example:
POST /upload HTTP/1.1
Host: www.hostname.com
Content-Type: multipart/related; boundary=xyz
Content-Length: [actual-content-length]
--xyz
Content-Type: application/json; charset=UTF-8
{
"name": "Sample image",
"desc": "...",
...
}
--xyz
Content-Type: image/jpeg
[image data]
[image data]
[image data]
...
--foo_bar_baz--
Here is my approach API (i use example) - as you can see, you I don't use any file_id (uploaded file identifier to the server) in API:
Create photo object on server:
POST: /projects/{project_id}/photos
body: { name: "some_schema.jpg", comment: "blah"}
response: photo_id
Upload file (note that file is in singular form because it is only one per photo):
POST: /projects/{project_id}/photos/{photo_id}/file
body: file to upload
response: -
And then for instance:
Read photos list
GET: /projects/{project_id}/photos
response: [ photo, photo, photo, ... ] (array of objects)
Read some photo details
GET: /projects/{project_id}/photos/{photo_id}
response: { id: 666, name: 'some_schema.jpg', comment:'blah'} (photo object)
Read photo file
GET: /projects/{project_id}/photos/{photo_id}/file
response: file content
So the conclusion is that, first you create an object (photo) by POST, and then you send second request with the file (again POST). To not have problems with CACHE in this approach we assume that we can only delete old photos and add new - no update binary photo files (because new binary file is in fact... NEW photo). However if you need to be able to update binary files and cache them, then in point 4 return also fileId and change 5 to GET: /projects/{project_id}/photos/{photo_id}/files/{fileId}.
I know this question is old, but in the last days I had searched whole web to solution this same question. I have grails REST webservices and iPhone Client that send pictures, title and description.
I don't know if my approach is the best, but is so easy and simple.
I take a picture using the UIImagePickerController and send to server the NSData using the header tags of request to send the picture's data.
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:[NSURL URLWithString:#"myServerAddress"]];
[request setHTTPMethod:#"POST"];
[request setHTTPBody:UIImageJPEGRepresentation(picture, 0.5)];
[request setValue:#"image/jpeg" forHTTPHeaderField:#"Content-Type"];
[request setValue:#"myPhotoTitle" forHTTPHeaderField:#"Photo-Title"];
[request setValue:#"myPhotoDescription" forHTTPHeaderField:#"Photo-Description"];
NSURLResponse *response;
NSError *error;
[NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];
At the server side, I receive the photo using the code:
InputStream is = request.inputStream
def receivedPhotoFile = (IOUtils.toByteArray(is))
def photo = new Photo()
photo.photoFile = receivedPhotoFile //photoFile is a transient attribute
photo.title = request.getHeader("Photo-Title")
photo.description = request.getHeader("Photo-Description")
photo.imageURL = "temp"
if (photo.save()) {
File saveLocation = grailsAttributes.getApplicationContext().getResource(File.separator + "images").getFile()
saveLocation.mkdirs()
File tempFile = File.createTempFile("photo", ".jpg", saveLocation)
photo.imageURL = saveLocation.getName() + "/" + tempFile.getName()
tempFile.append(photo.photoFile);
} else {
println("Error")
}
I don't know if I have problems in future, but now is working fine in production environment.
FormData Objects: Upload Files Using Ajax
XMLHttpRequest Level 2 adds support for the new FormData interface.
FormData objects provide a way to easily construct a set of key/value pairs representing form fields and their values, which can then be easily sent using the XMLHttpRequest send() method.
function AjaxFileUpload() {
var file = document.getElementById("files");
//var file = fileInput;
var fd = new FormData();
fd.append("imageFileData", file);
var xhr = new XMLHttpRequest();
xhr.open("POST", '/ws/fileUpload.do');
xhr.onreadystatechange = function () {
if (xhr.readyState == 4) {
alert('success');
}
else if (uploadResult == 'success')
alert('error');
};
xhr.send(fd);
}
https://developer.mozilla.org/en-US/docs/Web/API/FormData
Since the only missing example is the ANDROID example, I'll add it.
This technique uses a custom AsyncTask that should be declared inside your Activity class.
private class UploadFile extends AsyncTask<Void, Integer, String> {
#Override
protected void onPreExecute() {
// set a status bar or show a dialog to the user here
super.onPreExecute();
}
#Override
protected void onProgressUpdate(Integer... progress) {
// progress[0] is the current status (e.g. 10%)
// here you can update the user interface with the current status
}
#Override
protected String doInBackground(Void... params) {
return uploadFile();
}
private String uploadFile() {
String responseString = null;
HttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost("http://example.com/upload-file");
try {
AndroidMultiPartEntity ampEntity = new AndroidMultiPartEntity(
new ProgressListener() {
#Override
public void transferred(long num) {
// this trigger the progressUpdate event
publishProgress((int) ((num / (float) totalSize) * 100));
}
});
File myFile = new File("/my/image/path/example.jpg");
ampEntity.addPart("fileFieldName", new FileBody(myFile));
totalSize = ampEntity.getContentLength();
httpPost.setEntity(ampEntity);
// Making server call
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity httpEntity = httpResponse.getEntity();
int statusCode = httpResponse.getStatusLine().getStatusCode();
if (statusCode == 200) {
responseString = EntityUtils.toString(httpEntity);
} else {
responseString = "Error, http status: "
+ statusCode;
}
} catch (Exception e) {
responseString = e.getMessage();
}
return responseString;
}
#Override
protected void onPostExecute(String result) {
// if you want update the user interface with upload result
super.onPostExecute(result);
}
}
So, when you want to upload your file just call:
new UploadFile().execute();
I wanted send some strings to backend server. I didnt use json with multipart, I have used request params.
#RequestMapping(value = "/upload", method = RequestMethod.POST)
public void uploadFile(HttpServletRequest request,
HttpServletResponse response, #RequestParam("uuid") String uuid,
#RequestParam("type") DocType type,
#RequestParam("file") MultipartFile uploadfile)
Url would look like
http://localhost:8080/file/upload?uuid=46f073d0&type=PASSPORT
I am passing two params (uuid and type) along with file upload.
Hope this will help who don't have the complex json data to send.
You could try using https://square.github.io/okhttp/ library.
You can set the request body to multipart and then add the file and json objects separately like so:
MultipartBody requestBody = new MultipartBody.Builder()
.setType(MultipartBody.FORM)
.addFormDataPart("uploadFile", uploadFile.getName(), okhttp3.RequestBody.create(uploadFile, MediaType.parse("image/png")))
.addFormDataPart("file metadata", json)
.build();
Request request = new Request.Builder()
.url("https://uploadurl.com/uploadFile")
.post(requestBody)
.build();
try (Response response = client.newCall(request).execute()) {
if (!response.isSuccessful()) throw new IOException("Unexpected code " + response);
logger.info(response.body().string());
#RequestMapping(value = "/uploadImageJson", method = RequestMethod.POST)
public #ResponseBody Object jsongStrImage(#RequestParam(value="image") MultipartFile image, #RequestParam String jsonStr) {
-- use com.fasterxml.jackson.databind.ObjectMapper convert Json String to Object
}
Please ensure that you have following import. Ofcourse other standard imports
import org.springframework.core.io.FileSystemResource
void uploadzipFiles(String token) {
RestBuilder rest = new RestBuilder(connectTimeout:10000, readTimeout:20000)
def zipFile = new File("testdata.zip")
def Id = "001G00000"
MultiValueMap<String, String> form = new LinkedMultiValueMap<String, String>()
form.add("id", id)
form.add('file',new FileSystemResource(zipFile))
def urld ='''http://URL''';
def resp = rest.post(urld) {
header('X-Auth-Token', clientSecret)
contentType "multipart/form-data"
body(form)
}
println "resp::"+resp
println "resp::"+resp.text
println "resp::"+resp.headers
println "resp::"+resp.body
println "resp::"+resp.status
}