So basically, I have a HttpClient that attempts to obtain any form of JSON data from an endpoint. I previously utilized Newtonsoft.Json to achieve this easily but after migrating all of the functions to STJ, I started to notice improper parsing.
Platforms tested: macOS & Linux (Google Kubernetes Engine)
Framework: .NET Core 3.1 LTS
The code screenshots below show an API that returns a JSON Array. I simply stream it, load it into a JsonDocument, and then attempt to peek into it. Nothing comes out as expected. Code below is provided along with the step debug var results.
using System;
using System.ComponentModel;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
using System.Web;
using System.Xml;
namespace HttpCallDemo
{
class Program
{
static async Task Main(string[] args)
{
using (var httpClient = new HttpClient())
{
// FLUSH
httpClient.DefaultRequestHeaders.Clear();
httpClient.MaxResponseContentBufferSize = 4096;
string body = string.Empty, customMediaType = string.Empty; // For POST/PUT
// Setup the url
var uri = new UriBuilder("https://api-pub.bitfinex.com/v2/tickers?symbols=ALL");
uri.Port = -1;
// Pull in the payload
var requestPayload = new HttpRequestMessage(HttpMethod.Get, uri.ToString());
HttpResponseMessage responsePayload;
responsePayload = await httpClient.SendAsync(requestPayload,
HttpCompletionOption.ResponseHeadersRead);
var byteArr = await responsePayload.Content.ReadAsByteArrayAsync();
if (byteArr.LongCount() > 4194304) // 4MB
return; // Too big.
// Pull the content
var contentFromBytes = Encoding.Default.GetString(byteArr);
JsonDocument payload;
switch (responsePayload.StatusCode)
{
case HttpStatusCode.OK:
// Return the payload distinctively
payload = JsonDocument.Parse(contentFromBytes);
#if DEBUG
var testJsonRes = Encoding.UTF8.GetString(
Utf8Json.JsonSerializer.Serialize(payload.RootElement));
// var testRawRes = contentStream.read
var testJsonResEl = payload.RootElement.GetRawText();
#endif
break;
default:
throw new InvalidDataException("Invalid HTTP response.");
}
}
}
}
}
Simply execute the above Minimal code, notice that the payload is different from its original after parsing? I'm sure there's something wrong with the options for STJ. Seems like we have to optimise or explicitly define its limits to allow it to process that JSON payload.
Diving deeper into the debug content made things even weirder. When the HttpClient obtains the payload, reads it to a string, it gives me the entire JSON string as is. However, once we attempt to parse it into a JsonDocument and the further invoking RootElement.Clone(), we'll end up with a JsonElement with much lesser data and while carrying an invalid JSON struct (Below).
ValueKind = Array : "[["tBTCUSD",11418,70.31212518,11419,161.93475693,258.02141213,0.0231,11418,2980.0289306,11438,11003],["tLTCUSD",58.919,2236.00823543,58.95,2884.6718013699997,1.258,0.0218,58.998,63147.48344762,59.261,56.334],["tLTCBTC",0.0051609,962.80334198,0.005166,1170.07399991,-0.000012,-0.0023,0.0051609,4178.13148459,0.0051852,0.0051],["tETHUSD",396.54,336.52151165,396.55,384.37623341,8.26964946,0.0213,396.50930256,69499.5382821,397.77,380.5],["tETHBTC",0.034731,166.67781664000003,0.034751,356.03450125999996,-0.000054,-0.0016,0.034747,5855.04978836,0.035109,0.0343],["tETCBTC",0.00063087,15536.813429530002,0.00063197,16238.600279749999,-0.00000838,-0.0131,0.00063085,73137.62192801,0.00064135,0.00062819],["tETCUSD",7.2059,9527.40221867,7.2176,8805.54677899,0.0517,0.0072,7.2203,49618.78868196,7.2263,7],["tRRTUSD",0.057476,33577.52064154,0.058614,20946.501210000002,0.023114,0.6511,0.058614,210741.23592011,0.06443,0.0355],["tZECUSD",88.131,821.28048322,88.332,880.37484662,5.925,0.0
And of course, attempting to read its contents would result in:
System.InvalidOperationException: Operation is not valid due to the current state of the object.
at System.Text.Json.JsonElement.get_Item(Int32 index)
at Nozomi.Preprocessing.Abstracts.BaseProcessingService`1.ProcessIdentifier(JsonElement jsonDoc, String identifier) in /Users/nicholaschen/Projects/nozomi/Nozomi.Infra.Preprocessing/Abstracts/BaseProcessingService.cs:line 255
Here's proof that there is a proper 38KBs worth of data coming in from the endpoint.
UPDATE
Further testing with this
if (payload.RootElement.ValueKind.Equals(JsonValueKind.Array))
{
string testJsonArr;
testJsonArr = Encoding.UTF8.GetString(
Utf8Json.JsonSerializer.Serialize(
payload.RootElement.EnumerateArray()));
}
show that a larger array of arrays (exceeding 9 elements each with 11 elements) would result in an incomplete JSON struct, causing the issue i'm facing.
For those who are working with JsonDocument and JsonElement, take note that the step debug variables are not accurate. It is not advisable to inspect the variables during runtime as they do not display themselves entirely.
#dbc has proven that re-serializing the deserialized data will produce the complete dataset. I strongly suggest you wrap the serializers for debugging in a DEBUG preprocessor to make sure these redundant lines don't end up being executed out of development.
To interact with these entities, ensure you .clone() whenever you can to prevent disposals and ensure that you're accessing the RootElement and then subsequently traversing into it before viewing its value in step debug mode because large values will not be displayed.
I am querying the Wikipedia API and am getting JSON back that looks like this:
https://en.wikipedia.org/w/api.php?action=query&prop=pageimages&titles=cessna%20172&pithumbsize=500&format=json
{"batchcomplete":"","query":{"normalized":[{"from":"cessna 172","to":"Cessna 172"}],"pages":{"173462":{"pageid":173462,"ns":0,"title":"Cessna 172","thumbnail":{"source":"https://upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Cessna_172S_Skyhawk_SP%2C_Private_JP6817606.jpg/500px-Cessna_172S_Skyhawk_SP%2C_Private_JP6817606.jpg","width":500,"height":333},"pageimage":"Cessna_172S_Skyhawk_SP,_Private_JP6817606.jpg"}}}}
Using .Net Core 2.2, what is the proper way to get the image thumbnail out of this (the source property in this case)?
Parsing JSON is not a built in feature in .Net core 2.2 so you will want to add the Newtonsoft.Json package to the project with dotnet add package Newtonsoft.Json --version 12.0.3.
From there include Newtonsoft.Json by adding using Newtonsoft.Json.Linq; to the top of the file. and using System.Net; to use WebClient.
From there the code retrieves the string from the url. JObject.Parse parses the string as a JObject. We can get the property you want by chaining indexers: ["query"]["pages"]["173462"]["thumbnail"]["source"].
Full source:
using System;
using System.Net;
using Newtonsoft.Json.Linq;
class Program
{
static void Main(string[] args)
{
const string url = "https://en.wikipedia.org/w/api.php?action=query&prop=pageimages&titles=cessna%20172&pithumbsize=500&format=json";
using (WebClient client = new WebClient())
{
string rawString = client.DownloadString(url);
var jsonResult = JObject.Parse(rawString);
string thumbnail = jsonResult["query"]["pages"]["173462"]["thumbnail"]["source"];
Console.WriteLine(thumbnail);
}
}
}
Ideally, you will have to define a class and de-serialised the json. Example :
Batch batch = JsonConvert.DeserializeObject<Account>(json);
More details here.
However, at times, just to get one/two values, it might be overhead to use an entire class structure. In this case, a quick way might be to parse the json dynamically. Example which is taken from here:
public void JValueParsingTest()
{
var jsonString = #"{""Name"":""Rick"",""Company"":""West Wind"",
""Entered"":""2012-03-16T00:03:33.245-10:00""}";
dynamic json = JValue.Parse(jsonString);
// values require casting
string name = json.Name;
string company = json.Company;
DateTime entered = json.Entered;
Assert.AreEqual(name, "Rick");
Assert.AreEqual(company, "West Wind");
}
I am looking for existing solutions to match dynamic parameters with HttpCore. What I have in mind is something similar to constraints in ruby on rails, or dynamic parameters with sails (see here for example).
My objective is to define a REST API where I could easily match requests like GET /objects/<object_id>.
To give a little bit of context, I have an application that creates an HttpServer using the following code
server = ServerBootstrap.bootstrap()
.setListenerPort(port)
.setServerInfo("MyAppServer/1.1")
.setSocketConfig(socketConfig)
.registerHandler("*", new HttpHandler(this))
.create();
And the HttpHandler class that matches the requested URI and dispatches it to the corresponding backend method:
public void handle(final HttpRequest request, final HttpResponse response, final HttpContext context) {
String method = request.getRequestLine().getMethod().toUpperCase(Locale.ROOT);
// Parameters are ignored for the example
String path = request.getRequestLine().getUri();
if(method.equals("POST") && path.equals("/object/add") {
if(request instanceof HttpEntityEnclosingRequest) {
addObject(((HttpEntityEnclosingRequest)request).getEntity())
}
[...]
For sure I can replace path.equals("/object/add") by something more sophisticated with RegEx to match these dynamic parameters, but before doing so I'd like to know if I am not reinventing the wheel, or if there is an existing lib/class I didn't see in the docs that could help me.
Using HttpCore is a requirement (it is already integrated in the application I am working on), I know some other libraries provide high-level routing mechanisms that support these dynamic parameters, but I can't really afford switching the entire server code to another library.
I am currently using httpcore 4.4.10, but I can upgrade to a newer version of this might help me.
At present HttpCore does not have a fully featured request routing layer. (The reasons for that are more political than technical).
Consider using a custom HttpRequestHandlerMapper to implement your application specific request routing logic.
final HttpServer server = ServerBootstrap.bootstrap()
.setListenerPort(port)
.setServerInfo("Test/1.1")
.setSocketConfig(socketConfig)
.setSslContext(sslContext)
.setHandlerMapper(new HttpRequestHandlerMapper() {
#Override
public HttpRequestHandler lookup(HttpRequest request) {
try {
URI uri = new URI(request.getRequestLine().getUri());
String path = uri.getPath();
// do request routing based on the request path
return new HttpFileHandler(docRoot);
} catch (URISyntaxException e) {
// Provide a more reasonable error handler here
return null;
}
}
})
.setExceptionLogger(new StdErrorExceptionLogger())
.create();
I've recently moved from MVC5 over to .NET Core 2.1 (MVC). Can anyone help me with this please.
I have my ApplicationUser and I've extended the model/table to store the user's FirstName.
In the View, I want to be able to output the current user firstname value.
User in the view is a ClaimsPrincipal so I need to go off to the DB to grab the value I need or access UserManager to get it.
Now, I know I can get that in the controller but I don't want to have to create a JQuery call to grab it every time I need it.
What I do want is to be able to access it server side, ideally via a static helper class.
In the MVC5 I'd have a helper to do the job no problem. Something like this for example:
public static string GetCurrentUserFirstName()
{
string _usrRef = HttpContext.Current.User.Identity.GetUserId();
var user = HttpContext.Current.GetOwinContext().GetUserManager<ApplicationUserManager>().FindById(_usrRef);
return user.FirstName;
}
However, .NET Core doesn't work that way.
In a controller I could say:
var user = await _userManager.GetUserAsync(User);
string firstName = user.FirstName;
or I could go off to the DB via a call using Dapper w/ my connection string.
I can't inject the UserManager or ConnectionStrings into the helper via the constructor as it is static.
Is there a way to access either of those in this static helper?
It's the little changes that get you the most!
Thanks to #Kirk Larkin - I've found the solution.
I have to admit, it feels a little more convoluted having to pass things around to gain access to them but this is a good, working solution.
The View:
#using Microsoft.AspNetCore.Identity
#using MyApplication.Helpers
#inject UserManager<ApplicationUser> UserManager
<div>
#await MyHelper.GetLoggedInUserFirstName(UserManager, User)
</div>
The MyHelper file:
public static async Task<string> GetLoggedInUserFirstName(UserManager<ApplicationUser> userManager, ClaimsPrincipal user)
{
string output = "";
try
{
var currentUser = await userManager.GetUserAsync(user);
if(currentUser!=null)
{
output = currentUser.FirstName ?? currentUser.Email;
}
}
catch(Exception e) { }
return output;
}
I am getting a 415 Error when sending a form entry to another client resource via JSON. The target URI in my code below ("/message") works when not using the form (i.e. hit "/message" with a test mock object).
Here is my code to get the values of the form and do the post to the target resource. Am I missing something that needs to be done?
I am using the following:
Restlet: 2.1 RC5
GAE: 1.6.1
Form Restlet:
#Post
public void handlePost(Representation entity) {
final Form webForm = new Form(entity);
MessageEntity newMessage = new MessageEntity();
String subject = webForm.getFirstValue("subject");
String sendto = webForm.getFirstValue("email");
String message = webForm.getFirstValue("message");
newMessage.setCategoryID(subject);
newMessage.setAccountID(sendto);
newMessage.setMessageText(message);
ClientResource cr = new ClientResource(getRootRef()+ "/message");
cr.post(newMessage, MediaType.APPLICATION_JSON);
}
Target Resource ("/message")
#Post("json")
public void HandleRequest(MessageEntity messageEntity) {
// Logic here
}
Please let me know if you need more information
Thanks!
I have code that is very similar to yours that works fine. I am also running similar versions of Restlet and GAE. First question I have is are there other #Post methods in your Target Resource as sometimes the ordering matters.
Here are two versions of code that I have that work....
1)
public Representation postHandler() {
Reference commitsRef = new Reference(Consts.RESOURCE_BASE + "commitments/");
ClientResource commitsResource = new ClientResource(getContext(), commitsRef);
....
Representation commitsRep = commitsResource.post(commitForm);
That is posting a form to a Target resource that handles both #Post("json") and #Post("form")
2)
public Representation doPostFromGet() {
Reference takeActRef = new Reference(Consts.RESOURCE_BASE + "commitment/"
+ commitmentId + "/userActs/");
ClientResource takeActResource = new ClientResource(getContext(), takeActRef);
...
Representation takeActRep = takeActResource.post(newAct);
That is posting a Java object to a form that uses what I call the "Peierls magic". See:
http://tembrel.blogspot.com/2012/03/converting-forms-in-restlet-to-pojos.html
It allows you to have one post() in the Target and accept both forms and pojos.
On a minor note, if you are doing a post to add a new message, should the url be "/messages/" (plural) - and perhaps there is a typo somewhere? (An unlikely possibility, but I thought I would mention it).
Good luck,
RB