So basically, I have a HttpClient that attempts to obtain any form of JSON data from an endpoint. I previously utilized Newtonsoft.Json to achieve this easily but after migrating all of the functions to STJ, I started to notice improper parsing.
Platforms tested: macOS & Linux (Google Kubernetes Engine)
Framework: .NET Core 3.1 LTS
The code screenshots below show an API that returns a JSON Array. I simply stream it, load it into a JsonDocument, and then attempt to peek into it. Nothing comes out as expected. Code below is provided along with the step debug var results.
using System;
using System.ComponentModel;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
using System.Web;
using System.Xml;
namespace HttpCallDemo
{
class Program
{
static async Task Main(string[] args)
{
using (var httpClient = new HttpClient())
{
// FLUSH
httpClient.DefaultRequestHeaders.Clear();
httpClient.MaxResponseContentBufferSize = 4096;
string body = string.Empty, customMediaType = string.Empty; // For POST/PUT
// Setup the url
var uri = new UriBuilder("https://api-pub.bitfinex.com/v2/tickers?symbols=ALL");
uri.Port = -1;
// Pull in the payload
var requestPayload = new HttpRequestMessage(HttpMethod.Get, uri.ToString());
HttpResponseMessage responsePayload;
responsePayload = await httpClient.SendAsync(requestPayload,
HttpCompletionOption.ResponseHeadersRead);
var byteArr = await responsePayload.Content.ReadAsByteArrayAsync();
if (byteArr.LongCount() > 4194304) // 4MB
return; // Too big.
// Pull the content
var contentFromBytes = Encoding.Default.GetString(byteArr);
JsonDocument payload;
switch (responsePayload.StatusCode)
{
case HttpStatusCode.OK:
// Return the payload distinctively
payload = JsonDocument.Parse(contentFromBytes);
#if DEBUG
var testJsonRes = Encoding.UTF8.GetString(
Utf8Json.JsonSerializer.Serialize(payload.RootElement));
// var testRawRes = contentStream.read
var testJsonResEl = payload.RootElement.GetRawText();
#endif
break;
default:
throw new InvalidDataException("Invalid HTTP response.");
}
}
}
}
}
Simply execute the above Minimal code, notice that the payload is different from its original after parsing? I'm sure there's something wrong with the options for STJ. Seems like we have to optimise or explicitly define its limits to allow it to process that JSON payload.
Diving deeper into the debug content made things even weirder. When the HttpClient obtains the payload, reads it to a string, it gives me the entire JSON string as is. However, once we attempt to parse it into a JsonDocument and the further invoking RootElement.Clone(), we'll end up with a JsonElement with much lesser data and while carrying an invalid JSON struct (Below).
ValueKind = Array : "[["tBTCUSD",11418,70.31212518,11419,161.93475693,258.02141213,0.0231,11418,2980.0289306,11438,11003],["tLTCUSD",58.919,2236.00823543,58.95,2884.6718013699997,1.258,0.0218,58.998,63147.48344762,59.261,56.334],["tLTCBTC",0.0051609,962.80334198,0.005166,1170.07399991,-0.000012,-0.0023,0.0051609,4178.13148459,0.0051852,0.0051],["tETHUSD",396.54,336.52151165,396.55,384.37623341,8.26964946,0.0213,396.50930256,69499.5382821,397.77,380.5],["tETHBTC",0.034731,166.67781664000003,0.034751,356.03450125999996,-0.000054,-0.0016,0.034747,5855.04978836,0.035109,0.0343],["tETCBTC",0.00063087,15536.813429530002,0.00063197,16238.600279749999,-0.00000838,-0.0131,0.00063085,73137.62192801,0.00064135,0.00062819],["tETCUSD",7.2059,9527.40221867,7.2176,8805.54677899,0.0517,0.0072,7.2203,49618.78868196,7.2263,7],["tRRTUSD",0.057476,33577.52064154,0.058614,20946.501210000002,0.023114,0.6511,0.058614,210741.23592011,0.06443,0.0355],["tZECUSD",88.131,821.28048322,88.332,880.37484662,5.925,0.0
And of course, attempting to read its contents would result in:
System.InvalidOperationException: Operation is not valid due to the current state of the object.
at System.Text.Json.JsonElement.get_Item(Int32 index)
at Nozomi.Preprocessing.Abstracts.BaseProcessingService`1.ProcessIdentifier(JsonElement jsonDoc, String identifier) in /Users/nicholaschen/Projects/nozomi/Nozomi.Infra.Preprocessing/Abstracts/BaseProcessingService.cs:line 255
Here's proof that there is a proper 38KBs worth of data coming in from the endpoint.
UPDATE
Further testing with this
if (payload.RootElement.ValueKind.Equals(JsonValueKind.Array))
{
string testJsonArr;
testJsonArr = Encoding.UTF8.GetString(
Utf8Json.JsonSerializer.Serialize(
payload.RootElement.EnumerateArray()));
}
show that a larger array of arrays (exceeding 9 elements each with 11 elements) would result in an incomplete JSON struct, causing the issue i'm facing.
For those who are working with JsonDocument and JsonElement, take note that the step debug variables are not accurate. It is not advisable to inspect the variables during runtime as they do not display themselves entirely.
#dbc has proven that re-serializing the deserialized data will produce the complete dataset. I strongly suggest you wrap the serializers for debugging in a DEBUG preprocessor to make sure these redundant lines don't end up being executed out of development.
To interact with these entities, ensure you .clone() whenever you can to prevent disposals and ensure that you're accessing the RootElement and then subsequently traversing into it before viewing its value in step debug mode because large values will not be displayed.
I wanted to generate a test program to execute against our client tenants to verify we could handle all the data our new Microsoft graph app collects. My plan was to serialize the data using
XmlSerializer serializer = new XmlSerializer(typeof(List<T>));
It failed on the first entity I tried, Microsoft.Graph.Domain ( in this case with the error
Cannot serialize member Microsoft.Graph.Entity.AdditionalData of type ... because it is an interface.
A search on stack overflow found suggestions to decorate the problematic class property with XmlIgnore so XmlSerializer will ignore it, others recommended implementing a new IXmlSerializer. One post seemed to propose using serializing to XAML.
Open to a better way to collect real customer data which I can import into my unit tests? As a developer I do not have direct access to customer accounts.
Does anyone have other suggestions on how to serialize Microsoft Graph Entities.
I replaced my XmlSerializer with a Json one.
public void SerializeObjectsToJson<T>(List<T> serializableObjects)
{
var jsonStr = JsonConvert.SerializeObject(serializableObjects);
}
public List<T> DeSerializeObjectsFromXml<T>()
{
TextReader textReader = new StreamReader(fqpathname, Encoding.UTF8);
var jsonStr = textReader.ReadToEnd();
data = JsonConvert.DeserializeObject<List<T>>(jsonStr);
}
This all seems to work with Domain, User, SubscribedSkus, Organization, etc.
I want to serialize data on my .net Standard app into a local file and I would like to avoid sqlite if possible.
The standard recommendation for cross plattform app seems to have been PCL Storage, but according to this link, PCL Storage is not maintained anymore, offers no .net Standard support and the alternative PCLExt is not mature.
Can you tell me if it is possible to simply serialize my data e.g. with json?
Tank you very much!
You do not have complete access over the OS's filesystem and platform-specific features like Android's ContentResolver, but for basic file read/write within your app's sandbox (or external filesystem if your app has access to it) .NetStandard 2.0 works fine, and thus works for storing and retrieving text-based files for serializing/deserializing Json.
Example, if you have a Xamarin.Forms based solution and add a .NetStandard 2.0 library project to the solution and also add Newtonsoft.Json to it. You could create these functions in it:
.NetStandard library functions:
public static class Common
{
public static void WriteFile(string fileName, Tuple<string, string> obj)
{
var path = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
File.WriteAllText(Path.Combine(path, fileName), JsonConvert.SerializeObject(obj));
}
public static Tuple<string, string> ReadFile(string fileName)
{
var path = Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
return JsonConvert.DeserializeObject<Tuple<string, string>>(File.ReadAllText(Path.Combine(path, fileName));
}
}
Now in your Xamarin.Forms project (.NetStandard * Shared project), reference the project/library you created you could do something like this:
ReadWriteCommand = new Command(() =>
{
var someObj = Tuple.Create("Stack", "Overflow");
Common.WriteFile("SushiHangover.txt", someObj);
var readSomeObj = Common.ReadFile("SushiHangover.txt");
if ((someObj.Item1 != readSomeObj.Item1) || (someObj.Item2 != readSomeObj.Item2))
throw new Exception();
});
In a Windows Store app I can only store WinRT types in the ApplicationSettings, according to the documentation. For roamed settings that should be held together I can use ApplicationDataCompositeValue. Trying to store an instance of an own class or struct results in an Exception with the message " WinRT information: Error trying to serialize the value to be written to the application data store. Additional Information: Data of this type is not supported". The term "trying to serialize" indicates that there must be some way so serialize a type for the application data API.
Does anyone know how I could achieve that?
I tried DataContract serialization but it did not work.
I think custom/own types are not supported.
See http://msdn.microsoft.com/en-us/library/windows/apps/hh464917.aspx:
"The Windows Runtime data types are supported for app settings."
But you can serialize your objects to XML and save as string... (see code below)
public static string Serialize(object obj)
{
using (var sw = new StringWriter())
{
var serializer = new XmlSerializer(obj.GetType());
serializer.Serialize(sw, obj);
return sw.ToString();
}
}
public static T Deserialize<T>(string xml)
{
using (var sw = new StringReader(xml))
{
var serializer = new XmlSerializer(typeof(T));
return (T)serializer.Deserialize(sw);
}
}
https://github.com/MyToolkit/MyToolkit/blob/master/src/MyToolkit/Serialization/XmlSerialization.cs
Check out this class too:
https://github.com/MyToolkit/MyToolkit/wiki/XmlSerialization
Disclaimer: The above links are from my project
We're designing report with multilingual interface. We have developed web service which return translation of specific words. Is there any way to call the translation of labels used in report through web service or specific URL.
For e.g. something like
http://domain.com/translate?w=WORD-TO-Translate&L=ar
I was going to recommend creating a custom assembly that would do this, but while it works from Report Builder it doesn't seem to work from my SSRS server. I'm wondering if there is an issue connecting to a web service from a custom assembly (or maybe I'm doing it wrong). Instead, I'll point you to another method for doing translations.
In case you want to pick up the custom assembly approach, here's the code I'm using:
using System;
using System.IO;
using System.Net;
namespace SSRSCustomAssembly
{
public class Translate
{
public static string TranslateString(string input, string locale)
{
string url = string.Format("http://domain.com/translate?w={0}&L={1}", input, locale);
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Method = "GET";
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
return sr.ReadToEnd();
}
}
}
In your report, just add a reference to the assembly, the call it by having an expression:
=SSRSCustomAssembly.Translate.TranslateString("word", "en")