So basically, I have a HttpClient that attempts to obtain any form of JSON data from an endpoint. I previously utilized Newtonsoft.Json to achieve this easily but after migrating all of the functions to STJ, I started to notice improper parsing.
Platforms tested: macOS & Linux (Google Kubernetes Engine)
Framework: .NET Core 3.1 LTS
The code screenshots below show an API that returns a JSON Array. I simply stream it, load it into a JsonDocument, and then attempt to peek into it. Nothing comes out as expected. Code below is provided along with the step debug var results.
using System;
using System.ComponentModel;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
using System.Web;
using System.Xml;
namespace HttpCallDemo
{
class Program
{
static async Task Main(string[] args)
{
using (var httpClient = new HttpClient())
{
// FLUSH
httpClient.DefaultRequestHeaders.Clear();
httpClient.MaxResponseContentBufferSize = 4096;
string body = string.Empty, customMediaType = string.Empty; // For POST/PUT
// Setup the url
var uri = new UriBuilder("https://api-pub.bitfinex.com/v2/tickers?symbols=ALL");
uri.Port = -1;
// Pull in the payload
var requestPayload = new HttpRequestMessage(HttpMethod.Get, uri.ToString());
HttpResponseMessage responsePayload;
responsePayload = await httpClient.SendAsync(requestPayload,
HttpCompletionOption.ResponseHeadersRead);
var byteArr = await responsePayload.Content.ReadAsByteArrayAsync();
if (byteArr.LongCount() > 4194304) // 4MB
return; // Too big.
// Pull the content
var contentFromBytes = Encoding.Default.GetString(byteArr);
JsonDocument payload;
switch (responsePayload.StatusCode)
{
case HttpStatusCode.OK:
// Return the payload distinctively
payload = JsonDocument.Parse(contentFromBytes);
#if DEBUG
var testJsonRes = Encoding.UTF8.GetString(
Utf8Json.JsonSerializer.Serialize(payload.RootElement));
// var testRawRes = contentStream.read
var testJsonResEl = payload.RootElement.GetRawText();
#endif
break;
default:
throw new InvalidDataException("Invalid HTTP response.");
}
}
}
}
}
Simply execute the above Minimal code, notice that the payload is different from its original after parsing? I'm sure there's something wrong with the options for STJ. Seems like we have to optimise or explicitly define its limits to allow it to process that JSON payload.
Diving deeper into the debug content made things even weirder. When the HttpClient obtains the payload, reads it to a string, it gives me the entire JSON string as is. However, once we attempt to parse it into a JsonDocument and the further invoking RootElement.Clone(), we'll end up with a JsonElement with much lesser data and while carrying an invalid JSON struct (Below).
ValueKind = Array : "[["tBTCUSD",11418,70.31212518,11419,161.93475693,258.02141213,0.0231,11418,2980.0289306,11438,11003],["tLTCUSD",58.919,2236.00823543,58.95,2884.6718013699997,1.258,0.0218,58.998,63147.48344762,59.261,56.334],["tLTCBTC",0.0051609,962.80334198,0.005166,1170.07399991,-0.000012,-0.0023,0.0051609,4178.13148459,0.0051852,0.0051],["tETHUSD",396.54,336.52151165,396.55,384.37623341,8.26964946,0.0213,396.50930256,69499.5382821,397.77,380.5],["tETHBTC",0.034731,166.67781664000003,0.034751,356.03450125999996,-0.000054,-0.0016,0.034747,5855.04978836,0.035109,0.0343],["tETCBTC",0.00063087,15536.813429530002,0.00063197,16238.600279749999,-0.00000838,-0.0131,0.00063085,73137.62192801,0.00064135,0.00062819],["tETCUSD",7.2059,9527.40221867,7.2176,8805.54677899,0.0517,0.0072,7.2203,49618.78868196,7.2263,7],["tRRTUSD",0.057476,33577.52064154,0.058614,20946.501210000002,0.023114,0.6511,0.058614,210741.23592011,0.06443,0.0355],["tZECUSD",88.131,821.28048322,88.332,880.37484662,5.925,0.0
And of course, attempting to read its contents would result in:
System.InvalidOperationException: Operation is not valid due to the current state of the object.
at System.Text.Json.JsonElement.get_Item(Int32 index)
at Nozomi.Preprocessing.Abstracts.BaseProcessingService`1.ProcessIdentifier(JsonElement jsonDoc, String identifier) in /Users/nicholaschen/Projects/nozomi/Nozomi.Infra.Preprocessing/Abstracts/BaseProcessingService.cs:line 255
Here's proof that there is a proper 38KBs worth of data coming in from the endpoint.
UPDATE
Further testing with this
if (payload.RootElement.ValueKind.Equals(JsonValueKind.Array))
{
string testJsonArr;
testJsonArr = Encoding.UTF8.GetString(
Utf8Json.JsonSerializer.Serialize(
payload.RootElement.EnumerateArray()));
}
show that a larger array of arrays (exceeding 9 elements each with 11 elements) would result in an incomplete JSON struct, causing the issue i'm facing.
For those who are working with JsonDocument and JsonElement, take note that the step debug variables are not accurate. It is not advisable to inspect the variables during runtime as they do not display themselves entirely.
#dbc has proven that re-serializing the deserialized data will produce the complete dataset. I strongly suggest you wrap the serializers for debugging in a DEBUG preprocessor to make sure these redundant lines don't end up being executed out of development.
To interact with these entities, ensure you .clone() whenever you can to prevent disposals and ensure that you're accessing the RootElement and then subsequently traversing into it before viewing its value in step debug mode because large values will not be displayed.
I wanted to generate a test program to execute against our client tenants to verify we could handle all the data our new Microsoft graph app collects. My plan was to serialize the data using
XmlSerializer serializer = new XmlSerializer(typeof(List<T>));
It failed on the first entity I tried, Microsoft.Graph.Domain ( in this case with the error
Cannot serialize member Microsoft.Graph.Entity.AdditionalData of type ... because it is an interface.
A search on stack overflow found suggestions to decorate the problematic class property with XmlIgnore so XmlSerializer will ignore it, others recommended implementing a new IXmlSerializer. One post seemed to propose using serializing to XAML.
Open to a better way to collect real customer data which I can import into my unit tests? As a developer I do not have direct access to customer accounts.
Does anyone have other suggestions on how to serialize Microsoft Graph Entities.
I replaced my XmlSerializer with a Json one.
public void SerializeObjectsToJson<T>(List<T> serializableObjects)
{
var jsonStr = JsonConvert.SerializeObject(serializableObjects);
}
public List<T> DeSerializeObjectsFromXml<T>()
{
TextReader textReader = new StreamReader(fqpathname, Encoding.UTF8);
var jsonStr = textReader.ReadToEnd();
data = JsonConvert.DeserializeObject<List<T>>(jsonStr);
}
This all seems to work with Domain, User, SubscribedSkus, Organization, etc.
I am doing a development to integrate a non Java OMS system with QuickFIX/J to send buy/sell orders to multiple brokerage systems .
I have written the belog logic to send the messages
I have written this under main function which is in the same class created by implementing Application "public class Initiator implements Application"
InputStream inp = InitiatorSocket.class.getResourceAsStream("test.cfg");
SessionSettings sessionSetting = new SessionSettings(inp);
Application myApp = new Initiator();
FileStoreFactory factory = new FileStoreFactory(sessionSetting);
ScreenLogFactory sfactory = new ScreenLogFactory(sessionSetting);
DefaultMessageFactory defaultMsgFactory = new DefaultMessageFactory();
initiator = new SocketInitiator(myApp, factory, sessionSetting,sfactory,defaultMsgFactory);
initiator.start();
SessionID sessionId = initiator.getSessions().get(0);
I am using the below code to send messages after continuously listening a directory using while Loop.
while(true)
{
readFilefromSrcDirectory();
prepareFixMessage();
Session.sendToTarget(fixMessage, sessionId);
}
My above code is getting executed while debugging but when I run it normally, the Session.sendToTarget(fixMessage, sessionId); and other file read related logic which is next to initiator.start(); is not getting executed.
Kindly note that the same above code is getting executed if we add some console print statements such as System.out.print("Test");
Please help me.
Are your test.cfg settings between debug and run different? I would add console print statements everywhere and work out exactly where the runtime is failing.
In a Windows Store app I can only store WinRT types in the ApplicationSettings, according to the documentation. For roamed settings that should be held together I can use ApplicationDataCompositeValue. Trying to store an instance of an own class or struct results in an Exception with the message " WinRT information: Error trying to serialize the value to be written to the application data store. Additional Information: Data of this type is not supported". The term "trying to serialize" indicates that there must be some way so serialize a type for the application data API.
Does anyone know how I could achieve that?
I tried DataContract serialization but it did not work.
I think custom/own types are not supported.
See http://msdn.microsoft.com/en-us/library/windows/apps/hh464917.aspx:
"The Windows Runtime data types are supported for app settings."
But you can serialize your objects to XML and save as string... (see code below)
public static string Serialize(object obj)
{
using (var sw = new StringWriter())
{
var serializer = new XmlSerializer(obj.GetType());
serializer.Serialize(sw, obj);
return sw.ToString();
}
}
public static T Deserialize<T>(string xml)
{
using (var sw = new StringReader(xml))
{
var serializer = new XmlSerializer(typeof(T));
return (T)serializer.Deserialize(sw);
}
}
https://github.com/MyToolkit/MyToolkit/blob/master/src/MyToolkit/Serialization/XmlSerialization.cs
Check out this class too:
https://github.com/MyToolkit/MyToolkit/wiki/XmlSerialization
Disclaimer: The above links are from my project
Is there a way to create a NService Bus ticket within SSIS package.
I am new at using SSIS packages (MSSQL 2012) and was thinking of serializing an NServiceBus ticket using a SCRIPT TASK and sending it to a MSMQ queue using a Messaging Task.
Am I thinking in the right direction?
We have done this internally, you just have to make sure you get the serialization right. You also have to be aware of what gets sent in the message label. Here is what is working on v2.5 which may have to be modified for 3.x:
String label = String.Format("<CorrId></CorrId><WinIdName>{0}\\Administrator</WinIdName>", Environment.MachineName);
Message message = new Message();
XmlSerializer serializer = new XmlSerializer(typeof(IMyMessage));
serializer.Serialize(message.BodyStream, command);
message.Label = label;
MessageQueue q = new MessageQueue(queueName);
if (q != null )
{
q.Send(message, MessageQueueTransactionType.Single);
}