'System.OutOfMemoryException' was thrown while loading bulk data - json

When I loaded 100,000 rows with 20 columns intp kendo grid, I am getting 500 error.
So i have checked the response of json , getting memory out of exception . This is the following code.
In the webconfig,have set
<system.web.extensions>
<scripting>
<webServices>
<jsonSerialization maxJsonLength="500000000"/>
</webServices>
</scripting>
</system.web.extensions>
This is the mvc controller.
public JsonResult JqueryKendoGridVirtualScrolling()
{
using (var s = new KendoEntities())
{
var x = s.premiumsbytreaties.ToList().Take(100000);
if (x != null)
{
var jsonResult = Json(x, JsonRequestBehavior.AllowGet);
jsonResult.MaxJsonLength = int.MaxValue;
return jsonResult;
//return Json(x.ToList(), JsonRequestBehavior.AllowGet);
}
else
{
return null;
}
};
}
It is working fine for 6 columns.But not working for 15 columns.
working fine with 20,000 thousand record,see the output

Two problems here:
500,000,000 characters is too long for the JSON content of an HTTP Response. This will take time to pass over the network and that is before the client browser can parse and process the data (taking time and memory).
1% of this is still too much.
You are not using Kendo's support for "Grid / Virtualization of remote data".
In their example the action's first parameter is [DataSourceRequest] DataSourceRequest request and then this is used (with the ToDataSource extension method) in creating the response.
Like with AJAX methods supporting Kendo Grids (and other controls)
these work together to pass paging, sorting and filtering parameters and
apply to the data before it is returned.
If your data is an IQueryable<T> for a suitable db provider this will
happen on the database.

Related

PostAsJsonAsync not calling the Controller when <Tvalue> has null fields

My TValue object has foreign key related objects, which has null values when posting; I am having the logic to set the FK objects in the repository. The issue I am facing is that API controller is not getting called when FK objects have all fields null. Please see screenshot. The same code works if I set the value for all but the ID field of the FK objects from the front end.
Is the issue because Json serializer checking for nulls? I have also tried to set the null check ignore option. I am not getting an error on PostAsJsonAsync and the control simply goes to the next line of code
return await result.Content.ReadFromJsonAsync();
without calling the API controller and send an exception
public async Task<SubContract> AddSubContract(SubContract subContract)
{
/* On the injected httpClient, call the PostAsJsonAsync method and pass the subContractObject
* We also need to specifi the api Uri in the parameter list */
JsonSerializerOptions option = new()
{
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
};
var result = await httpClient
.PostAsJsonAsync<SubContract>("api/SubContracts", subContract, option);
//Use the content object and ReadFromJsonSync method and typecast it to <SubContract>
return await result.Content.ReadFromJsonAsync<SubContract>();
}
Screenshot
--- Further observations ---
#Serge Thanks for the response. You are right, I am using .Net 6. I have now commented out the nullable but I still have the same issue. Further, I tried to change the function to PostAsync instead of PostAsJsonAsync; below is the new code
// ---- Post Asysc Option -----
var subContractSeralized = JsonSerializer.Serialize(subContract, option);
var stringContent = new StringContent(subContractSeralized,
encoding: System.Text.Encoding.UTF8, "application/json");
var response = await httpClient.PostAsync("api/subcontract", stringContent);
return await response.Content.ReadFromJsonAsync<SubContract>();
I initially thought it was a serialization issue because of the nulls in the nested object but when I debug the new code, I get the below result
subContractSerealized = '{"Id":0,"Name":"Aquatic-Repairs","Status":"In-Progress","WorkTypeId":1002,"WorkType":{"Id":0},"SiteId":3,"Site":{"Id":0},"OrganizationId":3,"Organization":{"Id":0}}'
If you compare this with the Debug screen shot in my first post, you can see that the null value fields in the nested objects are omitted out
Response StatusCode = “Not Found-404”
I am not sure how Response Status code is obtained as the API is not called. I.e. httpClient.PostAsync does not transfer control to the API and my debug breakpoint is not hit.
I tried the same code for an Entity model that has no nested foreign key related objects and it works fine and I am able to add the record to the DB. I have the “Required” validation set on the field properties of the entity models; however, after the API call, I have my repository that is taking care of it. So, I doubt that is an issue. In any case, the code is not even hitting the API and simply returns an 404 NotFound on httpClient.PostAsync.
you must be using Net 6 API, and it causes a validation error. Try to comment Nullable in your API project (your serializer option is not working in this case)
<PropertyGroup>
<TargetFramework>net6.0</TargetFramework>
<!--<Nullable>enable</Nullable>-->
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>

JsonDocument incomplete parsing with larger payloads

So basically, I have a HttpClient that attempts to obtain any form of JSON data from an endpoint. I previously utilized Newtonsoft.Json to achieve this easily but after migrating all of the functions to STJ, I started to notice improper parsing.
Platforms tested: macOS & Linux (Google Kubernetes Engine)
Framework: .NET Core 3.1 LTS
The code screenshots below show an API that returns a JSON Array. I simply stream it, load it into a JsonDocument, and then attempt to peek into it. Nothing comes out as expected. Code below is provided along with the step debug var results.
using System;
using System.ComponentModel;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
using System.Web;
using System.Xml;
namespace HttpCallDemo
{
class Program
{
static async Task Main(string[] args)
{
using (var httpClient = new HttpClient())
{
// FLUSH
httpClient.DefaultRequestHeaders.Clear();
httpClient.MaxResponseContentBufferSize = 4096;
string body = string.Empty, customMediaType = string.Empty; // For POST/PUT
// Setup the url
var uri = new UriBuilder("https://api-pub.bitfinex.com/v2/tickers?symbols=ALL");
uri.Port = -1;
// Pull in the payload
var requestPayload = new HttpRequestMessage(HttpMethod.Get, uri.ToString());
HttpResponseMessage responsePayload;
responsePayload = await httpClient.SendAsync(requestPayload,
HttpCompletionOption.ResponseHeadersRead);
var byteArr = await responsePayload.Content.ReadAsByteArrayAsync();
if (byteArr.LongCount() > 4194304) // 4MB
return; // Too big.
// Pull the content
var contentFromBytes = Encoding.Default.GetString(byteArr);
JsonDocument payload;
switch (responsePayload.StatusCode)
{
case HttpStatusCode.OK:
// Return the payload distinctively
payload = JsonDocument.Parse(contentFromBytes);
#if DEBUG
var testJsonRes = Encoding.UTF8.GetString(
Utf8Json.JsonSerializer.Serialize(payload.RootElement));
// var testRawRes = contentStream.read
var testJsonResEl = payload.RootElement.GetRawText();
#endif
break;
default:
throw new InvalidDataException("Invalid HTTP response.");
}
}
}
}
}
Simply execute the above Minimal code, notice that the payload is different from its original after parsing? I'm sure there's something wrong with the options for STJ. Seems like we have to optimise or explicitly define its limits to allow it to process that JSON payload.
Diving deeper into the debug content made things even weirder. When the HttpClient obtains the payload, reads it to a string, it gives me the entire JSON string as is. However, once we attempt to parse it into a JsonDocument and the further invoking RootElement.Clone(), we'll end up with a JsonElement with much lesser data and while carrying an invalid JSON struct (Below).
ValueKind = Array : "[["tBTCUSD",11418,70.31212518,11419,161.93475693,258.02141213,0.0231,11418,2980.0289306,11438,11003],["tLTCUSD",58.919,2236.00823543,58.95,2884.6718013699997,1.258,0.0218,58.998,63147.48344762,59.261,56.334],["tLTCBTC",0.0051609,962.80334198,0.005166,1170.07399991,-0.000012,-0.0023,0.0051609,4178.13148459,0.0051852,0.0051],["tETHUSD",396.54,336.52151165,396.55,384.37623341,8.26964946,0.0213,396.50930256,69499.5382821,397.77,380.5],["tETHBTC",0.034731,166.67781664000003,0.034751,356.03450125999996,-0.000054,-0.0016,0.034747,5855.04978836,0.035109,0.0343],["tETCBTC",0.00063087,15536.813429530002,0.00063197,16238.600279749999,-0.00000838,-0.0131,0.00063085,73137.62192801,0.00064135,0.00062819],["tETCUSD",7.2059,9527.40221867,7.2176,8805.54677899,0.0517,0.0072,7.2203,49618.78868196,7.2263,7],["tRRTUSD",0.057476,33577.52064154,0.058614,20946.501210000002,0.023114,0.6511,0.058614,210741.23592011,0.06443,0.0355],["tZECUSD",88.131,821.28048322,88.332,880.37484662,5.925,0.0
And of course, attempting to read its contents would result in:
System.InvalidOperationException: Operation is not valid due to the current state of the object.
at System.Text.Json.JsonElement.get_Item(Int32 index)
at Nozomi.Preprocessing.Abstracts.BaseProcessingService`1.ProcessIdentifier(JsonElement jsonDoc, String identifier) in /Users/nicholaschen/Projects/nozomi/Nozomi.Infra.Preprocessing/Abstracts/BaseProcessingService.cs:line 255
Here's proof that there is a proper 38KBs worth of data coming in from the endpoint.
UPDATE
Further testing with this
if (payload.RootElement.ValueKind.Equals(JsonValueKind.Array))
{
string testJsonArr;
testJsonArr = Encoding.UTF8.GetString(
Utf8Json.JsonSerializer.Serialize(
payload.RootElement.EnumerateArray()));
}
show that a larger array of arrays (exceeding 9 elements each with 11 elements) would result in an incomplete JSON struct, causing the issue i'm facing.
For those who are working with JsonDocument and JsonElement, take note that the step debug variables are not accurate. It is not advisable to inspect the variables during runtime as they do not display themselves entirely.
#dbc has proven that re-serializing the deserialized data will produce the complete dataset. I strongly suggest you wrap the serializers for debugging in a DEBUG preprocessor to make sure these redundant lines don't end up being executed out of development.
To interact with these entities, ensure you .clone() whenever you can to prevent disposals and ensure that you're accessing the RootElement and then subsequently traversing into it before viewing its value in step debug mode because large values will not be displayed.

Modify the rendered string of a response in Grails 3 before sending to the client

I need to create a benchmark report regarding whether in the grand scheme of things: minifying + GZIP dynamic HTML responses (generated through GSPs) on every request, which will lead to an additional overhead due to parsing of the generated dynamic HTML string then compressing using a Java library (which results to a smaller response size) is actually better than GZIP without minifying (which results to faster response time but a little larger response size). I got the feeling that this "improvement" maybe is insignificant, but I need the benchmark report to back it up to the team.
To do that, I modify controller actions like so:
// import ...MinifyPlugin
class HomeController {
def get() {
Map model = [:]
String htmlBody = groovyPageRenderer.render(view: "/get", model: model)
// This adds a few milliseconds and reduce few characters.
htmlBody = MinifyPlugin.minifyHtmlString(htmlBody)
render htmlBody
}
}
But the Grails project has almost a hundred actions and doing this on every existing action is impractical and not maintainable, especially that after the benchmarking, we may decide to not minify the HTML response. So I was thinking of doing this inside an Interceptor instead:
void afterView() {
if(response.getContentType().contains("text/html")) {
// This throws IllegalStateException: getWriter() has already been called for this response
OutputStream servletOutputStream = response.getOutputStream()
String htmlBody = new String(servletOutputStream.toByteArray())
htmlBody = MinifyingPlugin.minifyHtmlString(htmlBody)
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream()
byteArrayOutputStream.write(htmlBody.getBytes())
response.setCharacterEncoding("UTF-8")
response.setContentType("text/html")
response.outputStream << byteArrayOutputStream
}
}
But it seems that modification of the response body is impossible once it enters the afterView interceptor...? So is any other way to do this using Grails 3 Interceptors, or should I update every controller action we have manually and perform the modification there instead?
This is what I like to use Interceptors for.
The after() part of the interceptor can act on the model after it is returned from the controller (wherein 'before()' acts on the request before it is sent to the controller)
This allows you to manipulate all data for a set of endpoints (or one specific endpoint) prior to return to client
If you are wanting to render to a view, you do that in the interceptor rather than in the controller; you merely return data from the controller

html5 chrome video playback gets EofException,

I finally got video playback to work in chrome with seek feature using the headers Content-Range etc and status 206 returned. It worked well for smaller videos but fails with large videos. Just to note, I am not sending the actual byte ranges explicitly but deliver the entire stream to the webserver. I get the following errors:
org.eclipse.jetty.io.EofException,
this occurs in the backend dataserver that serves the entire inputstream to a servlet and jetty is the server being used. I am not sure how this process actually plays back and corrected the seek feature I needed but now the video fails after playing for a while. The following error also occurs in the browser debugger:
ERR_CONTENT_LENGTH_MISMATCH
I have an audio stream being requested at the same time and playedback as well since I do not know how to mix the two streams.
Any ideas or advice appreciated.
EDIT:
Thanks to the advice to change resourcehandler to defaultservlet; not sure where to do this so found the instances of where this is in the code:
private void addHttpContexts(ConfigNode cnode) throws Exception {
try {
// get all the http context nodes
ConfigNode[] httpContextNodes = cnode.getChildNode("HttpContextList").getChildNodes();
for (int s = 0; s < httpContextNodes.length; s++) {
String urlPath = httpContextNodes[s].getChildNode("ContextPath").getStringValueEx();
String resourceBase = httpContextNodes[s].getChildNode("ResourceBase").getStringValueEx();
ArrayList<String> welcomeFileList = new ArrayList<String>();
if (httpContextNodes[s].hasChildNode("WelcomeFile")) {
String welcomeFile = httpContextNodes[s].getChildNode("WelcomeFile").getStringValueEx();
welcomeFileList.add(welcomeFile);
}
ContextHandler context = new ContextHandler(contexts, urlPath);
ResourceHandler resourceHandler = new ResourceHandler();
resourceHandler.setResourceBase(resourceBase);
resourceHandler.setWelcomeFiles((String[]) welcomeFileList.toArray(new String[welcomeFileList.size()]));
context.setHandler(resourceHandler);
} catch (Exception ex) {
trace.warning("Configuration of http contexts failed", ex);
throw ex;
}
}
What is the appropriate methods for setResourceBase(resourceBase) and
setWelcomeFiles((String[]) welcomeFileList.toArray(new String[welcomeFileList.size()]));
This is the other place in the the same class I found DefaultSErvlet
ServletHolder holderDefault = new ServletHolder("default",DefaultServlet.class);
holderDefault.setInitParameter("dirAllowed","false");
and also already defined in web.xml
<servlet>
<servlet-name>default</servlet-name>
<servlet-class>org.eclipse.jetty.servlet.DefaultServlet</servlet-class>
<init-param>
<param-name>dirAllowed</param-name>
<param-value>false</param-value>
</init-param>
</servlet>
By default, Jetty's DefaultServlet will handle range requests properly for static content served by Jetty itself.
No other component in Jetty handles range requests on its own.
If you have custom code, your own Servlets, your own Jetty Handlers, a REST endpoint, specialized Filters, spring-mvc setup, etc... then you have to handle the range request yourself.
This is because its very impractical for the webserver to support this for custom code. (It would have to request the entire content from the custom code, and then only send the specific byte range to the requesting client).

Passing a very large json object to mvc4 controller

I've had a look and I can't seem to get this to work and I constantly get the error message that my json object is too large to pass to my controller.
The length of the string exceeds the value set on the maxJsonLength property.
So I have a json object that contains the base64 string of up to 3 images as well as some textual data.
I am trying to pass that object to the controller. It works fine if I do not include the images.
So I have tried, in web.config;
<system.web.extensions>
<scripting>
<webServices>
<jsonSerialization maxJsonLength="500000000"/>
</webServices>
</scripting>
</system.web.extensions>
as well as;
<add key="aspnet:MaxJsonDeserializerMembers" value="150000000" />
I have changed my controller from an ActionResult to a JsonResult.
I've read that I need to create my own json parser etc and a lot of the sites deal with passing data from the controller to the client.
I am trying to get a large amount of data from the client to the controller.
Any help here?
If you take a look at the source code ASP.NET MVC for the JsonResult class, you find this code (+):
if (Data != null)
{
JavaScriptSerializer serializer = new JavaScriptSerializer();
if (MaxJsonLength.HasValue)
{
serializer.MaxJsonLength = MaxJsonLength.Value;
}
if (RecursionLimit.HasValue)
{
serializer.RecursionLimit = RecursionLimit.Value;
}
response.Write(serializer.Serialize(Data));
}
The defualt values for MaxJsonLength property is set to 2MB and RecursionLimit is set to 100.
You can change the defualt values this way:
return new JsonResult
{
MaxJsonLength = Int32.MaxValue,
Data = customers,
JsonRequestBehavior = JsonRequestBehavior.AllowGet
};
more info.