I am aware that JSON strings often have a max length defined in either Apache on PHP, however where is the max length for JSON strings defined in Grails using TomCat?
The JSON string I am sending is 13,636 characters in length, however I can shorten it a lot (although I don't want to while we're testing) - also, we may be sending images via JSON in the future which I've read requires base64 encoding and thus a considerable overhead. If we were to do such a thing then I am worried that if this limit is causing problems, it's something we should overcome now.
If there is no limit, then perhaps I am doing something wrong. I have a finite amount of domain objects that I am encoding as JSON using domainInstance as grails.converters.deep.JSON - this is done using a for loop and each time the JSON string is appended to a StringBuilder
I then render the StringBuilder in a view using render(stringBuilder.toString()) and the first JSON string is fine, however the second is truncated near to the end. If I were to guestimate I'd say I am getting around 80% of the total length of the StringBuilder.
EDIT/SOLUTION: Apologies guys & girls, I've noticed that when I view source on the page I get the complete JSON string, however when I just view the page it's truncated. It's an odd error, I'll accept answers on why it's truncated, though. Thanks.
There is a maximum size in Tomcat for POST requests, which you may be concerned with later if you start sending huge JSON / Base64 image requests (like you mentioned).
The default value in tomcat is 2,097,152 bytes (2 MB); it can be changed by setting the maxPostSize attribute of the <Connector> in server.xml.
From the docs (for maxPostSize in Tomcat 7):
The maximum size in bytes of the POST which will be handled by the
container FORM URL parameter parsing. The limit can be disabled by
setting this attribute to a value less than or equal to 0. If not
specified, this attribute is set to 2097152 (2 megabytes).
This is pretty straightforward to configure if you're deploying a Grails war to a standalone Tomcat instance. However, I can't find much about actually configuring server.xml if you're using the tomcat plugin. If it were me, I'd probably just run large-file tests using the war instead of with grails run-app.
Related
I found the CacheService is quite fast (duh) so decided to create a CacheManager to store a whole manner of things.
JS Object --> JSON--> Blob --> Zip --> base64 encoded string
if the base64 string is > 1E5 chars (100kb) I create an MD5 checksum for the base64 string then split into 100kb sections and then cache those separately as a multipart zip string
I was able to store/ recall ~3MB of raw JSON data in this fashion in ~1.2s (similar speed to a DriveApp API call)
I tried searching for an overall limit for how many total cached objects could be created but didn't find much. Is anyone aware of of an overall limit or performance degradation with large numbers of cached strings?
Source Code for my "cache manager"
Edit: Fixed source URL
There is no documented value for this in the docs, but you can try the tests conducted according to this SO post. I think the value shouldn't exceed 10MB. Read more on the test conducted here.
I have a Nifi flow which processes data from the webhose API, webhose returns a whole webpage of text in its result as a attribute in Json. When I try to extract this using EvaluateJsonPath processor and write it to a new attribute it gives me the "nifi processor exception repository failed to update" error, the content is encoded in utf8 and I know that there is a limitation of 65535 bytes for an attribute in Json. Is there a workaround for this.
I believe this limitation should be resolved in Apache NiFi 1.2.0 from this JIRA:
https://issues.apache.org/jira/browse/NIFI-3389
Also, keep in mind that having a lot of large attributes is not ideal for performance.
I am trying to use the following command to get the page's source code
requests.get("website").text
But I get an error:
UnicodeEncodeError: 'gbk' codec can't encode character '\xe6' in position 356: illegal multibyte sequence
Then I tried to change the page code to utf-8
requests.get("website").text.encode('utf-8')
But in addition to English will become the following form
\xe6°\xb8\xe4\xb9\x85\xe6\x8f\x90\xe4\xbe\x9b\xe5\x85\x8dè\xb4\xb9VPN\xe5\xb8\x90\xe5\x8f·\xe5\x92\x8c\xe5\x85\x8dè\xb4\xb
How can I do?
Thank you for your help
You can query the encoding of the requests.Response object by accessing its Response.content attribute.
Whenever you call requests.Response.text, the response object uses requests.Response.encoding to decode the bytes.
This may, however, not always be the correct encoding, hence you sometimes have to set it manually by looking it up in the content attribute, since websites usually specify the encoding there, if it's not utf-8 or similar (this is from experience, I'm not sure if this is actual standard behavior).
See more on requests.Response contents here
I'm using RestSharp. I was using XML as the transport encoding for my data, but then I had problems with that, so I switched to using JSON. Now I'm having problems with that too!
All was well until I tried to pass an object containing a byte array (*). Now I get a de-serialization error complaining about "object has no parameterless constructor". (The JSON returned by the server looks Kosher - it's just not being correctly de-serialized by RestSharp).
I see I'm not the only one having problems. Is there no solution other than the baby-out-with-the-bathwater approach suggested in that post?
(*) I had tested it with a small hand-coded byte array early on in my development, just to check that it worked. It did work then, but doesn't work now. I don't know if that's due to the size of the array, the "characters" in the array, or what. Dammit, this has been such a time-sink!
For anyone else struggling with this, I ended up simply swapping out the JSON formatter in favour of JSON.NET. That works.
I want to serialize an object to AMF, and I want the result to be exactly the same as if it is serialized by NetConnection.call(). So, I use ByteArray.writeObject(), and the output bytes are usually the same as bytes sent by NetConnection.call(), but sometimes couple of bytes are different.
I found this in AMF3 spec: "Note that ByteArray.writeObject uses one version of AMF to encode the entire object. Unlike NetConnection, ByteArray does not start out in AMF 0 and switch to AMF 3 (with the objectEncoding property set to AMF 3)." It explains that differences.
How can I solve this problem?
The way that NetConnection.call works and how to construct valid requests and responses is documented in detail in the AMF0 specs in section 4. NetConnection.call has some additional functionality, like headers, the RPC method name, and whether or not the request was successful or ran into an error. This is why you can't just use writeObject to create a valid request.
The bit about switching from AMF0 to AMF3 is due to the fact that not every AS3 object can be written without a loss of data in AMF0, but original Flash Players all assumed that the body would be in AMF0. What happens is that during encoding, if you've specified that you want to use AMF3 for encoding, it writes out an AMF0-to-AMF3 marker (0x11) before calling writeObject in AMF3 mode.