I have a big json payload of around 12K characters. AWS Eventbridge is not letting me create event as there is a payload limit of 8192 characters.
How would I resolve this?
Thanks
According to API doc, this is a hard limit on the API level. A workaround would be to split it into two targets.
Another way to handle larger payloads on Eventbridge would be to put the payload to S3 and pass a reference (bucket name + key) as the payload.
Related
I have a dictionary file with 200,000 items in it.
I have a Dictionary Model which matches the SQLite db and the proper methods.
If I try to parse the whole file, it seems to hang. If I do 8000 items, it seems to do it quite quickly. Is there a size limit, or is just because there might be some corrupted data somewhere? This json was exported from the sqlite db as json pretty, so I would imagine it was done correctly. It also works fine with the first 8000 items.
String peuJson = await getPeuJson();
List<Dictionary> dicts = (json.decode(peuJson) as List)
.map((i) => Dictionary.fromJson(i))
.toList();
JSON is similar to other data formats like XML - if you need to transmit more data, you just send more data. There's no inherent size limitation to the JSON request. Any limitation would be set by the server parsing the request.
I'm trying to use ADF for the following scenario:
a JSON is uploaded to a Azure Storage Blob, containing an array of similar objects
this JSON is read by ADF with a Lookup Activity and uploaded via a Web Activity to an external sink
I cannot use the Copy Activity, because I need to create a JSON payload for the Web Activity, so I have to lookup the array and paste it like this (payload of the Web Activity):
{
"some field": "value",
"some more fields": "value",
...
"items": #{activity('GetJsonLookupActivity').output.value}
}
The Lookup activity has a known limitation of an upper limit of 5000 rows at a time. If the JSON is larger, only 5000 top rows will be read and all else will be ignored.
I know this, so I have a system that chops payloads into chunks of 5000 rows before uploading to storage. But I'm not the only user, so there's a valid concern that someone else will try uploading bigger files and the pipeline will silently pass with a partial upload, while the user would obviously expect all rows to be uploaded.
I've come up with two concepts for a workaround, but I don't see how to implement either:
Is there any way for me to check if the JSON file is too large and fail the pipeline if so? The Lookup Activity doesn't seem to allow row counting, and the Get Metadata Activity only returns the size in bytes.
Alternatively, the MSDN docs propose a workaround of copying data in a foreach loop. But I cannot figure out how I'd use Lookup to first get rows 1-5000 and then 5001-10000 etc. from a JSON. It's easy enough with SQL using OFFSET N FETCH NEXT 5000 ROWS ONLY, but how to do it with a JSON?
You can't set any index range(1-5,000,5,000-10,000) when you use LookUp Activity.The workaround mentioned in the doc doesn't means you could use LookUp Activity with pagination,in my opinion.
My workaround is writing an azure function to get the total length of json array before data transfer.Inside azure function,divide the data into different sub temporary files with pagination like sub1.json,sub2.json....Then output an array contains file names.
Grab the array with ForEach Activity, execute lookup activity in the loop. The file path could be set as dynamic value.Then do next Web Activity.
Surely,my idea could be improved.For example,you get the total length of json array and it is under 5000 limitation,you could just return {"NeedIterate":false}.Evaluate that response by IfCondition Activity to decide which way should be next.It the value is false,execute the LookUp activity directly.All can be divided in the branches.
I'm trying to insert very long string to firebase firestore
and I receive this error message
Exception has occurred.
PlatformException (PlatformException(Error performing setData,
INVALID_ARGUMENT: The value of property "json" is longer than 1048487 bytes., null))
my code is :
Future<void> addREportToDB(String addedCount, String deletCount, String updatedCount, String json) {
// Map decodedCloud = jsonDecode(json);
return Firestore.instance.collection("reports").add({
"dateCreate": new DateTime.now(),
"addedCount": addedCount,
"deletCount": deletCount,
"updatedCount": updatedCount,
"json": json,
// "json":decodedCloud,
}).then((doc) {
print(doc.documentID.toString());
});
}
and the (json)variable This is a text I get from another API and this text contains the data of more than 600 employees in the form of JSON string
and all I need to save this as it
any help will be appreciated
There is no way you can add into a single document data above that limitation. So there are some limits when it comes to how much data you can put into a single document. According to the official documentation regarding usage and limits:
Maximum size for a document: 1 MiB (1,048,576 bytes)
As you can see, you are limited to 1 MiB total of data in a single document. You can use alternative solution for storing larger amounts of data. You should try using Firebase Storage.
The maximum size of a document in Cloud Firestore is 1MB. If you want to store more data, consider either splitting it over more documents or (more likely in this case) storing it in Cloud Storage (for which a Flutter SDK also exists).
Just save them separately to firestore and when you want to use them in your app, you can use string operation to join the two seperate strings you saved to make it the original long string
I am trying to fetch BinaryDocuments uploaded by cbworkloadgen from Couchbase 4.0.0-4051 Community Edition. Couchbase Java client version is 2.4.1.
The exception given by decoder is -
WARNING: Decoding of document with BinaryTranscoder failed. exception: Flags (0x0) indicate non-binary document for id pymc0, could not decode., id: "pymc0", cas: 1486468016723525632, expiry: 0, flags: 0x0, status: SUCCESS, content size: 2048 bytes, content: "".
com.couchbase.client.java.error.TranscodingException: Flags (0x0) indicate non-binary document for id pymc0, could not decode.
com.couchbase.client.java.error.TranscodingException: Flags (0x0) indicate non-binary document for id pymc0, could not decode.
at com.couchbase.client.java.transcoder.BinaryTranscoder.doDecode(BinaryTranscoder.java:32)
at com.couchbase.client.java.transcoder.BinaryTranscoder.doDecode(BinaryTranscoder.java:26)
at com.couchbase.client.java.transcoder.AbstractTranscoder.decode(AbstractTranscoder.java:42)
at com.couchbase.client.java.CouchbaseAsyncBucket$1.call(CouchbaseAsyncBucket.java:274)
at com.couchbase.client.java.CouchbaseAsyncBucket$1.call(CouchbaseAsyncBucket.java:270)
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
I use following to get document -
AbstractDocument<?> doc = destinationBucket.get((String) row.key(), isJson ? JsonDocument.class : BinaryDocument.class);
For JsonDocument things work okay. row is AsyncViewRow.
What am I doing wrong? Can someone please tell me? Or is this a bug related to incorrect value in flags field?
Well, due to want of time, I changed the approach since I was also getting out of memory errors when iterating a view asynchronously on a bucket million documents.
As regard to this issue, it may be that he flags field set by cbworkloadgen without -j option for each document is 0 and the BinaryTranscoder thinks it's not a binary document because of this value. I got around the problem by using N1ql instead of doing get(). However, I am not sure if this is an issue with cbworkloadgen where it's not setting correct flags.
You can not decode binary documents on your own. If you save something that implements Serializable, it will be serialized and saved to Couchbase and you can retrieve the same easily. But if you fire a N1QL query and try fetching the binary data, you won't be able to decode it. This is something Couchbase doesn't support yet. You can do the same with Json docs.
I'm trying to parse a large message with Logstash using a file input, a json filter, and an elasticsearch output. 99% of the time this works fine, but when one of my log messages is too large, I get JSON parse errors, as the initial message is broken up into two partial invalid JSON streams. The size of such messages is about 40,000+ characters long. I've looked to see if there is any information on the size of the buffer, or some max length that I should try to stay under, but haven't had any luck. The only answers I found related to the udp input, and being able to change the buffer size.
Does Logstash has a limit size for each event-message?
https://github.com/elastic/logstash/issues/1505
This could also be similar to this question, but there were never any replies or suggestions: Logstash Json filter behaving unexpectedly for large nested JSONs
As a workaround, I wanted to split my message up into multiple messages, but I'm unable to do this, as I need all the information to be in the same record in Elasticsearch. I don't believe there is a way to call the Update API from logstash. Additionally, most of the data is in an array, so while I can update an Elasticsearch record's array using a script (Elasticsearch upserting and appending to array), I can't do that from Logstash.
The data records look something like this:
{ "variable1":"value1",
......,
"variable30": "value30",
"attachements": [ {5500 charcters of JSON},
{5500 charcters of JSON},
{5500 charcters of JSON}..
...
{8th dictionary of JSON}]
}
Does anyone know of a way to have Logstash process these large JSON messages, or a way that I can split them up and have them end up in the same Elasticsearch record (using Logstash)?
Any help is appreciated, and I'm happy to add any information needed!
If your elasticsearch output has a document_id set, it will update the document (the default action in logstash is to index the data -- which will update the document if it already exists)
In your case, you'd need to include some unique field as part of your json messages and then rely on that to do the merge in elasticsearch. For example:
{"key":"123455","attachment1":"something big"}
{"key":"123455","attachment2":"something big"}
{"key":"123455","attachment3":"something big"}
And then have an elasticsearch output like:
elasticsearch {
host => localhost
document_id => "%{key}"
}