When issuing a POST request with a large body to my Play framework action method, I get null when extracting the data. If the body is fairly small, I can retrieve the data just fine.
Here's a sample short data set:
{
"creator": "zoltan",
"sport": "hike",
"geometry": [
{
"time": "2009-07-10 12:56:10 +0000",
"x": 10.275514,
"y": 47.514749,
"z": 756.587
},
{
"time": "2009-07-10 12:56:19 +0000",
"x": 10.275563,
"y": 47.514797,
"z": 757.417
}
]
}
When I issue a POST request with this JSON in the body, everything works fine. However, if I add many more (~4000) points in the geometry array, I get null in the action.
Here's my action method:
#Transactional
//#BodyParser.Of(Json.class) // tried with this as well
public static Result createTour() {
LOG.debug("Raw request body: " + request().body().asText());
JsonNode node = request().body().asJson();
LOG.debug("JSON request body: " + node);
TourDto tourDto;
try {
tourDto = jsonToTour(node);
int id = TourDataAccessUtils.create(tourDto);
return created(toJson(id));
} catch (JsonProcessingException e) {
LOG.error("While parsing JSON request.", e);
return Results.badRequest(
toJson(Throwables.getRootCause(e).getMessage()));
}
}
I tried using both Advanced REST Client in chrome and ċurl to send the request, both failed.
What could be the problem? Could it be that I need to include a Content-Lenght header for large requests? If so, how can I manually calculate it for arbitrary JSON data?
Please check the PlayFramework documentation, They mentionned that the default maximum length for request is 100KB:
Max content length
Text based body parsers (such as text, json, xml or
formUrlEncoded) use a max content length because they have to load all
the content into memory.
There is a default content length (the default is 100KB).
Tip: The default content size can be defined in application.conf:
parsers.text.maxLength=128K
You can also specify a maximum content length via the #BodyParser.Of
annotation:
// Accept only 10KB of data.
#BodyParser.Of(value = BodyParser.Text.class, maxLength = 10 * 1024)
pulic static Result index() {
if(request().body().isMaxSizeExceeded()) {
return badRequest("Too much data!");
} else {
ok("Got body: " + request().body().asText());
}
}
Related
I did see this same error, in c# has an open question, but I'm using PowerShell and using POST rather than PATCH so I've opened a separate question.
I'm having an issue when using JSON batching, particularly that when I include the header for Content-Type, I receive:
Invoke-RestMethod : {
"error": {
"code": "BadRequest",
"message": "Invalid batch payload format.",
"innerError": {
"date": "2020-10-14T00:25:46",
"request-id": "aa535dbb-efe8-450e-911d-143554ed9027",
"client-request-id": "aa535dbb-efe8-450e-911d-143554ed9027"
}
}
}
I first had missed including the headers at all, and was receiving the error:
{
"error": {
"code": "BadRequest",
"message": "Write request id : 2 does not contain Content-Type header or body.",
"innerError": {
"date": "2020-10-14T00:46:58",
"request-id": "3601be6d-a861-4947-936b-451cd9de80c3",
"client-request-id": "3601be6d-a861-4947-936b-451cd9de80c3"
}
}
}
The body of my HTTP request to https://graph.microsoft.com/v1.0/$batch is an array of PSCustomObjects that look like this:
id : 1
method : POST
url : sites/8c3cb1ef-4116-b0e4-6d0b-25d0f333a4ed/lists/a2b2d34e-6d32-df22-d562-472d3d8385d2/items
body : {
"fields": {
"DisplayName": "user1#contoso.com",
"CreatedDateTime": "2019-10-13",
"UserId": "c963d785-59fc-4384-5e7d-d466=2118e3347",
"UserType": "Guest",
}
}
headers : {
"Content-Type": "application/json"
}
I found it odd that when I omit headers, the payload is seemingly OK, and it sees that Content-Type has not been supplied. Once I add it in, then suddenly payload is not OK.
If I individually perform the requests, all the same data, it's also fine (items create in SPO list without issue). I feel like this fact confirms that the issue is not with the fields in the body.
I have no issues when batching GET requests (e.g. batching 100's of requests for auditLogs/signIns). These are essentially the same payload, minus no body/headers, so just id, url, and method.
Has anyone had any experienced this and found a solution?
PS. The Guid's are all fakes.
My issue ended up being that I needed to use ConvertTo-Json's -Depth parameter to successfully capture all of my body's content. It was 5 levels deep (Requests: [ Each Request: { body: { fields: { field names/values } } } ]).
This was only an issue with POST method requests, since a body is then needed, which in this case was for adding SPO list items, which requires the fields: key with children names and values for the list's columns / cells.
I'm trying to authenticate RADIUS Requests against a RESTful API (provided by Customer) using rlm_rest.
The problem I am facing is that
response JSON format (of REST API provided by Customer), is different from rlm_rest default format (indicated in etc/raddb/mods-enabled/rest).
My Virtual Server configuration as below:
Default
authorize {
...
...
rest
if (ok) {
update control {
Auth-Type := rest
}
}
}
mods-enabled/rest
authorize {
uri = "https://3rd-party-API/auth"
method = 'post'
body = 'json'
chunk = 0
tls = ${..tls}
data = '{
"code": 1,
"identifier": %I,
"avps": {
"User-Name": ["%{User-Name}"],
"NAS-IP-Address": ["%{NAS-IP-Address}"],
"Called-Station-Id": ["%{Called-Station-Id}"],
"Calling-Station-Id": ["%{Calling-Station-Id}"],
"NAS-Identifier": ["%{NAS-Identifier}"]
}
}'
}
Result
/sbin/radiusd -Xxx
HTTP response code
200
JSON Body
{
"code": "2",
"identifier": "91",
"avps": {
"Customer-Attributes": "Hello"
...
...
"Acct-Interim-Interval": "300"
}
}
The JSON structure is different from the example, and xlat parse
"code"
"identifier"
"avps"
And, of course, xlat finds no attributes match with the dictionary, while it cannot find "avps" and won't dig deeper.
So I was wondering is there anyway to either
Define the response JSON structure for xlat to parsing
Insert a "is_json" or "do_xlat" flag into the JSON ("avps"), and hope xlat will then dig deeper
Save the JSON and parse with exec/rlm_exec (using JQ or any other bash/JSON tools)
Please advise if there is any workaround. Thanks!
In FreeRADIUS version 4, there's a rlm_json module, which implements a custom node query language based on xpath (jpath), it is extremely limited and only supports some very basic queries (feel free to enhance it via PR :) ).
Below is an example I pulled out of my library of customer configurations. You can see here it's pulling out two keys (externalID and macAddress) from the root level of the JSON doc and assigning them to a couple of custom attributes (Subscriber-ID and Provisioned-MAC).
map json "%{rest_api:https://${modules.rest[rest_api].server}/admin/api/${modules.rest[rest_api].api_key}/external/getDeviceBySerialNumber?certificateSerialNumber=%{lpad:&TLS-Client-Cert-Serial 40 0}}" {
&Subscriber-ID := '$.externalId'
&Provisioned-MAC := '$.macAddress'
}
The xlat expansion can also be modified to send HTTP body data. Just put a space after the URL and pass your custom JSON blob.
I am brand new to webhooks and have had a nightmare trying to get my first one to function. I am trying to use a Particle Photon to send a single float temperature variable to a site that can graph the data. I've tried creating webhooks to thingspeak.com and librato.com with no success. My main attempts have been trying to use the json code in tutorials to create the webhook.
Here is my Photon's code:
#include "Particle.h"
float temp = 70.1000;
float adjust = 0.4;
int acOn = 0;
void setup()
{
}
void loop()
{
if(temp < 72)
adjust = random(1620000)/1000000.0;
else if(temp < 74)
adjust = random(1000000)/1000000.0;
else
adjust = -1*random(500000,2200000)/1000000.0;
temp = temp + adjust;
Particle.publish("librato_", String(temp), 60, PRIVATE);
Particle.process();
delay(30000);
}
and the webhook JSON code (for the librato attempt)
{
"event": "librato_",
"url": "https://metrics-api.librato.com/v1/metrics",
"requestType": "POST",
"auth": {
"username": "YOUR_LIBRATO_USERNAME",
"password": "YOUR_LIBRATO_API_TOKEN"
},
"json": {
"gauges": [
{
"name": "{{NAME}}",
"value": "{{temp}}",
"source": "{{PARTICLE_DEVICE_ID (particle API token?)}}"
}
]
},
"mydevices": true
}
where I have filled in the username and API token found on librato and also the particle api token in the IDE settings. Thanks in advance for the help.
Looks like this user had a similar problem which was solved with a simple renaming or the webhook:
https://community.particle.io/t/connection-to-librato-solved/19230
Check with the Particle CLI if any events are being generated. If you are still not seeing data in Librato, reach out to support#librato.com so that we can check the API for errors.
When we have many documents to be returned from the response, we can paginate the result and give the link in the response to retrieve the next set of documents to be fetched from the API. However my case is different, each document in my response can be very large. How do I deal with such response ?
For Example consider the sample Json below:
{
"excludes": null,
"endDate": 1422918663,
"appliedText": "Applied text",
"id": "135699543",
...
"assignedTo": {
"productPart": [
{
"part": "1"
},
{
"part": "2"
},
{
"part": "3"
},
{
"part": "4"
},
...
]
}
...
}
If you see here, the productPart can grow to tens of thousands. How can I restrict such large response yet retain the structure.
Fetch product parts with a separate request (if you have the possibility) which then in turn can be paginated.
Alt. return the first x product parts and then provide a link where more product parts can be obtained.
I believe you should try something like JSON-Streaming.
Streaming Processing (also known as Incremental Processing) is the most efficient way to process JSON content. It has the lowest memory and processing overhead, and can often match performance of many binary data formats available on Java platform
Here is the main idea:
try {
InputStream in = [..];
JsonParser parser = new JsonFactory().createParser(in);
JsonToken token = null;
while ((token = parser.nextToken()) == JsonToken.FIELD_NAME) {
/*
* Parse parts and store them in client DB
* use later client side pagination
*
* {
* "part": "i"
* }
*/
}
} catch (Exception e) {
//
}
I am working on an iOS app using the MonoTouch framework. I am using Visual Studio 2010 Professional SP1 with the Xamarin.iOS (v1.3.250) extension. I have been able to open a valid FacebookConnect.FBSession by using the FacebookConnect.FBLoginView with no issues but when I try to make a Graph API request using FacebookConnect.FBRequest I recieve a non-standard JSON style string. When I run following request through the Graph API Explorer:
me?fields=albums.fields(id,name,cover_photo)
I receive the following response:
{
"id": "111111111111111111",
"albums": {
"data": [
{
"id": "111111111111111111",
"name": "Some Album (#1)",
"cover_photo": "111111111111111111",
"created_time": "000-00-00T00:00:00+0000"
},
{
"id": "111111111111111111",
"name": "Some Album (#2)",
"cover_photo": "111111111111111111",
"created_time": "000-00-00T00:00:00+0000"
},
],
"paging": {
"cursors": {
"after": "xxxxxxxx=",
"before": "xxxxxxxx="
}
}
}
}
Now all of this is just fine and is what I expect to receive but when I make the same Graph API request from my app like this:
public static void GetPhotoAlbums(string _userID)
{
// _userID = "me"
mFBRequest = new FBRequest(FBSession.ActiveSession, _userID + "?fields=albums.fields(id,name,cover_photo)");
FBRequestConnection fbRequestConnection = new FBRequestConnection();
fbRequestConnection.AddRequest(mFBRequest, OnPhotoAlbumsReceived);
fbRequestConnection.Start();
}
static void OnPhotoAlbumsReceived(FBRequestConnection _connection, NSObject _result, NSError _error)
{
if (_error == null)
{
Console.WriteLine("FacebookManager.OnPhotoAlbumsReceived() - JSON: " + _result.Description);
object o = JsonConvert.DeserializeObject(_result.Description);
// ...
}
}
I receive this JSON 'like' response:
{
albums = {
data = (
{
"cover_photo" = 111111111111111111;
"created_time" = "000-00-00T00:00:00+0000";
id = 111111111111111111;
name = "Some Album (#1)";
},
{
"cover_photo" = 111111111111111111;
"created_time" = "000-00-00T00:00:00+0000";
id = 111111111111111111;
name = "Some Album (#2)";
},
);
paging = {
cursors = {
after = "xxxxxxxx=";
before = "xxxxxxxx=";
};
};
};
"id": "111111111111111111";
}
I'm not really sure how/why I'm getting a response formatted in a non-standard way but needless to say, I get Newtonsoft.Json.JsonReaderException when attempting to deserialize the data because it does not follow the standard formatting rules (ie, = instead of : to separate key/value pairs, ; instead of , to separate elements of a container, some keys having quotes while others do not, etc...)
I'm pretty new to Facebook and JSON stuff in general and am really at a loss for what is happening to the response string I receive. Any help, feedback, ideas are greatly appreciated. Thanks.
* Solution *
After a bunch of digging around it seems to be that the Graph API response is indeed JSON but it gets converted to an FBGraphObject which holds a NSMutableArray as it the data makes its way through the MonoTouch->.NET bindings so when I pulled _result.Description (equivalent to _result.ToString() it returned me the string representation of that object which happens to look a lot like JSON but is not. After finding all this out (and a lot of runtime experimentation), I was finally able to extract the data into a usable state by doing this:
static void OnPhotoAlbumsReceived(FBRequestConnection _connection, NSObject _result, NSError _error)
{
if (_error == null)
{
NSArray fieldData = (NSArray) _result.ValueForKeyPath(new NSString("albums.data.name"))
string[] names = NSArray.StringArrayFromHandle(fieldData.Handle);
// ...
}
}
Although this works for me, I have a feeling that there is a better or more robust way to get the data I requested, so if any developers out there can offer any additional tips for improving this solution, I would love to hear them.
As stated in Facebook SDK documentation Graph API:
When a request returns a non-JSON response (such as a "true" literal),
that response will be wrapped into a dictionary using this const as
the key. This only applies for very few Graph API prior to v2.1.
So you can check first if result is an NSDictionary, otherwise you can deserialize the JSON data as usual.
Below some obj-c code you can translate into C#/MonoTouch (I don't know the framework, I hope it is helpful).
NSDictionary *dict;
if ([graphResult isKindOfClass:[NSDictionary class]]) {
dict = (NSDictionary *)graphResult;
} else {
NSError *JSONError;
dict = [NSJSONSerialization JSONObjectWithData:graphResult options:NSJSONReadingAllowFragments error:&JSONError];
if (JSONError) {
NSLog(#"Facebook: JSON parse error: %#", JSONError);
// Handle error
}
}