whats the best/recommended RESTful way to handle large single Json response? - json

Question

When we have many documents to be returned from the response, we can paginate the result and give the link in the response to retrieve the next set of documents to be fetched from the API. However my case is different, each document in my response can be very large. How do I deal with such response ?
For Example consider the sample Json below:
{
"excludes": null,
"endDate": 1422918663,
"appliedText": "Applied text",
"id": "135699543",
...
"assignedTo": {
"productPart": [
{
"part": "1"
},
{
"part": "2"
},
{
"part": "3"
},
{
"part": "4"
},
...
]
}
...
}
If you see here, the productPart can grow to tens of thousands. How can I restrict such large response yet retain the structure.

Answer 1

Fetch product parts with a separate request (if you have the possibility) which then in turn can be paginated.
Alt. return the first x product parts and then provide a link where more product parts can be obtained.

Answer 2

I believe you should try something like JSON-Streaming.
Streaming Processing (also known as Incremental Processing) is the most efficient way to process JSON content. It has the lowest memory and processing overhead, and can often match performance of many binary data formats available on Java platform
Here is the main idea:
try {
InputStream in = [..];
JsonParser parser = new JsonFactory().createParser(in);
JsonToken token = null;
while ((token = parser.nextToken()) == JsonToken.FIELD_NAME) {
/*
* Parse parts and store them in client DB
* use later client side pagination
*
* {
* "part": "i"
* }
*/
}
} catch (Exception e) {
//
}