Retrofit 2 streaming large JSON response - json

Question

I'm trying to parse / process a JSON response while receiving (streaming / chunked). But I cannot get this to work. When I receive JSON I can only process it after the whole response has been received.
If I test the same code with a static file download, this is working correctly.
Retrofit: 2.6.1
OkHTTP 3.12.0
When I download the 5MB file the notice 'Call OK' will be shown immediately and the 'download complete' later on. For the JSON files the 'Call OK' will take a while.
Logger.info("Start download")
val response = fileApi.download5MBFile() // Streaming
val response = dataHubApi.download3MBJson() //Does not stream
val response = articleApi.download10MBJson() // Does not stream
if (response.isSuccessful) {
Logger.info("Call ok")
val input = response.body()?.byteStream()
val buffer = ByteArray(8192)
var size = 0
while (true) {
val read = input!!.read(buffer)
if (read == -1) {
break
}
size += read
//Logger.info("Progress: ${size/1024/1024}mb")
}
Logger.info("Download complete")
} else {
Logger.info("Call not ok")
}
The factory methods for creating the APIs are all the same like this:
fun create(): FileApi {
val logLevel = Level.HEADERS
val retrofit = Retrofit.Builder()
.baseUrl(URL)
.client(getOkHttpClient(logLevel))
.build()
return retrofit.create(FileApi::class.java)
}
private fun getOkHttpClient(logLevel: Level): OkHttpClient {
return OkHttpClient.Builder()
.readTimeout(60L, TimeUnit.SECONDS) // default retrofit value is 10sec
.build()
}
And all the interface are also in the same format:
interface FileApi {
#GET("/5MB.zip")
#Streaming
suspend fun download5MB(): Response<ResponseBody>
}
What can I do to also stream the JSON?

Answer 1

If it is that large, probably it is because it contains a list of several items instead of one or a small chunk of elements.
[
{ item1 },
{ item2 },
...
]
Even if it is something as a firebase json representation you'll have something as:
{
"key1": {element },
"key2": {element },
"key3": { element },
...
}
If you have trouble parsing the complete json, there is a good chance that you'll also face some trouble saving that into a kotlin list, so the best thing would be to directly save it into a DB where you can manage individual objects or pages of objects later on.
So, the trick is to download the JSON as a streamed text that you parse as you stream, and determine when each element is split, so you will only have to process the element as JSON on the text chunk instead of all the text.
so for example with a stack you may start adding braces, but once you reach 0 level stack, you surely has the stream of one element and parse the text in it.