Read streaming json files in S3 using pyspark - json

Question

I am using python spark sql to read streaming files sent from Kinesis. The nested Json streams are saved and zipped in S3 bucket. These gzip files are not standard json format, it just has a bunch of json objects without comma to separate. And it does not have [ ] at the beginning nor the end of the file. Here is the sample, there are 2 json objects in the example, I expect each object could be a row in the output:
{
"id": "0001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate" },
{ "id": "1003", "type": "Blueberry" },
{ "id": "1004", "type": "Devil's Food" }
]
}
}
{
"id": "0002",
"type": "donut",
"name": "Cake",
"ppu": 0.65,
"batters":
{
"batter":
[
{ "id": "1221", "type": "Regular" },
{ "id": "1223", "type": "Chocolate" },
{ "id": "1225", "type": "Blueberry" },
{ "id": "1228", "type": "Devil's Food" }
]
}
}
I can use pyspark sqlContext.read.json('streaming.json') to read this file, but the function only returns first object. When I add [ ] and comma seperate to those json objects, it reads those 2 objects succeesfully. But in my case, I have 1 TB of files for each day, its hard to translate the streaming into standard json files. I am new to spark, is there a way to read those streaming json objects using spark or sparkSQL? The output I expect is json flatten csv or dataframe saved back to S3. Thanks a lot.