Ad-hoc queries to a massive JSON dataset - json

Question

I have a massive dataset stored in Azure BLOB in JSON format. Some apps are constantly adding new data to it. BLOBs are organized in partitions like
/dataset={name}/date={YYYY-MM-DD}/one_or_more_json_files
Data pieces do not follow any particular schema. JSON field names are not in consistent letter case. Some JSON rows can be broken.
Could someone advise a good way to query this data without defining schema in advance. I would like to do something like
select * from my_huge_json_dataset where dataset='mydataset' and date>'2015-04-01'
without defining explicit schema for the table
My first consideration was HIVE but it turns out that SerDe needs schema to be defined to create a table. json_tuple could be an answer but it is case-sensitive and crashes if meets malformed json row.
I am also considering Apache Drill and Pig but have no experience with them and would like some guidance.

Answer 1

You could use Apache Drill, you only need to configure a new storage pointing to your dataset folder:
{
"type": "file",
"enabled": true,
"connection": "file:///",
"config": null,
"workspaces": {
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": null
},
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null
}
},
"formats": {
"json": {
"type": "json",
"extensions": [
"json"
]
}
}
So, if you defined that Storage Plugin as 'dfs' for example you could query from the root directory without defining any schema using ANSI SQL, just like:
SELECT * FROM dfs.dataset.date.`file.json`;
or even filter by your folder name in the same query using dir0.
I encourage you to visit their documentation site Apache Drill documentation in your case specially Querying JSON files