Multi Object JSON containing static and dynamic data - json

I have more than 1 million rows of data in excel and I want to convert it to JSON so i can visualize it using D3js and other web based applications. Data is comprised of two subsets:
General information of each data points, including geographical location, ID.... (static data not changing once it is written)
Monthly measurements at each data point. This data set updates monthly once new data arrives
This is how data looks like:
ID: 2411976, State: Texas, County: DEWITT, Latitude: 29 Longitude:-96,
Data: 11/1/2013 27.516; 12/1/2013 15.3566; 1/1/2014 27.6418;
2/1/2014 13.45; 3/1/2014 11.21; 4/1/2014 20
ID: 2321771, State: Texas, County: DEWITT, Latitude: 29 Longitude:-96,
Data: 11/1/20134 19; 12/1/2014 21; 1/1/2015 30; 2/1/2015 50; 3/1/2015 10;
4/1/2015 5
.....
Is it possible to wrap all data points in one JSON document that contains both temporal data and static data?

This is indeed possible, as you can represent arrays/objects in a nested structure, like this:
{
"locations": [{
"id": 2411976,
"state": "Texas",
"county": "DEWITT",
"latitude": 29,
"longitude":-96,
"data": [{
"date": "2013-11-01T00:00:00.000Z",
"value": 27.516
}, {
"date": "2013-12-01T00:00:00.000Z",
"value": 15.3566
}, {
"date": "2014-01-01T00:00:00.000Z",
"value": 27.6418
}, {
"date": "2014-02-01T00:00:00.000Z",
"value": 13.45
}, {
"date": "2014-03-01T00:00:00.000Z",
"value": 11.21
}, {
"date": "2014-04-01T00:00:00.000Z",
"value": 20
}]
}, {
"id": 2321771,
"state": "Texas",
"county": "DEWITT",
"latitude": 29,
"longitude":-96,
"data": [{
"date": "2014-11-01T00:00:00.000Z",
"value": 19
}, {
"date": "2014-12-01T00:00:00.000Z",
"value": 21
}, {
"date": "2015-01-01T00:00:00.000Z",
"value": 30
}, {
"date": "2015-02-01T00:00:00.000Z",
"value": 50
}, {
"date": "2015-03-01T00:00:00.000Z",
"value": 10
}, {
"date": "2015-04-01T00:00:00.000Z",
"value": 5
}]
}]
}
This is just one way of doing it. Depending on what the consumer of this data expects as input, you could adapt accordingly.

Related

Update query to convert existing complex json object to array using SQL

I have a table that contains a JSON column. Currently, the JSON column holds data which has the below schema :
Input or Source JSON looks like below :
"Q1": {
"id": 1,
"value": 12
},
"Q2": [{
"id": 1,
"value": 12
}, {
"id": 1,
"value": 12
}],
"Q3": {
"id": 1,
"additional": true
}
}
I have to perform a migration activity wherein I need to convert the Q1 key's value type from JSON object to JSON array. I don't see any utility functions which can help me in MySQL. Please suggest a suitable approach that can help me.
Result JSON should look like below
{
"Q1": [{
"id": 1,
"value": 12
}],
"Q2": [{
"id": 1,
"value": 12
}, {
"id": 1,
"value": 12
}],
"Q3": {
"id": 1,
"additional": true
}
}

How should I structure this nested JSON database to import into firebase firestore?

I have a database for an eCommerce website in this structure:
{
"hats": {
"id": 1,
"title": "Hats",
"routeName": "hats",
"items": [
{
"id": 1,
"name": "Brown Brim",
"imageUrl": "https://i.ibb.co/ZYW3VTp/brown-brim.png",
"price": 25
},
{
"id": 2,
"name": "Blue Beanie",
"imageUrl": "https://i.ibb.co/ypkgK0X/blue-beanie.png",
"price": 18
}
]
},
"sneakers": {
"id": 2,
"title": "Sneakers",
"routeName": "sneakers",
"items": [
{
"id": 10,
"name": "Adidas NMD",
"imageUrl": "https://i.ibb.co/0s3pdnc/adidas-nmd.png",
"price": 220
}
]
},
"jackets": {
"id": 3,
"title": "Jackets",
"routeName": "jackets",
"items": [
{
"id": 18,
"name": "Black Jean Shearling",
"imageUrl": "https://i.ibb.co/XzcwL5s/black-shearling.png",
"price": 125
},
{
"id": 19,
"name": "Blue Jean Jacket",
"imageUrl": "https://i.ibb.co/mJS6vz0/blue-jean-jacket.png",
"price": 90
}
]
}
}
I'm trying to find a way to import it to firestore without too much hassle.
As you can see the items field should be a new sub-collection, but whenever I try to import it to firestore, I get this error:
{
status: false,
message: 'Value for argument "data" is not a valid Firestore document. Input is not a plain JavaScript object.'
}
In the meantime I managed to solve this issue, by importing my JSON file to Realtime Database, editing there, then exporting it to Firestore!

What would be the best way to format JSON data consumed by a SPA?

I'm working with a friend on a single page application (in React, but I believe that the framework doesn't really matter, the same question applies to Angular as well).
There is a database with 2 tables:
Feature
Car
Both tables are connected in the database with many-to-many relation.
We differ in how we should pass the data from the backend to the frontend (more precisely, CarManagementComponent that will let user work on cars/features (edit/update/delete etc)). We want to have ability to perform several actions before, actually, sending a request back to the backend to update the database so that the user has desktop application-like interface experience.
Please, keep in mind that there are more tables in the database but for the example's simplicity, we're talking here only about 2 of them.
1) My approach:
{
"Features": [
{
"Id": 0,
"Price": 3000,
"Name": "led lights",
"Color": "transparent",
"Brand": "Valeo",
"Guarantee": 12
},
{
"Id": 1,
"Price": 1000,
"Name": "air conditioning",
"Color": "",
"Brand": "Bosch",
"Guarantee": 12
},
{
"Id": 2,
"Price": 600,
"Name": "tinted windows",
"Color": "",
"Brand": "Bosch",
"Guarantee": 36
}
],
"Cars": [
{
"Id": 0,
"Name": "Ford Mustang GT",
"Weight": 2210,
"Features":[
{
"Id": 0, // id of many-to-many relations record
"FeatureId": 2
},
{
"Id": 1, // id of many-to-many relations record
"FeatureId": 1
}
]
},
{
"Id": 1,
"Name": "Volkswagen Arteon",
"Weight": 1650,
"Features":[
{
"Id": 2, // id of many-to-many relations record
"FeatureId": 2
}
]
}
]
}
2) My friend's approach:
{
"Cars": [
{
"Id": 0,
"Name": "Ford Mustang GT",
"Weight": 2210,
"Features": [
{
"Id": 1,
"Price": 1000,
"Name": "air conditioning",
"Color": "",
"Brand": "Bosch",
"Guarantee": 12
},
{
"Id": 2,
"Price": 600,
"Name": "tinted windows",
"Color": "",
"Brand": "Bosch",
"Guarantee": 36
}
]
},
{
"Id": 1,
"Name": "Volkswagen Arteon",
"Weight": 1650,
"Features": [
{
"Id": 2,
"Price": 600,
"Name": "tinted windows",
"Color": "",
"Brand": "Bosch",
"Guarantee": 36
}
]
}
]
}
I belive that the 1st approach is better because:
it weighs less (no data redundancy)
it would be easier to convert such data into object-oriented structure
eg. we are able to see all Feature records (in 2nd approach, we'd only see records that are being connected with Cars and another backend request would be needed)
eg. unlike the 2nd approach, we're able to obtain all the needed data in just 1 request (less problems with synchronization) and we could be saving modified data in a single request as well
My friend says 2nd approach is better because:
it'd be easier to achieve that using ORM (hibernate)
he's never seen 1st approach in his life (which could lead to a conclusion, that it's being done in a wrong way)
What do you think? Which solution is better? Maybe both of them in some areas? Maybe there's a 3rd solution we didn't think of yet?
I would say the approach I mostly like is yours for 2 main reasons:
Keeping in mind that data duplication is bad in an HTTP request, your approach is avoiding it.
You let the FeatureId inside the car object and it is enough to get the feature in an efficient performance O(N).
To make it even better, you could change your feature structure to this:
"Features": {
0: { // <- If the id is unique, you can use it as a key.
"Id": 0,
"Price": 3000,
"Name": "led lights",
"Color": "transparent",
"Brand": "Valeo",
"Guarantee": 12
},
1: {
"Id": 1,
"Price": 1000,
"Name": "air conditioning",
"Color": "",
"Brand": "Bosch",
"Guarantee": 12
},
2: {
"Id": 2,
"Price": 600,
"Name": "tinted windows",
"Color": "",
"Brand": "Bosch",
"Guarantee": 36
}
},
This way, you can get the Feature in O(1).

How to extract a JSON field that has a random string with the jq command?

Please bear with me I'm still learning about JSON and programming in general. So I have this JSON file:
{
"root_200888546292191": {
"fields": {
"buyerId": "31392191"
},
"id": "200718546292191",
"tag": "root",
"type": "biz"
},
"shippingInfo_#package#OF04472002179150#WAREHOUSE_ACCEPTED": {
"fields": {
"delivery": {
"createdAt": "Sen 09 Apr - Rab 11 Apr",
"desc": "Standar",
"email": null,
"method": "Standard",
"status": "info"
},
"statusMap": {
"active": "Dalam proses",
"all": ["Dalam proses", "Dalam pengiriman", "Telah diterima"]
},
"trackingList": [{
"info": "Status One",
"updatedAt": "05 Apr 2018 - 11:00"
}, {
"info": "Status Two",
"updatedAt": "05 Apr 2018 - 11:00"
}]
},
"id": "#package#OF04472002179150#WAREHOUSE_ACCEPTED",
"tag": "shippingInfo",
"type": "biz"
},
"shippingInfo_#package#AAAAAAAAAAAAA#NOT_WAREHOUSE_ACCEPTED": {
"fields": {
"delivery": {
"createdAt": "Sen 09 Apr - Rab 11 Apr",
"desc": "Standar",
"email": null,
"method": "Standard",
"status": "info"
},
"statusMap": {
"active": "Dalam proses",
"all": ["Dalam proses", "Dalam pengiriman", "Telah diterima"]
},
"trackingList": [{
"info": "Status Three",
"updatedAt": "05 Apr 2018 - 11:00"
}, {
"info": "Status Four",
"updatedAt": "05 Apr 2018 - 11:00"
}]
},
"id": "#package#AAAAAAAAAAAAA#NOT_WAREHOUSE_ACCEPTED",
"tag": "shippingInfo",
"type": "biz"
},
"login_200718577292191": {
"fields": {
"buyerEmail": "myemail#gmail.com",
"buyerName": "myname"
},
"id": "200718522292191",
"tag": "login",
"type": "biz"
}
}
And I want to extract Info in shippingInfo_ > fields > trackingList So the output that I want is like this:
Status One
Status Two
Status Three
Status Four
The string after shippingInfo_ is always random, how do I extract it with jq?
This is as far as I've got jq '.shippingInfo_*.fields.trackingList.info'
Direct approaches
There are many direct approaches, e.g.:
Using paths
paths as $p
| select( $p|length == 5 and
($p[0] | startswith("shippingInfo_")) and
$p[1:3] == ["fields", "trackingList"] and
$p[4] == "info")
| getpath($p)
Using to_entries
to_entries[]
| select(.key | startswith("shippingInfo_"))
| .value
| .fields.trackingList[]
| .info
Indirect approaches
There are also some indirect approaches that are worth mentioning, e.g.
Using a helper function
def dot(s):
to_entries[] | select(.key|test(s)) | .value ;
dot("^shippingInfo_")
| .fields.trackingList[]
| .info
The last-mentioned filter can be abbreviated to:
dot("^shippingInfo_").fields.trackingList[].info
Relaxed requirements
If it is acceptable to ignore the "^shippingInfo_" requirement, the following may be worth considering as well:
.[].fields.trackingList[]?.info
or even:
.. | objects.fields.trackingList[]?.info

Xpath to JSON - still needs tidying

I am trying to get the table of data from this website. and serve it out as JSON.
I am taking my first steps with Yahoo Pipes/XPath, so apologies in advance for the noob Qs.
I have a pipe that just uses an XPath expression:
//table1/tbody/tr
Which gives me the following JSON output (snippet, 1 of about 30 items):
"items": [{
"td": [{
"a": {
"href": "http:\/\/www.hydro.com.au\/system\/files\/water-storage\/Web_Lakes_AUGUSTA.pdf",
"content": "Lake Augusta"
}
},
{
"p": "2.58"
},
{
"p": "Steady"
}],
"description": null,
"title": null
},
What I want to end up with is a flattened out version, something like this:
"items": [{
{
"href": "http:\/\/www.hydro.com.au\/system\/files\/water-storage\/Web_Lakes_AUGUSTA.pdf",
},
{
"content": "Lake Augusta"
},
{
"p": "2.58"
},
{
"p": "Steady"
}],
},
Can someone help me achieve this? Should I be doing more with the original XPath, or is there a particular processor within Pipes that will help me accomplish this?
It's not possible to select the data structure you want directly with XPath.
However, it's not difficult to manipulate selected data in Yahoo Pipes, for example by combining a "Loop" module with a "Item Builder" module.
The result is a feed that looks like this:
{
"count": 43,
"value": {
"title": "Hydro Tasmania Lake Levels",
"description": "Hydro Tasmania Lake Levels\n\nfrom StackOverflow Question http://stackoverflow.com/q/23248288/18771",
"link": "http://pipes.yahoo.com/pipes/pipe.info?_id=d2fe64963964bf6636b58fb7b9814ef0",
"pubDate": "Fri, 25 Apr 2014 06:12:35 +0000",
"generator": "http://pipes.yahoo.com/pipes/",
"callback": "",
"items": [
{
"title": "Lake Augusta",
"link": "http://www.hydro.com.au/system/files/water-storage/Web_Lakes_AUGUSTA.pdf",
"meters": "2.58",
"comment": "Steady",
"description": null
},
... more ...
]
}
}