I am trying to create a query in SQL to retrieve DNS answer information so that I can visualize it in Grafana with the add of TimescaleDB. Right now, I am struggling to get postgres to query more than one element at a time. The structure of my JSON that I am trying to query looks like this:
{
"Z": 0,
"AA": 0,
"ID": 56559,
"QR": 1,
"RA": 1,
"RD": 1,
"TC": 0,
"RCode": 0,
"OpCode": 0,
"answer": [
{
"ttl": 19046,
"name": "i.stack.imgur.com",
"type": 5,
"class": 1,
"rdata": "i.stack.imgur.com.cdn.cloudflare.net"
},
{
"ttl": 220,
"name": "i.stack.imgur.com.cdn.cloudflare.net",
"type": 1,
"class": 1,
"rdata": "104.16.30.34"
},
{
"ttl": 220,
"name": "i.stack.imgur.com.cdn.cloudflare.net",
"type": 1,
"class": 1,
"rdata": "104.16.31.34"
},
{
"ttl": 220,
"name": "i.stack.imgur.com.cdn.cloudflare.net",
"type": 1,
"class": 1,
"rdata": "104.16.0.35"
}
],
"ANCount": 13,
"ARCount": 0,
"QDCount": 1,
"question": [
{
"name": "i.stack.imgur.com",
"qtype": 1,
"qclass": 1
}
]
}
There can be any number of answers, including zero, so I would like to figure out a way to query all answers. For example, I am trying to retrieve the ttl field from every index answer, and I can query a specific index, but have trouble querying all occurrences.
This works for querying a single index:
SELECT (data->'answer'->>0)::json->'ttl'
FROM dns;
When I looked around, I found this as a potential solution for querying all indices within the array, but it did not seem to work and told me "cannot extract elements from a scalar":
SELECT answer->>'ttl' ttl
FROM dns, jsonb_array_elements(data->'answer') answer, jsonb_array_elements(answer->'ttl') ttl
Using jsonb_array_elements() will give you a row for every object in the answer array. You can then dereference that object:
select a.obj->>'ttl' as ttl, a.obj->>'name' as name, a.obj->>'rdata' as rdata
from dns d
cross join lateral jsonb_array_elements(data->'answer') as a(obj)
I have an order_summary JSON column in my orders table with the following structure
{
"total": 16.895,
"products": { // products is an object of objects instead of array of objects
"98": {
"price": "2.400",
"quantity": 2,
"sub_total": 4.8,
"product_id": 100,
"variant_id": 98
},
"395": {
"price": "3.900",
"quantity": 1,
"sub_total": 3.9,
"product_id": 401,
"variant_id": 395
},
"732": {
"price": "7.695",
"quantity": 1,
"sub_total": 7.695,
"product_id": 754,
"variant_id": 732
}
}
}
What I am trying to do,
I have to calculate the total number of items sold for a given variant_id, So I want to fetch the sum the quantity field inside the products object.
The problem
Sadly the products field is not an array of objects instead its an object of objects. With the keys being variant_id itself. So my following query is not working
SELECT sum(order_summary->"$.products.quantity") FROM orders where order_summary->"$.products.variant_id" = 98 (which returns an empty result set)
I think this is because of the presence of the keys in the products array. So how can I tackle this?
I am using Mysql 8, kindly let me know if any more information is needed.
After some trial and error and with the help of a friend, I found the following solution,
SELECT SUM(order_summary -> '$.products."98".quantity') as sum
FROM orders
WHERE JSON_CONTAINS_PATH(order_summary, 'all', '$.products."98"')
I have seen proposed solutions that worked for others in the following pages but I can't get it to work for me using the jqplay browser shell:
https://github.com/stedolan/jq/issues/344
https://github.com/stedolan/jq/issues/345
https://github.com/stedolan/jq/issues/1304
Given this data:
{
"api_version": 4,
"error": null,
"result": [
{
"labelId": "ALL",
"labelName": "ALL",
"samples": 30104,
"avgResponseTime": 6.849,
"90line": 8,
"95line": 9,
"99line": 36,
"minResponseTime": 2,
"maxResponseTime": 1951,
"avgLatency": 5.287,
"geoMeanResponseTime": 5.484,
"stDev": 23.765,
"duration": 302,
"avgBytes": 110.224,
"avgThroughput": 99.682,
"medianResponseTime": 5,
"errorsCount": 0,
"errorsRate": 0,
"hasLabelPassedThresholds": null
},
{
"labelId": "3687c89fac2385d28d53b356d4785418",
"labelName": "100b 3600s Cache",
"samples": 7300,
"avgResponseTime": 6.028,
"90line": 7,
"95line": 8,
"99line": 11,
"minResponseTime": 2,
"maxResponseTime": 680,
"avgLatency": 6.021,
"geoMeanResponseTime": 5.203,
"stDev": 16.233,
"duration": 300,
"avgBytes": 16.581,
"avgThroughput": 24.333,
"medianResponseTime": 5,
"errorsCount": 0,
"errorsRate": 0,
"hasLabelPassedThresholds": null
},
{
"labelId": "f88f8ff81bf9b521134637639a0277be",
"labelName": "100b NonCache",
"samples": 729,
"avgResponseTime": 6.143,
"90line": 7,
"95line": 7,
"99line": 9,
"minResponseTime": 3,
"maxResponseTime": 877,
"avgLatency": 6.136,
"geoMeanResponseTime": 4.627,
"stDev": 32.817,
"duration": 295,
"avgBytes": 1.64,
"avgThroughput": 2.471,
"medianResponseTime": 4,
"errorsCount": 0,
"errorsRate": 0,
"hasLabelPassedThresholds": null
}
]
}
I originally attempted:
[.result[] | {labelName: .labelName, samples: .samples, avgResponseTime: .avgResponseTime, 90line: .90line, 95line: .95line, 99line: .99line, minResponseTime: .minResponseTime, maxResponseTime: .maxResponseTime, avgLatency: .avgLatency, geoMeanResponseTime: .geoMeanResponseTime, stDev: .stDev, durationSeconds: .durationSeconds, avgBytes: .avgBytes, avgThroughput: .avgThroughput, medianResponseTime: .medianResponseTime, errorCount: .errorsCount, errorRate: .errorsRate, hasLabelPassedThresholds: .hasLabelPassedThresholds}]
and got
jq: error: syntax error, unexpected LITERAL (Unix shell quoting issues?) at <top-level>, line 1:
[.result[] | {labelName: .labelName, samples: .samples, avgResponseTime: .avgResponseTime, 90line: .90line, 95line: .95line, 99line: .99line, minResponseTime: .minResponseTime, maxResponseTime: .maxResponseTime, avgLatency: .avgLatency, geoMeanResponseTime: .geoMeanResponseTime, stDev: .stDev, durationSeconds: .durationSeconds, avgBytes: .avgBytes, avgThroughput: .avgThroughput, medianResponseTime: .medianResponseTime, errorCount: .errorsCount, errorRate: .errorsRate, hasLabelPassedThresholds: .hasLabelPassedThresholds}]
jq: 1 compile error
exit status 3
Looking at the similar questions asked in the prior links, I attempted to fix with queries like this:
[.result[] | {labelName: .labelName, samples: .samples, avgResponseTime: .avgResponseTime, 90line: .[“90line”], 95line: .[“95line”], 99line: .[“99line”], minResponseTime: .minResponseTime, maxResponseTime: .maxResponseTime, avgLatency: .avgLatency, geoMeanResponseTime: .geoMeanResponseTime, stDev: .stDev, durationSeconds: .durationSeconds, avgBytes: .avgBytes, avgThroughput: .avgThroughput, medianResponseTime: .medianResponseTime, errorCount: .errorsCount, errorRate: .errorsRate, hasLabelPassedThresholds: .hasLabelPassedThresholds}]
to deal with the fields beginning with numeric characters. I still get the same syntax errors, however, so I'm unable to line up my expectations with what I'm seeing in those other solutions. I just can't figure out what I'm doing wrong, I've tried all kinds of other quoting or trying stuff like | tostring to no avail.
EDIT: in response to a proposed answer below:
Hmm, I just can't get it. To recap:
{
"api_version": 4,
"error": null,
"result": [
{
"labelId": "ALL",
"labelName": "ALL",
"samples": 30104,
"avgResponseTime": 6.849,
"90line": 8,
"95line": 9,
"99line": 36,
"minResponseTime": 2,
"maxResponseTime": 1951,
"avgLatency": 5.287,
"geoMeanResponseTime": 5.484,
"stDev": 23.765,
"duration": 302,
"avgBytes": 110.224,
"avgThroughput": 99.682,
"medianResponseTime": 5,
"errorsCount": 0,
"errorsRate": 0,
"hasLabelPassedThresholds": null
}
]
}
jq bit:
jq '[.result[] | {labelName: .labelName, samples: .samples, avgResponseTime: .avgResponseTime, minResponseTime: .minResponseTime, maxResponseTime: .maxResponseTime, avgLatency: .avgLatency, geoMeanResponseTime: .geoMeanResponseTime, stDev: .stDev, durationSeconds: .durationSeconds, avgBytes: .avgBytes, avgThroughput: .avgThroughput, medianResponseTime: .medianResponseTime, errorCount: .errorsCount, errorRate: .errorsRate, hasLabelPassedThresholds: .hasLabelPassedThresholds, “90line”: .[“90line”], “95line”: .[“95line”], “99line”: .[“99line”]}]'
Returns:
jq: error: syntax error, unexpected INVALID_CHARACTER (Unix shell quoting issues?) at , line 1:
You need to quote the key names of keys that begin with a numeral, e.g.
"90line": .["90line"]
Note also that the jq expression {"90line": .["90line"]} can be abbreviated to just {"90line"}.
Example
With your input:
$ jq '[.result[] | {labelName, "90line": .["90line"] } ]' input.json
[
{
"labelName": "ALL",
"90line": 8
},
{
"labelName": "100b 3600s Cache",
"90line": 7
},
{
"labelName": "100b NonCache",
"90line": 7
}
]
If your JSON looks like this (in a file named your.json):
{
"banana": {
"9zz": true
}
}
And you want to grab 9zz with jq, do
jq '.banana["9zz"]' your.json
In other words, the two following lines are identical, but the bottom one works with values beginning with a number:
jq '.one.two' your.json
jq '.one["two"]' your.json
My Json file looks something like this (it's huge, so this is just simplified):
{
"foo": {
"id": [
20,
1,
3,
4,
60,
1,
],
"times": [
330.89,
5.33,
353.89,
33.89,
14.5,
207.5,
]
},
"poo": {
"id": [
20,
1,
3,
4,
60,
1,
],
"times": [
3.5,
323.89,
97.7,
154.5,
27.5,
265.60,
]
}
}
I have a similar json file as the one above, but a much more complex one. What I want to do is to use the "time" and "id" data and perform an action for the right "id" at the exact time. So the variables id and times are actually mapped to each other (has the same index). Is there a method to take out the right id for the right time to perform an action without having too many complicated loops?
I am working on a RESTful APIs. One of our screen shows table with Grand total.
Below are two JSON responses for returning data
First
[
{
"name": "Richard",
"bank_balance": 3000,
"assets_worth": 4000,
"total": 7000
},
{
"name": "John",
"bank_balance": 1000,
"assets_worth": 2000,
"total": 3000
},
{
"name": "Total",
"bank_balance":4000,
"assets_worth": 6000,
"total": 10000
}
]
Second
{
"rows": [
{
"name": "Richard",
"bank_balance": 3000,
"assets_worth": 4000,
"total": 7000
},
{
"name": "John",
"bank_balance": 1000,
"assets_worth": 2000,
"total": 3000
}
],
"grand_total":
{
"name": "Total",
"bank_balance":4000,
"assets_worth": 6000,
"total": 10000
}
}
Which one is more correct considering REST standard?
REST is merely an architecture style for designing networked applications. It doesn't directly answer to your question on data structuring.
Personally I would go with the first approach (just without the total row) as grand total can be trivially calculated from the row data, resulting in something like:
[
{
name: "Richard",
bank_balance: 3000,
assets_worth: 4000,
total: 7000
},
{
name: "John",
bank_balance: 1000,
assets_worth: 2000,
total: 3000
}
]
I think the important design principle here is that your API should not be opinionated about data representation. Some applications that use your API may choose to display data in tabular format, while other applications may choose some other representations. A good API is able to cater equally well different applications (and use cases).