I'm attempting to import some json data into grafana via the JSON API.
Here's a snippet of the json structure I'm working with:
[
{
"origin": "TS",
"id": "M8C8E02434D442725422CCB337057792F",
"type": "1.5.1:1",
"self": "https://metricsourcehost01/uimapiM8C8E02434D442725422CCB337057792F",
"source": "destinationhost01.our.domain.net",
"target": "destinationhost01.our.domain.net-0",
"probe": "cdm",
"for_computer_system": {
"id": "14873",
"self": "https://metricsourcehost01/uimapi/devices/14873",
"name": "destinationhost01.our.domain.net",
"ip": "10.1.1.16"
},
"for_device": {
"id": "D4F3D290D787D3FA4E7CD2824BFA6B1C8",
"self": "https://metricsourcehost01/uimapi/devices/D4F3D290D787D3FA4E7CD2824BFA6B1C8"
},
"for_configuration_item": {
"id": "CCE5006B73554FE7D307C1A355429286A",
"self": "https://metricsourcehost01/uimapi/TBD/CCE5006B73554FE7D307C1A355429286A",
"name": "CPU-0",
"qosName": "QOS_CPU_MULTI_USAGE",
"description": "Individual CPU Usage",
"unit": "%"
},
"uimMetricDefinition": null,
"minSampleValue": 61.17,
"maxSampleValue": 72.78,
"meanSampleValue": 64.864,
"sample": [
{
"time": "2021-09-02T00:50:32.000Z",
"timeSinceEpoch": 1630543832,
"value": 61.17,
"rate": 60
},
{
"time": "2021-09-02T00:49:32.000Z",
"timeSinceEpoch": 1630543772,
"value": 63.52,
"rate": 60
},
{
"time": "2021-09-02T00:48:32.000Z",
"timeSinceEpoch": 1630543712,
"value": 62.79,
"rate": 60
},
{
"time": "2021-09-02T00:47:32.000Z",
"timeSinceEpoch": 1630543652,
"value": 64.06,
"rate": 60
},
{
"time": "2021-09-02T00:46:32.000Z",
"timeSinceEpoch": 1630543592,
"value": 72.78,
"rate": 60
}
]
},
{
"origin": "TS",
"id": "M9D90857B9F9BE73EB15912D3314DB2DA",
"type": "1.5.1:1",
"self": "https://metricsourcehost01/uimapiM9D90857B9F9BE73EB15912D3314DB2DA",
"source": "destinationhost01.our.domain.net",
"target": "destinationhost01.our.domain.net-1",
"probe": "cdm",
"for_computer_system": {
"id": "14873",
"self": "https://metricsourcehost01/uimapi/devices/14873",
"name": "destinationhost01.our.domain.net",
"ip": "10.1.1.16"
},
"for_device": {
"id": "D4F3D290D787D3FA4E7CD2824BFA6B1C8",
"self": "https://metricsourcehost01/uimapi/devices/D4F3D290D787D3FA4E7CD2824BFA6B1C8"
},
"for_configuration_item": {
"id": "CF1D7A708DD4C6C9D303025AE3D2334AE",
"self": "https://metricsourcehost01/uimapi/TBD/CF1D7A708DD4C6C9D303025AE3D2334AE",
"name": "CPU-1",
"qosName": "QOS_CPU_MULTI_USAGE",
"description": "Individual CPU Usage",
"unit": "%"
},
"uimMetricDefinition": null,
"minSampleValue": 59.85,
"maxSampleValue": 72.31,
"meanSampleValue": 64.296,
"sample": [
{
"time": "2021-09-02T00:50:32.000Z",
"timeSinceEpoch": 1630543832,
"value": 59.85,
"rate": 60
},
{
"time": "2021-09-02T00:49:32.000Z",
"timeSinceEpoch": 1630543772,
"value": 63.88,
"rate": 60
},
{
"time": "2021-09-02T00:48:32.000Z",
"timeSinceEpoch": 1630543712,
"value": 60.17,
"rate": 60
},
{
"time": "2021-09-02T00:47:32.000Z",
"timeSinceEpoch": 1630543652,
"value": 65.27,
"rate": 60
},
{
"time": "2021-09-02T00:46:32.000Z",
"timeSinceEpoch": 1630543592,
"value": 72.31,
"rate": 60
}
]
}
]
It's CPU utilisation for 2 CPU cores from the same host.
Using $.[*].sample[*].time and $.[*].sample[*].value successfully returns the required time and value data which can be easily graphed:
| time | value |
| ------------------------ | ----- |
| 2021-09-02T00:50:32.000Z | 61.17 |
| 2021-09-02T00:49:32.000Z | 63.52 |
| 2021-09-02T00:48:32.000Z | 62.79 |
| 2021-09-02T00:47:32.000Z | 64.06 |
| 2021-09-02T00:46:32.000Z | 72.78 |
| 2021-09-02T00:50:32.000Z | 59.85 |
| 2021-09-02T00:49:32.000Z | 63.88 |
| 2021-09-02T00:48:32.000Z | 60.17 |
| 2021-09-02T00:47:32.000Z | 65.27 |
| 2021-09-02T00:46:32.000Z | 72.31 |
However, it combines all the data with no way to differenciate between the two CPU core data samples.
I've been trying to figure out a way to get a third column utilising the target value for each iteration of the sample array.
Ideally, the output should look like this when tabled:
| target | time | value |
| -------------------------------------- | ------------------------ | ----- |
| destinationhost01.our.domain.net-**0** | 2021-09-02T00:50:32.000Z | 61.17 |
| destinationhost01.our.domain.net-**0** | 2021-09-02T00:49:32.000Z | 63.52 |
| destinationhost01.our.domain.net-**0** | 2021-09-02T00:48:32.000Z | 62.79 |
| destinationhost01.our.domain.net-**0** | 2021-09-02T00:47:32.000Z | 64.06 |
| destinationhost01.our.domain.net-**0** | 2021-09-02T00:46:32.000Z | 72.78 |
| destinationhost01.our.domain.net-**1** | 2021-09-02T00:50:32.000Z | 59.85 |
| destinationhost01.our.domain.net-**1** | 2021-09-02T00:49:32.000Z | 63.88 |
| destinationhost01.our.domain.net-**1** | 2021-09-02T00:48:32.000Z | 60.17 |
| destinationhost01.our.domain.net-**1** | 2021-09-02T00:47:32.000Z | 65.27 |
| destinationhost01.our.domain.net-**1** | 2021-09-02T00:46:32.000Z | 72.31 |
Any advice would be greatly appreciated. I'm not sure it's even doable with jsonpath... hence why i'm reaching out to the experts.
Thanks
As the JSON API for Grafana uses the JSONPath Plus package, it's quite easy to accomplish what I was after.
The ^ is able to grab the parent of any matching item. Playing around with this in the JSONPath Demo site got me there. You can paste in my example from the original post and test the following queries:
$.[*].sample[*].time obtains the time from each sample.
$.[*].sample[*].value obtains the value from each sample.
$.[*].sample[*].value^^^^.for_configuration_item.name is the special sauce that will grab the for_configuration_item.name for each sample
Providing these three queries to Grafana makes a table like this:
core
time
value
CPU-0
2021-09-02T00:50:32.000Z
61.17
CPU-0
2021-09-02T00:49:32.000Z
63.52
CPU-0
2021-09-02T00:48:32.000Z
62.79
CPU-0
2021-09-02T00:47:32.000Z
64.06
CPU-0
2021-09-02T00:46:32.000Z
72.78
CPU-1
2021-09-02T00:50:32.000Z
59.85
CPU-1
2021-09-02T00:49:32.000Z
63.88
CPU-1
2021-09-02T00:48:32.000Z
60.17
CPU-1
2021-09-02T00:47:32.000Z
65.27
CPU-1
2021-09-02T00:46:32.000Z
72.31
From there, using the Group by feature in the Experimental tab on the core column graphs the values exactly as required.
Related
I am trying to fetch nested JSON objects and JSON List from Database using QueryDSL. I have used a native query with LISTAGG and JSON_OBJECT.
Native Query :
SELECT b.id,b.bankName,b.account,b.branch,(select CONCAT(CONCAT('[',LISTAGG(JSON_OBJECT('accountId' value c.accountId, 'name' value customer_name,'amount' value c.amount),',')),']') from CUSTOMER_DETAILS c where c.bankId = b.id) as customers from BANK_DETAILS b
BANK_DETAILS
+----+---------+---------+----------+
| id | BankName| account | branch |
+----+---------+---------+----------+
| 1 | bank1 | savings | branch1 |
| 2 | bank2 | current | branch2 |
+----+---------+---------+----------+
CUSTOMER_DETAILS
+----+-----------+---------------+----------+-----------+
| id | accountId | customer_name | amount | BankId |
+----+-----------+---------------+----------+-----------+
| 1 | 50123 | Abc1 | 150000 | 1 |
| 2 | 50124 | Abc2 | 25000 | 1 |
| 3 | 50125 | Abc3 | 50000 | 2 |
| 4 | 50126 | Abc4 | 250000 | 2 |
+----+-----------+---------------+----------+-----------+
Expected Output for the above tables
[{
"id": "1",
"bankName": "bank1",
"account": "savings",
"branch": "branch1",
"customers": [
{
"accountId": "50123",
"Name": "Abc1",
"amount": 150000
},
{
"accountId": "50124",
"Name": "Abc2",
"amount": 25000
},
]
},{
"id": "2",
"bankName": "bank3",
"account": "current",
"branch": "branch2",
"customers": [
{
"accountId": "50125",
"name": "Abc3",
"amount": 50000
},
{
"accountId": "50126",
"Name": "Abc4",
"amount": 250000
},
]
}]
i have tried with writing this native query in QueryDSL with the below multiple queries for make the same expected output with the forEach loop.
class Repository {
private SQLQueryFactory queryFactory;
public Repository (SQLQueryFactory queryFactory){
this.queryFactory = queryFactory;
}
public void fetchBankDetails(){
List<BankDetails> bankList = queryFactory.select(QBankDetails.bankDetails)
.from(QBankDetails.bankDetails);
bankList.forEach(bankData ->{
List<CustomerDetails> customerList = queryFactory.select(QCustomerDetails.customerDetails)
.from(QCustomerDetails.customerDetails)
.where(QCustomerDetails.customerDetails.bankId.eq(bankData.bankId));
bankData.setCustomerList(customerList)
});
System.out.println(bankList);
}
}
I need to improve my code and convert it into a single query using QueryDSL to return the expected output
Is there any other way or any suggestions?
{
"schema": {
"type": "struct",
"fields": [
{
"type": "int32",
"optional": true,
"field": "c1"
},
{
"type": "string",
"optional": true,
"field": "c2"
},
{
"type": "int64",
"optional": false,
"name": "org.apache.kafka.connect.data.Timestamp",
"version": 1,
"field": "create_ts"
},
{
"type": "int64",
"optional": false,
"name": "org.apache.kafka.connect.data.Timestamp",
"version": 1,
"field": "update_ts"
}
],
"optional": false,
"name": "foobar"
},
"payload": {
"c1": 67,
"c2": "foo",
"create_ts": 1663920002000,
"update_ts": 1663920002000
}
}
I have my json string in this format and I don't want the whole data into data into table , wanted the table in this format.
| c1 | c2 | create_ts | update_ts |
+------+------+------------------+---------------------+
| 1 v| foo | 2022-09-21 10:47:54 | 2022-09-21 10:47:54 |
| 28 | foo | 2022-09-21 13:16:45 | 2022-09-21 13:16:45 |
| 29 | foo | 2022-09-21 14:19:10 | 2022-09-21 14:19:10 |
| 30 | foo | 2022-09-21 14:19:20 | 2022-09-21 14:19:20 |
| 31 | foo | 2022-09-21 14:29:19 | 2022-09-21 14:29:19 |
Skip other (nested) attributes by specifying the only one you want to see in the resulting output:
(
spark
.read
.option("multiline","true")
.json("/path/json-path")
.select("payload.*")
.show()
)
The immediate answer to the question is to use json_objectfunction.
However this function is not avliable on my DB as it's older version.
We have plans to upgrade but it will take a while.
How do I convert this:
SELECT name, phone, order_id FROM orders;
| name | phone | order_id
| Jack | 12345 | 120
| Jack | 12345 | 121
To this:
[
{
"name": "Jack",
"order_id": "120",
"phone": 12345
},
{
"name": "Jack",
"order_id": "121",
"phone": 12345
}
]
In a SQL query without using json_object function
I have a source data which contains the following type of a JSON array:
[
[
"source 1",
250
],
[
"other source",
58
],
[
"more stuff",
42
],
...
]
There can be 1..N pairs of strings and values like this. How can I sum all the values together from this JSON?
You can use FLATTEN, it will produce a single row for each element of the input array. Then you can access the number in that element directly.
Imagine you have this input table:
create or replace table input as
select parse_json($$
[
[
"source 1",
250
],
[
"other source",
58
],
[
"more stuff",
42
]
]
$$) as json;
FLATTEN will do this:
select index, value from input, table(flatten(json));
-------+-------------------+
INDEX | VALUE |
-------+-------------------+
0 | [ |
| "source 1", |
| 250 |
| ] |
1 | [ |
| "other source", |
| 58 |
| ] |
2 | [ |
| "more stuff", |
| 42 |
| ] |
-------+-------------------+
And so you can just use VALUE[1] to access what you want
select sum(value[1]) from input, table(flatten(json));
---------------+
SUM(VALUE[1]) |
---------------+
350 |
---------------+
I have table prices:
+----+----------+-----------------------------------------------------------
| id | reseller | price |
+----+----------+-----------------------------------------------------------
| 20 | 1 | {"1": "4.00", "2": "5.00", "3": "3.00", "4": "2.00", "5": "1.00", "18": "0.00", "28": "0.00"} |
| 21 | 6 | {"1": "0.00", "2": "0.00", "3": "0.00", "4": "0.00", "5": "0.00", "18": "1.00", "28": "0.00"} |
| 22 | 2 | {"1": "0.00", "2": "0.00", "3": "0.00", "4": "0.00", "5": "0.00", "18": "0.00", "28": "0.00"} |
+----+----------+-----------------------------------------------------------------------------------------------+
And need to get only values that are not 0.00 (to show in ma form only values that have defined price)...so i need to get this from output:
+----+----------+-----------------------------------------------------------
| id | reseller | price |
+----+----------+------------------------------------------------------------
| 20 | 1 | {"1": "4.00", "2": "5.00", "3": "3.00", "4": "2.00", "5": "1.00"}
-----------------------------------------------------------------------------
I try only using this:
SELECT id, JSON_EXTRACT(price, '$.1') FROM prices WHERE reseller=1;
to try to get first index value witch is 4.00 but i im getting error:
ERROR 3143 (42000): Invalid JSON path expression. The error is around character position 3.
What sql query i need to do to show only values that have defined price and value is not 0.00?
Use JSON_EXTRACT(price, '$."1"') and you able to run that query without the error, but I agre with Sloan, I don't sure if you can select ONLY the json values diferent to "0.00" in a single simple query.
This is your query fixed:
SELECT id, reseller, JSON_EXTRACT(price, '$."1"') AS price FROM prices WHERE reseller=1;
+----+----------+---------------------------
| id | reseller | price |
+----+----------+---------------------------
| 20 | 1 | "4.00" |
--------------------------------------------