Access deeper elements of a JSON using postgresql 9.4 - json

I want to be able to access deeper elements stored in a json in the field json, stored in a postgresql database. For example, I would like to be able to access the elements that traverse the path states->events->time from the json provided below. Here is the postgreSQL query I'm using:
SELECT
data#>> '{userId}' as user,
data#>> '{region}' as region,
data#>>'{priorTimeSpentInApp}' as priotTimeSpentInApp,
data#>>'{userAttributes, "Total Friends"}' as totalFriends
from game_json
WHERE game_name LIKE 'myNewGame'
LIMIT 1000
and here is an example record from the json field
{
"region": "oh",
"deviceModel": "inHouseDevice",
"states": [
{
"events": [
{
"time": 1430247045.176,
"name": "Session Start",
"value": 0,
"parameters": {
"Balance": "40"
},
"info": ""
},
{
"time": 1430247293.501,
"name": "Mission1",
"value": 1,
"parameters": {
"Result": "Win ",
"Replay": "no",
"Attempt Number": "1"
},
"info": ""
}
]
}
],
"priorTimeSpentInApp": 28989.41467999999,
"country": "CA",
"city": "vancouver",
"isDeveloper": true,
"time": 1430247044.414,
"duration": 411.53,
"timezone": "America/Cleveland",
"priorSessions": 47,
"experiments": [],
"systemVersion": "3.8.1",
"appVersion": "14312",
"userId": "ef617d7ad4c6982e2cb7f6902801eb8a",
"isSession": true,
"firstRun": 1429572011.15,
"priorEvents": 69,
"userAttributes": {
"Total Friends": "0",
"Device Type": "Tablet",
"Social Connection": "None",
"Item Slots Owned": "12",
"Total Levels Played": "0",
"Retention Cohort": "Day 0",
"Player Progression": "0",
"Characters Owned": "1"
},
"deviceId": "ef617d7ad4c6982e2cb7f6902801eb8a"
}
That SQL query works, except that it doesn't give me any return values for totalFriends (e.g. data#>>'{userAttributes, "Total Friends"}' as totalFriends). I assume that part of the problem is that events falls within a square bracket (I don't know what that indicates in the json format) as opposed to a curly brace, but I'm also unable to extract values from the userAttributes key.
I would appreciate it if anyone could help me.
I'm sorry if this question has been asked elsewhere. I'm so new to postgresql and even json that I'm having trouble coming up with the proper terminology to find the answers to this (and related) questions.

You should definitely familiarize yourself with the basics of json
and json functions and operators in Postgres.
In the second source pay attention to the operators -> and ->>.
General rule: use -> to get a json object, ->> to get a json value as text.
Using these operators you can rewrite your query in the way which returns correct value of 'Total Friends':
select
data->>'userId' as user,
data->>'region' as region,
data->>'priorTimeSpentInApp' as priotTimeSpentInApp,
data->'userAttributes'->>'Total Friends' as totalFriends
from game_json
where game_name like 'myNewGame';
Json objects in square brackets are elements of a json array.
Json arrays may have many elements.
The elements are accessed by an index.
Json arrays are indexed from 0 (the first element of an array has an index 0).
Example:
select
data->'states'->0->'events'->1->>'name'
from game_json
where game_name like 'myNewGame';
-- returns "Mission1"

select
data->'states'->0->'events'->1->>'name'
from game_json
where game_name like 'myNewGame';
This did help me

Related

Sort by nested fields in json in Power Automate

I'm trying to sort a JSON String in Power Automate by a nested field called "orderHint".
My JSON String looks like this:
[
{
"id": "5134",
"value": {
"isChecked": false,
"title": "This is another test",
"orderHint": "8585298133570680672PF"
},
"lastModifiedDateTime": "2022-12-23T11:06:28.4256622Z"
},
{
"id": "26576",
"value": {
"isChecked": true,
"title": "This is a test",
"orderHint": "8585498133570680672DE"
},
"lastModifiedDateTime": "2022-12-23T11:06:28.4256622Z"
}
]
When I'm trying to sort by "orderHint", I get an error:
"'The template language function 'sort' did not find the named sortField 'orderHint' on one or more objects in the array."
I'm using the following expression:
sort(variables('varArrayChecked'), 'value/orderHint')
Sorting by other fields works fine, e.g.:
sort(variables('varArrayChecked'), 'id')
Is there any way how I can sort by a nested field in a JSON String?
Thanks in advance!
You can use the Advanced Data Operations connector as it will do it for you in a single step.
The Flatten Object Array step is perfect for the payload you've provided.
You can see that it will take the data, flatten it and you have the ability to sort it on the way out (noting that the Array variable contains the exact JSON you provided in your question) ...
Note: Balance Output must be set to true in order for the sorting to occur.
Result
This is the resulting JSON order by orderHint ascending.
[
{
"id": "5134",
"lastModifiedDateTime": "2022-12-23T11:06:28",
"value/isChecked": false,
"value/orderHint": "8585298133570680672PF",
"value/title": "This is another test"
},
{
"id": "26576",
"lastModifiedDateTime": "2022-12-23T11:06:28",
"value/isChecked": true,
"value/orderHint": "8585498133570680672DE",
"value/title": "This is a test"
}
]
... and to show it in descending order (which is obvious, but simply change the sort order object value from Asc to Desc) ...
[
{
"id": "26576",
"lastModifiedDateTime": "2022-12-23T11:06:28",
"value/isChecked": true,
"value/orderHint": "8585498133570680672DE",
"value/title": "This is a test"
},
{
"id": "5134",
"lastModifiedDateTime": "2022-12-23T11:06:28",
"value/isChecked": false,
"value/orderHint": "8585298133570680672PF",
"value/title": "This is another test"
}
]

How to retrieve multiple data from jsonb column in postgres?

In PostgreSQL database I have a json column called json. Data inside look like below:
{
"Version": "0.0.0.1",
"Items": [
{
"Id": "40000000-0000-0000-0000-000000141146",
"Name": "apple",
"Score": 64,
"Value": 1430000
},
{
"Id": "40000000-0000-0000-0000-000000141147",
"Name": "grapefruit",
"Score": 58,
"Value": 1190000
},
{
"Id": "40000000-0000-0000-0000-000000141148",
"Name": "mango",
"Score": 41,
"Value": 170000
}
]
}
What I would like to do is retrieving all Score data from Items elements.
I was trying to use SQL code:
select
substring(json ->> 'Items' from '"Score": (\d*),') as score
from vegetables;
However that returns just the score from first element instead of 3. I was trying to use '\g' flag which supposed to find all results globally, but the code was not working.
Could anyone advise how to do that properly? Thanks in advance!
Considering that the data type of json field is jsonb then no need to use substring or regex, simply lateral join with jsonb_array_elements will do the required things for you. Try below query.
select x->>'Score' "Score" from vegetables
cross join lateral jsonb_array_elements(json->'Items') x
DEMO

Using Power Query to extract data from nested arrays in JSON

I'm relatively new to Power Query, but I'm pulling in this basic structure of JSON from a web api
{
"report": "Cost History",
"dimensions": [
{
"time": [
{
"name": "2019-11",
"label": "2019-11",
…
},
{
"name": "2019-12",
"label": "2019-12",
…
},
{
"name": "2020-01",
"label": "2020-01",
…
},
…
]
},
{
"Category": [
{
"name": "category1",
"label": "Category 1",
…
},
{
"name": "category2",
"label": "Category 2",
…
},
…
]
}
],
"data": [
[
[
40419.6393798211
],
[
191.44
],
…
],
[
[
2299.652439184997
],
[
0.0
],
…
]
]
}
I actually have 112 categories and 13 "times". I figured out how to do multiple queries to turn the times into column headers and the categories into row labels (I think). But the data section is alluding me. Because each item is a list within a list I'm not sure how to expand it all out. Each object in the date array will have 112 numbers and there will be 13 objects. If that all makes sense.
So ultimately I want to make it look like
2019-11 2019-20 2020-01 ...
Category 1 40419 2299
Category 2 191 0
...
First time asking a question on here, so hopefully this all makes sense and is clear. Thanks in advance for any help!
i am also researching this exact thing and looking for a solution. In PQ, it displays nested arrays as a list and there is a function to extract values choosing a separating characterenter image description here
So this becomes, this
enter image description here
= Table.TransformColumns(#"Filtered Rows", {"aligned_to_ids", each Text.Combine(List.Transform(_, Text.From), ","), type text})
However the problem i'm trying to solve is when the nested json has multiple values like this: enter image description here
And when these LIST are extracted then an error message is caused, = Table.TransformColumns(#"Extracted Values1", {"collaborators", each Text.Combine(List.Transform(_, Text.From), ","), type text})
Expression.Error: We cannot convert a value of type Record to type Text.
Details:
Value=
id=15890
goal_id=323
role_id=15
Type=[Type]
It seems the multiple values are not handled and PQ does not recognise the underlying structure to enable the columns to be expanded.

MYSQL JSON column change array order after saving

I am using JSON column type in MySQL database table. When I try to save JSON values in table column, the JSON array automatically re-order(shuffle)
I have following JSON:
{"TIMER_HEADER": [{"XX!TIMERHDR": "XXTIMERHDR", "VER": " 7", "REL": " 0", "COMPANYNAME": "XXX", "IMPORTEDBEFORE": "N", "FROMTIMER": "N", "COMPANYCREATETIME": "12423426"}, {"XX!HDR": "XXHDR", "PROD": "Qics for Wnows", "VER": "Version 6.0", "REL": "Release R", "IIFVER": "1", "DATE": "2018-01-20", "TIME": "1516520267", "ACCNTNT": "N", "ACCNTNTSPLITTIME": "0"}], "COLUMN_HEADER": ["!TIMEACT", "DATE", "JOB", "EMP", "ITEM", "PITEM", "DURATION", "PROJ", "NOTE", "BILLINGSTATUS"]}
After saving in JSON-column of MySql table, this becomes:
{"TIMER_HEADER": [{"REL": " 0", "VER": " 7", "FROMTIMER": "N", "COMPANYNAME": "XXX", "XX!TIMERHDR": "XXTIMERHDR", "IMPORTEDBEFORE": "N", "COMPANYCREATETIME": "12423426"}, {"REL": "Release R", "VER": "Version 6.0", "DATE": "2018-01-20", "PROD": "Qics for Wnows", "TIME": "1516520267", "IIFVER": "1", "XX!HDR": "XXHDR", "ACCNTNT": "N", "ACCNTNTSPLITTIME": "0"}], "COLUMN_HEADER": ["!TIMEACT", "DATE", "JOB", "EMP", "ITEM", "PITEM", "DURATION", "PROJ", "NOTE", "BILLINGSTATUS"]}
I need the same order as I have original, because there is an validation on 3rd party.
Please help. Thanks.
The items of arrays don't change order in MySQL JSON columns, but object key/value pairs might.
MySQL manual says:
To make lookups more efficient, it also sorts the keys of a JSON object. You should be aware that the result of this ordering is subject to change and not guaranteed to be consistent across releases.
There is no standard that says it has to be in a certain order, so probably the best thing to do is to speak to someone on 3rd party to adjust validation.
We are using blob for storage. JSON validation is handled by our application.
Using this blob column, keys are not sorted since it not optimized for JSON.
Please refer the blob docs for details. https://dev.mysql.com/doc/refman/8.0/en/blob.html

Elastic Search + JSON import (ELK Stack)

I'm currently trying to do a basic JSON file import into my ELK stack. I tried importing it directly via a POST request like this:
curl -XPOST http://localhost:9200/kwd_results/TS_Cart -d #/home/local/TS_Cart.json
ES says ok for the import, but when I'm trying to view the logs in Kibanna, they are not indexed by the nodes of the JSON file. I'm guessing I need like a template mapping to view it properly.
My JSON file looks like this:
{
"testResults": {
"FitNesseVersion": "v20160618",
"rootPath": "K1System.CountryDe.DriverFirefox.TestCases.MainFolder.TestVariants.SmokeTests_B2C.TS_Cart",
"result": [
{
"counts": {
"right": "16",
"wrong": "2",
"ignores": "3",
"exceptions": "1"
},
"date": "2017-05-10T00:01:11+02:00",
"runTimeInMillis": "117242",
"relativePageName": "TestCase_1",
"pageHistoryLink": "K1System.CountryDe.DriverFirefox.TestCases.MainFolder.TestVariants.SmokeTests_B2C.TS_Cart.B2CFreeCatalogueOrder?pageHistory&resultDate=20170510000111",
"tags": "de, at"
},
{
"counts": {
"right": "16",
"wrong": "0",
"ignores": "0",
"exceptions": "0"
},
"date": "2017-05-10T00:03:08+02:00",
"runTimeInMillis": "85680",
"relativePageName": "TestCase_2",
"pageHistoryLink": "K1System.CountryDe.DriverFirefox.TestCases.MainFolder.TestVariants.SmokeTests_B2C.TS_Cart.B2CGiftCardOrderWithAdvancePayment?pageHistory&resultDate=20170510000308",
"tags": "at, de"
}
],
"finalCounts": {
"right": "4",
"wrong": "1",
"ignores": "0",
"exceptions": "0"
},
"totalRunTimeInMillis": "482346"
}
}
Basically I would need rootPath to be used as an index, while having the following childs: counts, relativePageName, date and tags. Notice that I have two nodes that are childs of the result[] array.
Any help would be greatly appreciated!
Thank you.
Well, it's one JSON document so Elasticsearch treats it as such.
You'll need to (programmatically) split up the document into the right documents and then you can store them (potentially with one _bulk request).
For the index name:
Must be lowercase, so you'll need to cast that value.
Will you have many different root paths with jut a few docs each? Then you shouldn't make all of them an index since there is an overhead for each one of them (actually the underlying shards).