Expanding a record with unknown keys in Power Query - json

I am working with a nested json file. The issue is that the keys of the nested json are dates and their value is not known beforehand. Therefore I am unable to apply expandRecordColumn method on it.
Each row has a unique refId and looks like this
{
"refId" : "XYZ",
"snapshotIndexes" : {
"19-07-2021" : {
url: "abc1",
value: "123"
},
"20-07-2021" : {
url: "abc2",
value: "567"
}
}
}
I finally want a table with these columns,
refid | date | url | value
XYZ | 19-7-2021 | abc1 | 123
XYZ | 20-7-2021 | abc2 | 567
PQR | 7-5-2021 | srt | 999
In the new table, refId and date will together make a unique entry.
This is powerBi snapshot
Records

I was able to solve it using Record.ToTable on each row to convert from record to table and then applying ExpandTableColumn
let
Source = DocumentDB.Contents("sourceurl"),
Source = Source{[id="dbid"]}[Collections],
SourceTable= Source{[db_id="dbid",id="PartnerOfferSnapshots"]}[Documents],
ExpandedDocument = Table.ExpandRecordColumn(SourceTable, "Document", {"refId", "snapshotIndexes"}, {"Document.refId", "Document.snapshotIndexes"}),
TransformColumns = Table.TransformColumns(ExpandedDocument,{"Document.snapshotIndexes", each Table.ExpandRecordColumn(Record.ToTable(_), "Value", {"url","id","images"}, {"url","id","images"})}),
ExpandedTable = Table.ExpandTableColumn(TransformColumns, "Document.snapshotIndexes", {"Name","url","id","images"}, {"Document.dates","Document.url","Document.id","Document.images"})
in
ExpandedTable

Related

Karate API framework how to match the response values with the table columns?

I have below API response sample
{
"items": [
{
"id":11,
"name": "SMITH",
"prefix": "SAM",
"code": "SSO"
},
{
"id":10,
"name": "James",
"prefix": "JAM",
"code": "BBC"
}
]
}
As per above response, my tests says that whenever I hit the API request the 11th ID would be of SMITH and 10th id would be JAMES
So what I thought to store this in a table and assert against the actual response
* table person
| id | name |
| 11 | SMITH |
| 10 | James |
| 9 | RIO |
Now how would I match one by one ? like first it parse the first ID and first name from the API response and match with the Tables first ID and tables first name
Please share any convenient way of doing it from KARATE
There are a few possible ways, here is one:
* def lookup = { 11: 'SMITH', 10: 'James' }
* def items =
"""
[
{
"id":11,
"name":"SMITH",
"prefix":"SAM",
"code":"SSO"
},
{
"id":10,
"name":"James",
"prefix":"JAM",
"code":"BBC"
}
]
"""
* match each items contains { name: "#(lookup[_$.id+''])" }
And you already know how to use table instead of JSON.
Please read the docs and other stack-overflow answers to get more ideas.

Parsing JSON data from SQL Server table column

I am trying to parse JSON data from a table in SQL Server 2017. I have a view that returns this data:
| Debrief Name | Version | Answer Question | Answer Options |
+-------------------+-----------+--------------------------+--------------------------------------------------------------------------------------------------------------------------+
| Observer Report | 7 | Division: | {"Options":[{"Display":"Domestic","Value":"Domestic"},{"Display":"International","Value":"International"}]} |
| Observer Report | 7 | Are you on reserve? | {"Options":[{"Display":"Yes - Long Call Line","Value":"Yes"},{"Display":"No","Value":"No"}]} |
| Observer Report | 11 | Crew Position: | {"Options":[{"Display":"CA","Value":"CA"},{"Display":"RC","Value":"RC"},{"Display":"FO","Value":"FO"}]} |
| Observer Report | 11 | Domicile: | {"VisibleLines":2,"Options":[{"Display":"BOS","Value":"BOS"},{"Display":"CLT","Value":"CLT"}]} |
| Training Debrief | 12 | TRAINING CREW POSITION | {"VisibleLines":2,"Options":[{"Display":"CA","Value":"CA"},{"Display":"FO","Value":"FO"}]} |
| Training Debrief | 12 | AIRCRAFT | {"VisibleLines":2,"Options":[{"Display":"777","Value":"777"},{"Display":"767","Value":"767"}]} |
| Security Debrief | 9 | Aircraft Type | {"Options":[{"Display":"MD-80","Value":"MD-80"},{"Display":"777","Value":"777"},{"Display":"767/757","Value":"767/757"}]}|
| News Digest | 2 | Do you read Digest? | {"Options":[{"Display":"Yes","Value":"Yes"},{"Display":"No","Value":"No"}]} |
The Debrief Name column can have multiple records for same debrief name and Version. Also there are multiple versions for each debrief. And for each debrief name and version combination, there are set of Answer Questions and related Answer Options. Now the column Answer Options contain JSON record which I need to parse.
So my initial query that is something like below:
SELECT *
FROM [dbo].<MY VIEW>
WHERE [Debrief Name] = 'Observer Report' AND Version = 11
which would return below data:
| Debrief Name | Version | Answer Question | Answer Options |
+---------------------+--------------+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| Observer Report | 11 | Crew Position: | {"Options":[{"Display":"CA","Value":"CA"},{"Display":"RC","Value":"RC"}]} |
| Observer Report | 11 | Domicile: | {"VisibleLines":2,"Options":[{"Display":"BOS","Value":"BOS"},{"Display":"CLT","Value":"CLT"}]} |
| Observer Report | 11 | Fleet: | {"Options":[{"Display":"330","Value":"330"},{"Display":"320","Value":"320"}]} |
| Observer Report | 11 | Division: | {"Options":[{"Display":"Domestic","Value":"Domestic"},{"Display":"International","Value":"International"}]} |
| Observer Report | 11 | Are you on reserve? | {"Options":[{"Display":"Yes - Long Call Line","Value":"Yes - Long Call Line"},{"Display":"No","Value":"No"}]} |
Now from this returned result, for each Answer Question I need to parse the related Answer Options JSON data and extract the Value field for all the display attribute. So for example the JSON string in Answer Options for question "Are you on reserver?" looks like this:
"Options":[
{
"Display":"330",
"Value":"330",
"Selected":false
},
{
"Display":"320",
"Value":"320",
"Selected":false
},
{
"Display":"S80",
"Value":"S80",
"Selected":false
}
]
So I need to extract "Value" fields and return something like an array with values {330, 320, 195}.
In conclusion I want to construct a query where when I provide the Debrief Name and VersionNumber, it returns me the Answer Question and all the Answer Option values.
I am thinking of using a stored procedure like below:
CREATE PROCEDURE myProc
#DebriefName NVARCHAR(255),
#Version INT
AS
SELECT *
FROM [dbo].[myView]
WHERE [Debrief Name] = #DebriefName
AND Version = #Version
GO;
And then have another stored procedure that will capture this result from myProc and then do the JSON parsing:
CREATE PROCEDURE parseJSON
#DebriefName NVARCHAR(255),
#Version INT
AS
EXEC myProc #DebriefName, #Version; //Need to capture the result data in a temp table or something
// Parse the JSON data for each question item in temp table
GO;
I am not an expert in SQL so not sure how to do this. I read about Json parsing in SQL here and feel like I can use that but not sure how to in my context.
If you want to parse JSON data in Answer Options column and extract the Value field, you may try with the following approach, using OPENJSON() and STRING_AGG():
DECLARE #json nvarchar(max)
SET #json = N'{
"Options": [
{
"Display": "330",
"Value": "330",
"Selected": false
},
{
"Display": "320",
"Value": "320",
"Selected": false
},
{
"Display": "195",
"Value": "195",
"Selected": false
}
]
}'
SELECT STRING_AGG(x.[value], ', ') AS [Values]
FROM OPENJSON(#json, '$.Options') j
CROSS APPLY (SELECT * FROM OPENJSON(j.[value])) x
WHERE x.[key] = 'Value'
Output:
Values
330, 320, 195
If you want to build your statement using stored procedure, use this approach:
CREATE TABLE myTable (
DebriefName nvarchar(100),
Version int,
AnswerQuestion nvarchar(1000),
AnswerOptions nvarchar(max)
)
INSERT INTO myTable
(DebriefName, Version, AnswerQuestion, AnswerOptions)
VALUES
(N'Observer Report', 7, N'Division:' , N'{"Options":[{"Display":"Domestic","Value":"Domestic"},{"Display":"International","Value":"International"}]}'),
(N'Observer Report', 7, N'Are you on reserve?' , N'{"Options":[{"Display":"Yes - Long Call Line","Value":"Yes"},{"Display":"No","Value":"No"}]}'),
(N'Observer Report', 11, N'Crew Position:' , N'{"Options":[{"Display":"CA","Value":"CA"},{"Display":"RC","Value":"RC"},{"Display":"FO","Value":"FO"}]}'),
(N'Observer Report', 11, N'Domicile:' , N'{"VisibleLines":2,"Options":[{"Display":"BOS","Value":"BOS"},{"Display":"CLT","Value":"CLT"}]}'),
(N'Training Debrief', 12, N'TRAINING CREW POSITION', N'{"VisibleLines":2,"Options":[{"Display":"CA","Value":"CA"},{"Display":"FO","Value":"FO"}]}'),
(N'Training Debrief', 12, N'AIRCRAFT' , N'{"VisibleLines":2,"Options":[{"Display":"777","Value":"777"},{"Display":"767","Value":"767"}]}'),
(N'Security Debrief', 9, N'Aircraft Type' , N'{"Options":[{"Display":"MD-80","Value":"MD-80"},{"Display":"777","Value":"777"},{"Display":"767/757","Value":"767/757"}]}'),
(N'News Digest', 2, N'Do you read Digest?' , N'{"Options":[{"Display":"Yes","Value":"Yes"},{"Display":"No","Value":"No"}]}')
SELECT
t.AnswerQuestion,
STRING_AGG(x.[value], ', ') AS [Values]
FROM myTable t
CROSS APPLY (SELECT * FROM OPENJSON(t.AnswerOptions, '$.Options')) j
CROSS APPLY (SELECT * FROM OPENJSON(j.[value])) x
WHERE
DebriefName = N'Observer Report' AND
t.Version = 11 AND
x.[key] = 'Value'
GROUP BY
t.DebriefName,
t.Version,
t.AnswerQuestion
Output:
AnswerQuestion Values
Crew Position: CA, RC, FO
Domicile: BOS, CLT

How to format the structure of data returned from SQL query

I got this data from my SQL query:
addon_id | addon_name | addon_category_id
---------+------------+------------------
1 | abc | 10
2 | def | 20
3 | ghi | 10
Now I have to send this in the following JSON format and group the addons based on addon_category_id:
[
{
addon_category_id: 10,
addons:
[
{
addon_id: 1,
addon_name: abc
},
{
addon_id: 3,
addon_name: ghi
}
]
},
{
addon_category_id: 20
addons:
[
{
addon_id: 2,
addon_name: def
}
]
}
]
How can I do this? What is the logic behind that? Do I have to do it programmatically using a for loop or is there any other way?
As mentioned in the comments it depends on the programming language you use. In SQL Server 2016 you can use FOR JSON AUTO
SELECT b.addon_category_id ,addons.addon_id , addons.addon_name
FROM addon a
JOIN addon addons
ON a.addon_category_id = b.addon_category_id
FOR JSON AUTO;

Postgres 9.4: Include sibling column in jsonb array on SELECT

If I have a table like this:
office_id int
employees jsonb
and the data looks something like this:
1
[{ "name" : "John" }, { "name" : "Jane" }]
Is there an easy way to query so that the results look like this:
office_id,employees
1,[{ "name" : "John", "office_id" : 1 }, { "name" : "Jane", "office_id" : 1 }]
For example data, check out this sqlfiddle: http://sqlfiddle.com/#!15/ac37b/1/0
The results should actually look like this:
id employees
1 [{ "name" : "John", "office_id" : 1 }, { "name" : "Jane", "office_id" : 1 }]
2 [{ "name" : "Jamal", "office_id" : 1 }]
I've been reading through the json functions and it seems like it's possible, but I can't seem to figure it out. I would rather not have to store the office_id on each nested object.
Note: This is similar to my other question about jsonb arrays, but the desired output is different.
I'm not sure if you are selecting from a Postgres table or a json object table. Doing a normal query and converting it to json can be done with json_agg().
Here is a normal query:
ao_db=# SELECT * FROM record.instance;
id | created_by | created_on | modified_by | modified_on
--------------------------------------+------------+-------------------------------+-------------+-------------------------------
18d8ca56-87b6-11e5-9c15-48d22415d991 | sysop | 2015-11-10 23:19:47.181026+09 | sysop | 2015-11-10 23:19:47.181026+09
190a0e86-87b6-11e5-9c15-48d22415d991 | sysop | 2015-11-10 23:19:47.56517+09 | sysop | 2015-11-10 23:19:47.56517+09
57611c9c-87b6-11e5-8c4b-48d22415d991 | admin | 2015-11-10 23:21:32.399775+09 | admin | 2015-11-10 23:22:27.975266+09
(3 行)
Here is the same query passed through json_agg():
ao_db=# WITH j AS (SELECT * FROM record.instance) SELECT json_agg(j) FROM j;
json_agg
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[{"id":"18d8ca56-87b6-11e5-9c15-48d22415d991","created_by":"sysop","created_on":"2015-11-10T23:19:47.181026+09:00","modified_by":"sysop","modified_on":"2015-11-10T23:19:47.181026+09:00"}, +
{"id":"190a0e86-87b6-11e5-9c15-48d22415d991","created_by":"sysop","created_on":"2015-11-10T23:19:47.56517+09:00","modified_by":"sysop","modified_on":"2015-11-10T23:19:47.56517+09:00"}, +
{"id":"57611c9c-87b6-11e5-8c4b-48d22415d991","created_by":"admin","created_on":"2015-11-10T23:21:32.399775+09:00","modified_by":"admin","modified_on":"2015-11-10T23:22:27.975266+09:00"}]

JSON issue in Big Query

I have scenario to parse the json data which is of one column in table.
Issue is that below Response column as json generated by Datastore backup to BigQuery. It has '\'attached to every data.
Reponse": "[
{
\"questionId\":5121566669012992,
\"answereId\":0,
\"answeredText\":\"Summer\"
},{
\"questionId\":5166851730440192,
\"answereId\":0,
\"answeredText\":\"Barcelona\"
},{
\"questionId\":6304057064947712,
\"answereId\":0,
\"answeredText\":\"Kitesurf\"
}
]"
How do I parse the below to get value for questionId using BigQuery?
JSON_EXTRACT cannot return REPEATED field, it can only do one match - hence no support for *
you can get the first position using hardcoded indexes as
SELECT JSON_EXTRACT_SCALAR('[
{
\"questionId\":5121566669012992,
\"answereId\":0,
\"answeredText\":\"Summer\"
},{
\"questionId\":5166851730440192,
\"answereId\":0,
\"answeredText\":\"Barcelona\"
},{
\"questionId\":6304057064947712,
\"answereId\":0,
\"answeredText\":\"Kitesurf\"
}
]', '$[0].questionId') AS str;
This returns:
+-----+------------------+---+
| Row | str | |
+-----+------------------+---+
| 1 | 5121566669012992 | |
+-----+------------------+---+