Postgres get search and get multiple array json to each row - json

I wants to get all subscriptions with interval "1 WEEK" from the following 'data' column
[
{
"id": "tran_6ac25129951962e99f28fa488993",
"amount": 1200,
"client": {
"id": "client_622bdf4cce2351f28243",
"subscription": [
{
"id": "sub_a67d59efb2bcbf73485a",
"amount": 3900,
"currency": "USD",
"interval": "1 WEEK"
},
{
"id": "sub_a67d59efb2bcbf73485a",
"amount": 3900,
"currency": "USD",
"interval": "1 WEEK"
}
]
},
"currency": "USD"
},
{
"id": "tran_xxxxxxx",
"amount": 1200,
"client": {
"id": "client_xxxxxx8243",
"subscription": [
{
"id": "sub_xxefb2bcbf73485a",
"amount": 3900,
"currency": "USD",
"interval": "1 Year"
},
{
"id": "sub_yyyyyb2bcbf73485a",
"amount": 3900,
"currency": "USD",
"interval": "1 WEEK"
}
]
},
"currency": "USD"
}
]
My table structure:
CREATE TABLE transactions
(
data json,
id bigserial NOT NULL,
created_date time without time zone,
CONSTRAINT transactions_pkey PRIMARY KEY (id)
)
In output I wants to get all "1 WEEk" subscription as rows. Above data should give 3 rows
I am using Postgres 9.3+

Its a nested query and I have tried writing it in as readable form as I can. I hope you can understand it -
select subscriptions from
(
select
cast
(
json_array_elements
(
json_array_elements(data)->'client'->'subscription'
)
as text
)
as subscriptions,
json_array_elements
(
json_array_elements(data)->'client'->'subscription'
)
->>'interval'
as intervals
from
transactions
)
as
xyz
where
intervals = '1 WEEK';
For information regarding these functions, you can refer to -
http://www.postgresql.org/docs/9.3/static/functions-json.html
Edit:-
As per performance requirements, I guess this will work better than the previous one -
select * from (
select cast (
json_array_elements (
json_array_elements(data)->'client'->'subscription'
) as text
) as subscription from transactions
) as temp
where subscription LIKE '%"interval":"1 WEEK"%';

Related

How to write JSON to Mysql?

sorry for my bad english.
I am inserting a json into mysql like this:
set #json = '[{"name":"ivan","city":"london","kurs":"1", },{"name":"lena","city":"tokio","kurs":"5"},{"name":"misha","city":"kazan","kurs":"3"}]';
select * from json_table(#json,'$[*]' columns(name varchar(20) path '$.name',
city varchar(20) path '$.city',
kurs varchar(20) path '$.kurs')) as jsontable;
But now there is a task to insert an unknown number of additional properties:
set #json = '[{"name":"ivan","city":"london","kurs":"1","options": [{
"ao_id": 90630,
"name": "Высота предмета",
"value": "3.7 см"
}, {
"ao_id": 90673,
"name": "Ширина предмета",
"value": "4 см"
}, {
"ao_id": 90745,
"name": "Ширина упаковки",
"value": "4 см"
}]},{"name":"lena","city":"tokio","kurs":"5", "options": [{
"ao_id": 90630,
"name": "Высота предмета",
"value": "9.7 см"
}]},{"name":"misha","city":"kazan","kurs":"3", "options": [{
"ao_id": 90999,
"name": "Высота",
"value": "5.7 см"
}]}]';
How can I best do this so that I can access the table in the future (search, index, output)?

How do you use JSON_QUERY with null json array inside of json object?

SELECT JSON_query([json], '$') from mytable
Returns fine the contents of [json] field
SELECT JSON_query([json], '$.Guid') from mytable
Returns null
SELECT JSON_query([json], '$.Guid[1]') from mytable
Returns null
I've also now tried:
SELECT JSON_query([json], '$[1].Guid')
SELECT JSON_query([json], '$[2].Guid')
SELECT JSON_query([json], '$[3].Guid')
SELECT JSON_query([json], '$[4].Guid')
and they all return null
So I'm stuck as to figuring out how create the path to get to the info. Maybe SQL Server json_query can't handle the null as the first array?
Below is the string that is stored inside of the [json] field in the database.
[
null,
{
"Round": 1,
"Guid": "15f4fe9d-403c-4820-8e35-8a8c8d78c33b",
"Team": "2",
"PlayerNumber": "78"
},
{
"Round": 1,
"Guid": "8e91596b-cc33-4ce7-bfc0-ac3d1dc5eb67",
"Team": "2",
"PlayerNumber": "54"
},
{
"Round": 1,
"Guid": "f53cd74b-ed5f-47b3-aab5-2f3790f3cd34",
"Team": "1",
"PlayerNumber": "23"
},
{
"Round": 1,
"Guid": "30297678-f2cf-4b95-a789-a25947a4d4e6",
"Team": "1",
"PlayerNumber": "11"
}
]
You need to follow the comments below your question. I'll just summarize them:
Probably the most appropriate approach in your case is to use OPENJSON() with explicit schema (the WITH clause).
JSON_QUERY() extracts a JSON object or a JSON array from a JSON string and returns NULL. If the path points to a scalar JSON value, the function returns NULL in lax mode and an error in strictmode. The stored JSON doesn't have a $.Guid key, so NULL is the actual result from the SELECT JSON_query([json], '$.Guid') FROM mytable statement.
The following statements provide a working solution to your problem:
Table:
SELECT *
INTO Data
FROM (VALUES
(N'[
null,
{
"Round": 1,
"Guid": "15f4fe9d-403c-4820-8e35-8a8c8d78c33b",
"Team": "2",
"PlayerNumber": "78",
"TheProblem": "doesn''t"
},
{
"Round": 1,
"Guid": "8e91596b-cc33-4ce7-bfc0-ac3d1dc5eb67",
"Team": "2",
"PlayerNumber": "54"
},
{
"Round": 1,
"Guid": "f53cd74b-ed5f-47b3-aab5-2f3790f3cd34",
"Team": "1",
"PlayerNumber": "23"
},
{
"Round": 1,
"Guid": "30297678-f2cf-4b95-a789-a25947a4d4e6",
"Team": "1",
"PlayerNumber": "11"
}
]')
) v (Json)
Statements:
SELECT j.Guid
FROM Data d
OUTER APPLY OPENJSON(d.Json) WITH (
Guid uniqueidentifier '$.Guid',
Round int '$.Round',
Team nvarchar(1) '$.Team',
PlayerNumber nvarchar(2) '$.PlayerNumber'
) j
SELECT JSON_VALUE(j.[value], '$.Guid')
FROM Data d
OUTER APPLY OPENJSON(d.Json) j
Result:
Guid
------------------------------------
15f4fe9d-403c-4820-8e35-8a8c8d78c33b
8e91596b-cc33-4ce7-bfc0-ac3d1dc5eb67
f53cd74b-ed5f-47b3-aab5-2f3790f3cd34
30297678-f2cf-4b95-a789-a25947a4d4e6

Select Json formatted like a report to a table using T-SQL

I have JSON stored in a SQL Server database table in the below format. I have been able to fudge a way to get the values I need but feel like there must be a better way to do it using T-SQL. The JSON is output from a report in the below format where the column names in "columns" correspond to the "rows"-"data" array values.
So column "Fiscal Month" corresponds to data value "11", "Fiscal Year" to "2019", etc.
{
"report": "Property ETL",
"id": 2648,
"columns": [
{
"name": "Fiscal Month",
"dataType": "int"
},
{
"name": "Fiscal Year",
"dataType": "int"
},
{
"name": "Portfolio",
"dataType": "varchar(50)"
},
{
"name": "Rent",
"dataType": "int"
}
],
"rows": [
{
"rowName": "1",
"type": "Detail",
"data": [
11,
2019,
"West Group",
10
]
},
{
"rowName": "2",
"type": "Detail",
"data": [
11,
2019,
"East Group",
10
]
},
{
"rowName": "3",
"type": "Detail",
"data": [
11,
2019,
"East Group",
10
]
},
{
"rowName": "Totals: ",
"type": "Total",
"data": [
null,
null,
null,
30
]
}
]
}
In order to get at the data in the 'data' array I currently have a 2 step process in T-SQL where I create a temp table, and insert the row key/values from '$.Rows' there. Then I can then select the individual columns for each row
CREATE TABLE #TempData
(
Id INT,
JsonData VARCHAR(MAX)
)
DECLARE #json VARCHAR(MAX);
DECLARE #LineageKey INT;
SET #json = (SELECT JsonString FROM Stage.Report);
SET #LineageKey = (SELECT LineageKey FROM Stage.Report);
INSERT INTO #TempData(Id, JsonData)
(SELECT [key], value FROM OPENJSON(#json, '$.rows'))
MERGE [dbo].[DestinationTable] TARGET
USING
(
SELECT
JSON_VALUE(JsonData, '$.data[0]') AS FiscalMonth,
JSON_VALUE(JsonData, '$.data[1]') AS FiscalYear,
JSON_VALUE(JsonData, '$.data[2]') AS Portfolio,
JSON_VALUE(JsonData, '$.data[3]') AS Rent
FROM #TempData
WHERE JSON_VALUE(JsonData, '$.data[0]') is not null
) AS SOURCE
...
etc., etc.
This works, but I want to know if there is a way to directly select the data values without the intermediate step of putting it into the temp table. The documentation and examples I've read seem to all require that the data have a name associated with it in order to access it. When I try and access the data directly at a position by index I just get Null.
I hope I understand your question correctly. If you know the columns names you need one OPENJSON() call with explicit schema, but if you want to read the JSON structure from $.columns, you need a dynamic statement.
JSON:
DECLARE #json nvarchar(max) = N'{
"report": "Property ETL",
"id": 2648,
"columns": [
{
"name": "Fiscal Month",
"dataType": "int"
},
{
"name": "Fiscal Year",
"dataType": "int"
},
{
"name": "Portfolio",
"dataType": "varchar(50)"
},
{
"name": "Rent",
"dataType": "int"
}
],
"rows": [
{
"rowName": "1",
"type": "Detail",
"data": [
11,
2019,
"West Group",
10
]
},
{
"rowName": "2",
"type": "Detail",
"data": [
11,
2019,
"East Group",
10
]
},
{
"rowName": "3",
"type": "Detail",
"data": [
11,
2019,
"East Group",
10
]
},
{
"rowName": "Totals: ",
"type": "Total",
"data": [
null,
null,
null,
30
]
}
]
}'
Statement for fixed structure:
SELECT *
FROM OPENJSON(#json, '$.rows') WITH (
[Fiscal Month] int '$.data[0]',
[Fiscal Year] int '$.data[1]',
[Portfolio] varchar(50) '$.data[2]',
[Rent] int '$.data[3]'
)
Dynamic statement:
DECLARE #stm nvarchar(max) = N''
SELECT #stm = CONCAT(
#stm,
N',',
QUOTENAME(j2.name),
N' ',
j2.dataType,
N' ''$.data[',
j1.[key],
N']'''
)
FROM OPENJSON(#json, '$.columns') j1
CROSS APPLY OPENJSON(j1.value) WITH (
name varchar(50) '$.name',
dataType varchar(50) '$.dataType'
) j2
SELECT #stm = CONCAT(
N'SELECT * FROM OPENJSON(#json, ''$.rows'') WITH (',
STUFF(#stm, 1, 1, N''),
N')'
)
PRINT #stm
EXEC sp_executesql #stm, N'#json nvarchar(max)', #json
Result:
--------------------------------------------
Fiscal Month Fiscal Year Portfolio Rent
--------------------------------------------
11 2019 West Group 10
11 2019 East Group 10
11 2019 East Group 10
30
Yes, it is possible without temporary table:
DECLARE #json NVARCHAR(MAX) =
N'
{
"report": "Property ETL",
"id": 2648,
"columns": [
{
"name": "Fiscal Month",
"dataType": "int"
},
{
"name": "Fiscal Year",
"dataType": "int"
},
{
"name": "Portfolio",
"dataType": "varchar(50)"
},
{
"name": "Rent",
"dataType": "int"
}
],
"rows": [
{
"rowName": "1",
"type": "Detail",
"data": [
11,
2019,
"West Group",
10
]
},
{
"rowName": "2",
"type": "Detail",
"data": [
11,
2019,
"East Group",
10
]
},
{
"rowName": "3",
"type": "Detail",
"data": [
11,
2019,
"East Group",
10
]
},
{
"rowName": "Totals: ",
"type": "Total",
"data": [
null,
null,
null,
30
]
}
]
}
}';
And query:
SELECT s.value,
rowName = JSON_VALUE(s.value, '$.rowName'),
[type] = JSON_VALUE(s.value, '$.type'),
s2.[key],
s2.value
FROM OPENJSON(JSON_QUERY(#json, '$.rows')) s
CROSS APPLY OPENJSON(JSON_QUERY(s.value, '$.data')) s2;
db<>fiddle demo
Or as a single row per detail:
SELECT s.value,
rowName = JSON_VALUE(s.value, '$.rowName'),
[type] = JSON_VALUE(s.value, '$.type'),
JSON_VALUE(s.value, '$.data[0]') AS FiscalMonth,
JSON_VALUE(s.value, '$.data[1]') AS FiscalYear,
JSON_VALUE(s.value, '$.data[2]') AS Portfolio,
JSON_VALUE(s.value, '$.data[3]') AS Rent
FROM OPENJSON(JSON_QUERY(#json, '$.rows')) s;
db<>fiddle demo 2

Multi Nested JSON in hive

I have the following input with nested JSON, I want to ingest this data into hive in multiple rows
"taxes": [{
"line_id": 1,
"commodity_code": "997159",
"fee": {
"amt": {
"curr_code": "USD",
"value": "71.4"
},
"type": "receiver"
},
"ship_addr": {
"admin_area_1": "MAHARASHTRA",
"country_code": "IN"
},
"total_tax": {
"curr_code": "USD",
"value": "12.8520000000"
},
"tax-details": [{
"exempt_option": false,
"auth_name": "India Maharashtra Central GST",
"doc_amt": {
"currency_code": "USD",
"value": "6.43"
},
"unrnd_doc_amt": {
"currency_code": "USD",
"value": "6.4260000000"
},
"rate": "0.09",
"rate_code": "SR",
"non_basis_doc_amt": "0.00",
"exempt_doc_amt": "0.00",
"jdx_memo": "INSS2: Tax payable in Seller location.",
"seller_reg_no": "27AAGCP4442G1ZF",
"admin_zone_level": "Country",
"auth_type": "CGST",
"erp_code": "MHCGST",
"inv_desc": "Standard Rate - CGST",
"basis_doc_amt": "71.40"
}, {
"exempt_option": false,
"auth_name": "India Maharashtra State GST",
"doc_amt": {
"currency_code": "USD",
"value": "6.43"
},
"unrnd_doc_amt": {
"currency_code": "USD",
"value": "6.4260000000"
},
"rate": "0.09",
"rate_code": "SR",
"non_basis_doc_amt": "0.00",
"exempt_doc_amt": "0.00",
"jdx_memo": "INSS2: Tax payable in Seller location.",
"seller_reg_no": "27AAGCP4442G1ZF",
"admin_zone_level": "Province",
"auth_type": "SGST",
"erp_code": "MHSGST",
"inv_desc": "Standard Rate - SGST",
"basis_doc_amt": "71.40"
}],
"transaction_type": "DS"
}]
I am using the following DDL
select
get_json_object(t.json,concat('$.taxes[',e.i,'].line_id')) as line_id
, get_json_object(t.json,concat('$.taxes[',e.i,'].commodity_code')) as commodity_code
, get_json_object(t.json,concat('$.taxes[',e.i,'].fee.amt.curr_code')) as curr_code
, get_json_object(t.json,concat('$.taxes[',e.i,'].fee.amt.value')) as value
, get_json_object(t.json,concat('$.taxes[',e.i,'].fee.type')) as type
, get_json_object(t.json,concat('$.taxes[',e.i,'].ship_addr.admin_area_1')) as admin_area
, get_json_object(t.json,concat('$.taxes[',e.i,'].ship_addr.country_code')) as country_code
, get_json_object(t.json,concat('$.taxes[',e.i,'].total_tax.curr_code')) as total_tax_curr_code
, get_json_object(t.json,concat('$.taxes[',e.i,'].total_tax.value')) as total_tax_value
get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].exempt_option')) as exempt_option
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].auth_name')) as auth_name
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].doc_amt.currency_code')) as doc_amt_currency_code
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].doc_amt.value')) as doc_amt_value
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].unrnd_doc_amt.currency_code')) as unrnd_doc_amt_currency_code
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].unrnd_doc_amt.value')) as unrnd_doc_amt_value
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].rate')) as rate
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].rate_code')) as rate_code
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].non_basis_doc_amt')) as non_basis_doc_amt
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].exempt_doc_amt')) as exempt_doc_amount
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].jdx_memo')) as jdx_memo
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].seller_reg_no')) as seller_reg_no
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].admin_zone_level')) as admin_zone_level
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].auth_type')) as auth_type
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].erp_code')) as erp_code
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].inv_desc')) as inv_desc
, get_json_object(t.json,concat('$.taxes.tax_details[',f.g,'].basis_doc_amt')) as basis_doc_amt
FROM json_2002 t
LATERAL VIEW POSEXPLODE (split(get_json_object(json,'$.taxes[*].line_id'),'","')) e as i,x
LATERAL VIEW POSEXPLODE (split(get_json_object(json,'$taxes.tax_details[*].exempt_option'),'","')) f as g,h
I was able to fix the issue by using regex_replace
LATERAL VIEW POSEXPLODE (split(regexp_replace(get_json_object(json,'$.taxes.tax_details[*].exempt_option'), '\[|\]', ''), ',' ) ) f as g,h
solved my problem

Get aggregate sum of json array in Postgres NOSQL json data

How to get aggregate SUM(amount) from "refunds" array in postgres json select
Following is my data schema and structure:
Table Name: transactions
Column name: data
{
"id": "tran_6ac25129951962e99f28fa488993",
"amount": 1200,
"origin_amount": 3900,
"status": "partial_refunded",
"description": "Subscription#sub_a67d59efb2bcbf73485a ",
"livemode": false,
"refunds": [
{
"id": "refund_ee4192ffb6d2caa490a1",
"amount": 1200,
"status": "refunded",
"created_at": 1426412340,
"updated_at": 1426412340,
},
{
"id": "refund_0e4a34e4ee7281d369df",
"amount": 1500,
"status": "refunded",
"created_at": 1426412353,
"updated_at": 1426412353,
}
]
}
Out put should be: 1200+1500 = 2700
Output
|---------
|total
|---------
|2700
Please provide global solution and not with static data
This should work on 9.3+
WITH x AS( SELECT
'{
"id": "tran_6ac25129951962e99f28fa488993",
"amount": 1200,
"origin_amount": 3900,
"status": "partial_refunded",
"description": "Subscription#sub_a67d59efb2bcbf73485a ",
"livemode": false,
"refunds": [
{
"id": "refund_ee4192ffb6d2caa490a1",
"amount": 1200,
"status": "refunded",
"created_at": 1426412340,
"updated_at": 1426412340
},
{
"id": "refund_0e4a34e4ee7281d369df",
"amount": 1500,
"status": "refunded",
"created_at": 1426412353,
"updated_at": 1426412353
}
]
}'::json as y),
refunds AS(
SELECT json_array_elements(y->'refunds') as j FROM x)
SELECT sum((j->>'amount')::int) FROM refunds;
WITH AllRefunds AS ( SELECT jsonb_array_elements(data->'refunds') AS refund FROM transactions)
SELECT SUM( CAST ( refund ->> 'amount' AS INTEGER )) FROM AllRefunds;
If you need to know how the query is built:
1.
WITH AllRefunds AS ( SELECT jsonb_array_elements(data->'refunds') FROM transactions)
SELECT * FROM AllRefunds;
This selects all elements as JSON objects (done via ->) from the array refunds that were found in transactions table and stores it in a new table AllRefunds. This new table only consists of one unnamed column.
2.
WITH AllRefunds AS ( SELECT jsonb_array_elements(data->'refunds') AS refund FROM transactions)
SELECT * FROM AllRefunds;
Here the added (second) AS renames the currently unnamed column inside AllRefunds to refund
3.
WITH AllRefunds AS ( SELECT jsonb_array_elements(data->'refunds') AS refund FROM transactions)
SELECT SUM( CAST ( refund ->> 'amount' AS INTEGER )) FROM AllRefunds;
Our array entries are JSON objects. So we return the field amount as a simple string with ->> that we then cast to Integers and SUM all entries up.