Postgres 10, select from json object containing json array of objects - json

In the database I have text field contains json with the structure:
"Limits":{
"fields":[
{
"key":"DAILY_LIMIT",
"value":"1559",
"lastModified":1543857829148,
},
{
"key":"MONTHLY_LIMIT",
"value":"25590",
"lastModified":1543857829148,
}
]
}
I need to check if daily_limit exists. It's easy to do with LIKE %DAILY_LIMIT% but performance is not so good and also I won't have access to value (right now I don't need it but maybe in future, it'll be needed). There is an option to check if this key exists without killing the db? I tried with 'Limits'->'fields'-> but I don't know what should be next... And it must be done by query, I cant pass object to backend and then check it

demo: db<>fiddle
If you want to do it the JSON way this could be a solution:
WITH data AS (
SELECT 'somedata' as somedata, '{"Limits":{"fields":[{"key":"DAILY_LIMITS","value":"1559","lastModified":1543857829148},{"key":"MONTHLY_LIMIT","value":"25590","lastModified":1543857829148}]}}'::jsonb as data
)
SELECT
d.*
FROM data d, jsonb_array_elements(data -> 'Limits' -> 'fields')
WHERE value ->> 'key' = 'DAILY_LIMITS'
jsonb_array_elements expands the array into one row each element. In the next step you are able to check the key's value.
But the demo shows, that a simple LIKE would be much faster as #404 mentioned correctly (have a look at the costs of both examples.)

Related

How can you write a query that checks the value of a JSON column?

Suppose I have a table with three columns: id, name, state. The state column is of type jsonb and the structure would always have (as a minimum) a key called active e.g.
{"name": "some_name", active: true, ...}
{"name": "one_two", active: false, ...}
I've got everything to work with serializing and deserializing by writing my own #WritingConverter and #ReadingConverter (to a PGobject) and so I could write a query which then at the Java level performs the filter so only those which are 'active' remain.
But how can I, in Spring Data JDBC, write a query that would allow me to interrogate the json column at the level of the DB?
You'd use use the #Query annotation and whatever facilities your database offers to query json.
Just as an example this tutorial for working with JSON in Postgres suggests that something like should work.
#Query("select * from mytable where state ->> 'active' = :active ")
List<MyTable> findAllActive(String active)
You probably can convert state ->> 'active' to something suitable so you can use a boolean as the method argument, but I'd start with something simple.

Selecting/Querying a JSON for a variable Object in SQL

I have a JSON in a SQL database that formerly looked like that:
{ "result": {
"topResult": {
"score": 1}
}
}
For a bigger query, I want to select the score in this object structure via SQL, which I did with this operator (or JSON_extract):
object->>'$.result.topResult.score' AS 'Score'
The problem that I now have is that the name "topResult" is variable/changing, so this query doesn't work anymore (or at least only case specific)
How can I make my Select statement a bit more generic to still output me the score regardless of the object before?
My ideas:
I can derive the name of the object from somewhere else in the JSON
--> could I reuse it as an alias, a variable or a concatenation to fill in the variable?
Is there a possibility to "skip" or index a JSON object?
Can ignore the path before and just look for a specific object like "score"?
Thanks for your hints!
SELECT *
FROM test
CROSS JOIN JSON_TABLE(test.val,
'$.result.*' COLUMNS (score INT PATH '$.score')) jsontable;
https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=46fbf88c6b4ce01b3cb5e0040c86451b

How to get a specific value from json with similar tags with T-SQL

I have a nvarchar(max) column with a json blob structured like this:
[
{"QuestionName":"Question1", "AnswerValue":"123456"},
{"QuestionName":"Question2", "AnswerValue":"987654"},
{"QuestionName":"Question3", "AnswerValue":"951753"}
]
I would like to do this in a set based operation as I am pulling back over 50k records. I would like to get the Answer Value where QuestionName = Question2.
In the case above I'd like to return 987654. This value may not be in the exact same place of the array every time.
I am very green when working with JSon, let alone using it in SQL Server. I believe what I want is something like
JSON_VALUE(MyJSonColumn, '???')
but I'm not sure how to actually get what I need.
You could use OPENJSON:
SELECT a.questionName, a.answerValue
FROM t
CROSS APPLY OPENJSON(col)
WITH(QuestionName VARCHAR(100) N'$."QuestionName"', AnswerValue INT N'$."AnswerValue"') AS a
WHERE questionName = 'Question2';
db<>fiddle demo

SQL Querying inside XML column

Am trying to Query inside an SQL table which has XML Column .
Table name: 'Purchase'
Column name: 'XML_COL'
Please find below xml data for column name 'XML_COL' under purchase table:
<ns1:Request xmlns:ns1="http://www.sample.com/hic/event/request"
xmlns:ns2="http://www.sample.com/hic/eventpayload/request">
<ns1:createeventRequest>
<ns1:eventPayLoad>
<ns2:eventPayLoad>
<Id>123456</Id>
</ns2:eventPayLoad>
</ns1:eventPayLoad>
</ns1:createeventRequest>
</ns1:Request>
I have written below query :
`select * from purchase,
XMLTABLE ('$d/Request/createeventRequest/eventPayLoad/eventPayLoad' PASSING XML_COL as "d"
COLUMNS
Id varchar(20) PATH 'Id') as a where(a.Id like '1234%');`
But this is returning me an empty column with no data.
But my requirement is it should fetch all the data for this particular Id.
Please help if any one faced this kind of issue.
Do we need to include namespaces as well while querying?? or am I missing any thing?
I think the expression PATH 'Id' is bit to simple...
I'm not familiar with MySQL's abilities to query XML... The Path Id would try to find an element "Id" from the current node (which is the root node in the first action). But there is no "Id"... You must either specify the full path, starting with a single / to start at the root node, or let the engine try a deep search, starting with two //
These paths should work:
SELECT ExtractValue(
'<ns1:Request xmlns:ns1="http://www.sample.com/hic/event/request" xmlns:ns2="http://www.sample.com/hic/eventpayload/request">
<ns1:createeventRequest>
<ns1:eventPayLoad>
<ns2:eventPayLoad>
<Id>123456</Id>
</ns2:eventPayLoad>
</ns1:eventPayLoad>
</ns1:createeventRequest>
</ns1:Request>',
'/ns1:Request[1]/ns1:createeventRequest[1]/ns1:eventPayLoad[1]/ns2:eventPayLoad[1]/Id[1]' ) AS result;
If there is only one element with a value (in your case "Id") you might use the simple deep search like this:
SELECT ExtractValue(
'<ns1:Request xmlns:ns1="http://www.sample.com/hic/event/request" xmlns:ns2="http://www.sample.com/hic/eventpayload/request">
<ns1:createeventRequest>
<ns1:eventPayLoad>
<ns2:eventPayLoad>
<Id>123456</Id>
</ns2:eventPayLoad>
</ns1:eventPayLoad>
</ns1:createeventRequest>
</ns1:Request>',
'//Id[1]' ) AS result;
But - in general - it is good advise to be as specific as possible...
Just cracked the query...When name spaces are being used in an XML, instead of the entire path, I found it's better to use '/*//' which traverses through the required element tag through XML.
Final Query:
select * from purchase,
XMLTABLE('$d' PASSING XML_COL as "d"
COLUMNS
Id varchar(20) PATH '/*//Id') as a where(a.Id like '1234%') with ur
Using 'with ur' helps to read the data that has not been committed in the database.
Please post comments if it is helpful.

Parse an object as string in output in Stream Azure Analytics

This question is regarding Stream Analytics. I want to export a blob into a SQL.I know the process, my question is with the query I have to use.
{"performanceCounter":[{"available_bytes":{"value":994164736.0},"categoryName":"Memory","instanceName":""}],"internal":{"data":{"id":"459bf840-d259-11e5-a640-1df0b6342362","documentVersion":"1.61"}},"context":{"device":{"type":"PC","network":"Ethernet","screenResolution":{},"locale":"en-US","id":"RD0003FF73B748","roleName":"Sdm.MyGovId.Static.Web","roleInstance":"Sdm.MyGovId.Static.Web_IN_1","oemName":"Microsoft Corporation","deviceName":"Virtual Machine","deviceModel":"Virtual Machine"},"application":{"version":"R2.0_20160205.5"},"location":{"continent":"North America","country":"United States","clientip":"104.41.209.0","province":"Washington","city":"Redmond"},"data":{"isSynthetic":false,"samplingRate":100.0,"eventTime":"2016-02-13T13:53:44.2667669Z"},"user":{"isAuthenticated":false,"anonAcquisitionDate":"0001-01-01T00:00:00Z","authAcquisitionDate":"0001-01-01T00:00:00Z","accountAcquisitionDate":"0001-01-01T00:00:00Z"},"operation":{},"cloud":{},"serverDevice":{},"custom":{"dimensions":[],"metrics":[]},"session":{}}}
{"performanceCounter":[{"percentage_processor_total":{"value":0.0123466420918703},"categoryName":"Processor","instanceName":"_Total"}],"internal":{"data":{"id":"459bf841-d259-11e5-a640-1df0b6342362","documentVersion":"1.61"}},"context":{"device":{"type":"PC","network":"Ethernet","screenResolution":{},"locale":"en-US","id":"RD0003FF73B748","roleName":"Sdm.MyGovId.Static.Web","roleInstance":"Sdm.MyGovId.Static.Web_IN_1","oemName":"Microsoft Corporation","deviceName":"Virtual Machine","deviceModel":"Virtual Machine"},"application":{"version":"R2.0_20160205.5"},"location":{"continent":"North America","country":"United States","clientip":"104.41.209.0","province":"Washington","city":"Redmond"},"data":{"isSynthetic":false,"samplingRate":100.0,"eventTime":"2016-02-13T13:53:44.2668221Z"},"user":{"isAuthenticated":false,"anonAcquisitionDate":"0001-01-01T00:00:00Z","authAcquisitionDate":"0001-01-01T00:00:00Z","accountAcquisitionDate":"0001-01-01T00:00:00Z"},"operation":{},"cloud":{},"serverDevice":{},"custom":{"dimensions":[],"metrics":[]},"session":{}}}
{"performanceCounter":[{"percentage_processor_time":{"value":0.0},"categoryName":"Process","instanceName":"w3wp"}],"internal":{"data":{"id":"459bf842-d259-11e5-a640-1df0b6342362","documentVersion":"1.61"}},"context":{"device":{"type":"PC","network":"Ethernet","screenResolution":{},"locale":"en-US","id":"RD0003FF73B748","roleName":"Sdm.MyGovId.Static.Web","roleInstance":"Sdm.MyGovId.Static.Web_IN_1","oemName":"Microsoft Corporation","deviceName":"Virtual Machine","deviceModel":"Virtual Machine"},"application":{"version":"R2.0_20160205.5"},"location":{"continent":"North America","country":"United States","clientip":"104.41.209.0","province":"Washington","city":"Redmond"},"data":{"isSynthetic":false,"samplingRate":100.0,"eventTime":"2016-02-13T13:53:44.2668342Z"},"user":{"isAuthenticated":false,"anonAcquisitionDate":"0001-01-01T00:00:00Z","authAcquisitionDate":"0001-01-01T00:00:00Z","accountAcquisitionDate":"0001-01-01T00:00:00Z"},"operation":{},"cloud":{},"serverDevice":{},"custom":{"dimensions":[],"metrics":[]},"session":{}}}
Well you can see 3 json objects which of them have different fields for the objects in the array performanceCounter. Basically the first object of every object. in the first is available_bytes, 2nd is percentage_processor_total, and 3rd is percentage_processor_time.
Because I'm exporting this to a sql table called performaceCounter, I should have a different column for every different object, so I would like to save this into an string and then I will parse it in my app.
As starting point I have this query that reads an input(the blob) and write into an output(SQL)
Select GetArrayElement(A.performanceCounter,0) as a
INTO
PerformanceCounterOutput
FROM PerformanceCounterInput A
This GetArrayElement takes the index 0 of the array in performanceCounter but then writes a different column for each different field that find in every object. So I should have all different counters and create a column for each one, but my idea is more like a column call performanceCounterData and save string like
'"available_bytes":"value":994164736.0},"categoryName":"Memory","instanceName":""'
or this
"{"percentage_processor_total":{"value":0.0123466420918703},"categoryName":"Processor","instanceName":"_Total"}"
or
"{"percentage_processor_time":"value":0.0},"categoryName":"Process","instanceName":"w3wp"}"
How can I cast an array like a String?
I tried CAST(GetArrayElement(A.performanceCounter,0) as nvarchar(max)) but I can't.
Please some good help will be rewarded
With the following solution I get 2 columns with the name of the property and another with the value of the property, that it was my initial purpose
With pc as
(
Select
GetArrayElement(A.[performanceCounter],0) as counter
,A.context.data.eventTime as eventTime
,A.context.location.clientip as clientIp
,A.context.location.continent as continent
,A.context.location.country as country
,A.context.location.province as province
,A.context.location.city as city
FROM PerformanceCounterInput A
)
select
props.propertyName,
props.propertyValue,
pc.counter.categoryName,
pc.counter.instanceName,
pc.eventTime,
pc.clientIp,
pc.continent,
pc.country,
pc.province,
pc.city
from pc
cross apply GetRecordProperties(pc.counter) as props
where props.propertyname<>'categoryname' and props.propertyname<>'instancename'
Anyway if somebody finds how to write an object in plain text in analytics, still rewarded and appreciated will be
You can do something like below, this gives the counters as (propertyName, propertyValue) pairs.
with T1 as
(
select
GetArrayElement(iotInput.performanceCounter, 0) Counter,
System.Timestamp [EventTime]
from
iotInput timestamp by context.data.eventTime
)
select
[EventTime],
Counter.categoryName,
Counter.available_bytes [Value]
from
T1
where
Counter.categoryName = 'Memory'
union all
select
[EventTime],
Counter.categoryName,
Counter.percentage_processor_time [Value]
from
T1
where
Counter.categoryName = 'Process'
Query that gives one column per counter type can also be done, you will have to either do a join or a group by with 'case' statements for every counter.