This question is regarding Stream Analytics. I want to export a blob into a SQL.I know the process, my question is with the query I have to use.
{"performanceCounter":[{"available_bytes":{"value":994164736.0},"categoryName":"Memory","instanceName":""}],"internal":{"data":{"id":"459bf840-d259-11e5-a640-1df0b6342362","documentVersion":"1.61"}},"context":{"device":{"type":"PC","network":"Ethernet","screenResolution":{},"locale":"en-US","id":"RD0003FF73B748","roleName":"Sdm.MyGovId.Static.Web","roleInstance":"Sdm.MyGovId.Static.Web_IN_1","oemName":"Microsoft Corporation","deviceName":"Virtual Machine","deviceModel":"Virtual Machine"},"application":{"version":"R2.0_20160205.5"},"location":{"continent":"North America","country":"United States","clientip":"104.41.209.0","province":"Washington","city":"Redmond"},"data":{"isSynthetic":false,"samplingRate":100.0,"eventTime":"2016-02-13T13:53:44.2667669Z"},"user":{"isAuthenticated":false,"anonAcquisitionDate":"0001-01-01T00:00:00Z","authAcquisitionDate":"0001-01-01T00:00:00Z","accountAcquisitionDate":"0001-01-01T00:00:00Z"},"operation":{},"cloud":{},"serverDevice":{},"custom":{"dimensions":[],"metrics":[]},"session":{}}}
{"performanceCounter":[{"percentage_processor_total":{"value":0.0123466420918703},"categoryName":"Processor","instanceName":"_Total"}],"internal":{"data":{"id":"459bf841-d259-11e5-a640-1df0b6342362","documentVersion":"1.61"}},"context":{"device":{"type":"PC","network":"Ethernet","screenResolution":{},"locale":"en-US","id":"RD0003FF73B748","roleName":"Sdm.MyGovId.Static.Web","roleInstance":"Sdm.MyGovId.Static.Web_IN_1","oemName":"Microsoft Corporation","deviceName":"Virtual Machine","deviceModel":"Virtual Machine"},"application":{"version":"R2.0_20160205.5"},"location":{"continent":"North America","country":"United States","clientip":"104.41.209.0","province":"Washington","city":"Redmond"},"data":{"isSynthetic":false,"samplingRate":100.0,"eventTime":"2016-02-13T13:53:44.2668221Z"},"user":{"isAuthenticated":false,"anonAcquisitionDate":"0001-01-01T00:00:00Z","authAcquisitionDate":"0001-01-01T00:00:00Z","accountAcquisitionDate":"0001-01-01T00:00:00Z"},"operation":{},"cloud":{},"serverDevice":{},"custom":{"dimensions":[],"metrics":[]},"session":{}}}
{"performanceCounter":[{"percentage_processor_time":{"value":0.0},"categoryName":"Process","instanceName":"w3wp"}],"internal":{"data":{"id":"459bf842-d259-11e5-a640-1df0b6342362","documentVersion":"1.61"}},"context":{"device":{"type":"PC","network":"Ethernet","screenResolution":{},"locale":"en-US","id":"RD0003FF73B748","roleName":"Sdm.MyGovId.Static.Web","roleInstance":"Sdm.MyGovId.Static.Web_IN_1","oemName":"Microsoft Corporation","deviceName":"Virtual Machine","deviceModel":"Virtual Machine"},"application":{"version":"R2.0_20160205.5"},"location":{"continent":"North America","country":"United States","clientip":"104.41.209.0","province":"Washington","city":"Redmond"},"data":{"isSynthetic":false,"samplingRate":100.0,"eventTime":"2016-02-13T13:53:44.2668342Z"},"user":{"isAuthenticated":false,"anonAcquisitionDate":"0001-01-01T00:00:00Z","authAcquisitionDate":"0001-01-01T00:00:00Z","accountAcquisitionDate":"0001-01-01T00:00:00Z"},"operation":{},"cloud":{},"serverDevice":{},"custom":{"dimensions":[],"metrics":[]},"session":{}}}
Well you can see 3 json objects which of them have different fields for the objects in the array performanceCounter. Basically the first object of every object. in the first is available_bytes, 2nd is percentage_processor_total, and 3rd is percentage_processor_time.
Because I'm exporting this to a sql table called performaceCounter, I should have a different column for every different object, so I would like to save this into an string and then I will parse it in my app.
As starting point I have this query that reads an input(the blob) and write into an output(SQL)
Select GetArrayElement(A.performanceCounter,0) as a
INTO
PerformanceCounterOutput
FROM PerformanceCounterInput A
This GetArrayElement takes the index 0 of the array in performanceCounter but then writes a different column for each different field that find in every object. So I should have all different counters and create a column for each one, but my idea is more like a column call performanceCounterData and save string like
'"available_bytes":"value":994164736.0},"categoryName":"Memory","instanceName":""'
or this
"{"percentage_processor_total":{"value":0.0123466420918703},"categoryName":"Processor","instanceName":"_Total"}"
or
"{"percentage_processor_time":"value":0.0},"categoryName":"Process","instanceName":"w3wp"}"
How can I cast an array like a String?
I tried CAST(GetArrayElement(A.performanceCounter,0) as nvarchar(max)) but I can't.
Please some good help will be rewarded
With the following solution I get 2 columns with the name of the property and another with the value of the property, that it was my initial purpose
With pc as
(
Select
GetArrayElement(A.[performanceCounter],0) as counter
,A.context.data.eventTime as eventTime
,A.context.location.clientip as clientIp
,A.context.location.continent as continent
,A.context.location.country as country
,A.context.location.province as province
,A.context.location.city as city
FROM PerformanceCounterInput A
)
select
props.propertyName,
props.propertyValue,
pc.counter.categoryName,
pc.counter.instanceName,
pc.eventTime,
pc.clientIp,
pc.continent,
pc.country,
pc.province,
pc.city
from pc
cross apply GetRecordProperties(pc.counter) as props
where props.propertyname<>'categoryname' and props.propertyname<>'instancename'
Anyway if somebody finds how to write an object in plain text in analytics, still rewarded and appreciated will be
You can do something like below, this gives the counters as (propertyName, propertyValue) pairs.
with T1 as
(
select
GetArrayElement(iotInput.performanceCounter, 0) Counter,
System.Timestamp [EventTime]
from
iotInput timestamp by context.data.eventTime
)
select
[EventTime],
Counter.categoryName,
Counter.available_bytes [Value]
from
T1
where
Counter.categoryName = 'Memory'
union all
select
[EventTime],
Counter.categoryName,
Counter.percentage_processor_time [Value]
from
T1
where
Counter.categoryName = 'Process'
Query that gives one column per counter type can also be done, you will have to either do a join or a group by with 'case' statements for every counter.
Related
I'm working on a MySQL Way of printing an "affiliate tree" and got the thing working with Common Table Expression. I'm using the following code right now:
WITH RECURSIVE recUsers AS
(
SELECT ID, username, sponsorID, 1 AS depth, username AS path
FROM users
WHERE id = 1
UNION ALL
SELECT c.ID, c.username, c.sponsorID, sc.depth + 1, CONCAT(sc.path, ' > ', c.username)
FROM recUsers AS sc
JOIN users AS c ON sc.ID = c.sponsorID
)
SELECT * FROM recUsers;
This selects the tree underneath the user with the id 1.
Now what I'd need to get is a way to pass that id as a parameter, so I don't need to define everything from the beginning every time I want to get the result.. So my idea is to put everything in a stored prodecure and pass the id in as a parameter.. However, so far I didn't get it working and always getting various errors that are very self speaking...
Basically what I've tried was
DELIMITER //
CREATE PROCEDURE getAffiliateTree(IN userid INT())
BEGIN
---my code here, the userid 1 replaced with userid
END//
DELIMITER;
However, this doesn't seem to work.. How can I get this done?
Two things I would suggest:
Use INT, not INT(). The optional length argument to integer types is deprecated in MySQL 8.0 (which I know you're using, because you're using CTE syntax). Even if you did use the length argument, using an empty argument is not legal syntax.
Make sure that the userid input parameter name is distinct from all of the columns in the tables you reference. That is, if the table has a column named userid (any capitalization), then change the name of your input parameter. Otherwise you may make ambiguous expressions like:
... WHERE userid = userid
Even though you intend one of these to be the column and the other to be the parameter, the SQL parser has no way of knowing that. It ends up treating both as the column name, so it's trivially true on all rows of the table.
Actually, a third thing I would suggest: when you ask questions, "it doesn't seem to work" isn't clear enough. Did it produce an error? If so, what was the full error message? Did it produce no error, but didn't give you the result you wanted? If so, show a mocked-up example of what you expected, and what the query produced that didn't match. It helps to be as clear as you can when you post questions, so readers don't have to guess what trouble you need help with.
While studying ways to optimize the size of tables with JSon content, I came across an unexpected situation: a column with a supposedly smaller JSON takes up more space in the database.
Two tables were created in the test, both with a bigint and an array JSon column. The difference between the tables ("current" and "proposal") is the smaller number of items that make up the array in the second table. Which, supposedly, decreased its size, right?
Not exactly. For some reason, the "proposal" table takes more space, even having less itens composing its JSON. To make sure I'm not doing nothing really stupid, follow some additional info and instructions used to read and create ddata.
SQL to read the above image data:
SELECT current.a as json_current,
pg_column_size(current.a) as column_size_current,
jsonb_array_length(current.a) as array_length_current,
proposal.a as json_proposal,
pg_column_size(proposal.a) as column_size_proposal,
jsonb_array_length(proposal.a) as array_length_proposal
FROM current
INNER join proposal on proposal.id = current.id
Both tables were created using:
CREATE TABLE table_name AS
SELECT source.id,
to_jsonb(
(SELECT array_to_json(array_agg(row_to_json(e)))
FROM (SELECT source.idprop1,
COALESCE(source.idprop2, 0),
COALESCE(source.idprop3, 0),
source.prop4,
source.prop5
FROM source
WHERE source.id_master = master_source.id_master
) e
)
) as a
FROM master_source
GROUP BY master_source.id
With the second table (proposal) having following additional clauses to its WHERE:
AND (source.prop2 is not null
OR source.prop3 is not null
OR (source.prop4 is not null and source.prop3 > 0)
OR (source.prop5 is not null and source.prop4 > 0)
Is there anything am I missing or expected from Postgres to achieve this result?
UPDATE: ABOUT TOAST
As expected, tables to store large text were created but, as we can see in the following image, JSON values are not large enought to be stored there.
I've created a MySQL sproc which returns 3 separate result sets. I'm implementing the npm mysql package downstream to exec the sproc and get a result structured in json with the 3 result sets. I need the ability to filter the json result sets that are returned based on some type of indicator in each result set. For example, if I wanted to get the result set from the json response which deals specifically with Suppliers then I could use some type of js filter similar to this:
var supplierResultSet = mySqlJsonResults.filter(x => x.ResultType === 'SupplierResults');
I think SQL Server provides the ability to include a hard-coded column value in a SQL result set like this:
select
'SupplierResults',
*
from
supplier
However, this approach appears to be invalid in MySQL b/c MySQL Workbench is telling me that the sproc syntax is invalid and won't let me save the changes. Do you know if something like what I'm trying to achieve is possible in MySQL and if not then can you recommend alternative approaches that would help me achieve my ultimate goal of including some type of fixed indicator in each result set to provide a handle for downstream filtering of the json response?
If I followed you correctly, you just need to prefix * with the table name or alias:
select 'SupplierResults' hardcoded, s.* from supplier s
As far as I know, this is the SQL Standard. select * is valid only when no other expression is added in the selec clause; SQL Server is lax about this, but most other databases follow the standard.
It is also a good idea to assign a name to the column that contains the hardcoded value (I named it hardcoded in the above query).
In MySQL you can simply put the * first:
SELECT *, 'SupplierResults'
FROM supplier
Demo on dbfiddle
To be more specific, in your case, in your query you would need to do this
select
'SupplierResults',
supplier.* -- <-- this
from
supplier
Try this
create table a (f1 int);
insert into a values (1);
select 'xxx', f1, a.* from a;
Basically, if there are other fields in select, prefix '*' with table name or alias
I have a table which records games of every day. The TEAMS table contains a column that contains json rows, each json string also contains another, something like below:
---------------------------------------------------------------------
| id | doc |
---------------------------------------------------------------------
| 1 | {'team1':{'num':3, 'players':{'bob', 'eli', 'jack'}, 'color':'red'},
'team2':{'num':3, 'players':{'a', 'eli', 'x'}, 'color':'blue'}}
This says that team1 and team2 had a game on a day.
Can I write a query which retrieves all of records that has 'eli' as player?
Thanks for your help
Caveat - I've only just begun working with JSON on Postgresql, so the following works, but might be sub-optimal...
Is that your actual JSON? Because Postgresql 9.3 coughed when I tried to import it; two problems, single quote and not double quotes, and your players were surrounded in curly braces and not square braces.
Anyway. If I got this right, I used this JSON:
create table json_table(data json);
insert into json_table(data)
values('{"team1":{"num":3, "players":["bob", "eli", "jack"], "color":"red"},
"team2":{"num":3, "players":["a", "eli", "x"], "color":"blue"}}')
The following query will work. Apparently, 9.4 has some additional functions that might make your life easier.
SELECT DISTINCT teamname FROM
(SELECT
json_data.key AS teamname,
json_array_elements(json_data.value->'players')::text as players
FROM
json_table,json_each(data) AS json_data)
a
WHERE players = '"eli"'
Stale question, sure, but it's got a much better answer nowadays.
There's a simple version, where you don't care about the top-level key, only the top-level value:
select json_path_query(
doc,
'$.* ? (#.players[*] == $player)',
'{"player": "eli"}'
) from TEAMS
If you want the top-level key, I don't have a good suggestion that doesn't involve conversion and/or expansion.
select json_path_query(
(team).value,
'$ ? (#.players[*] == $player)',
'{"player": "eli"}'
) from (
select json_each(doc) as team from TEAMS
) schedule
As this is an XY problem, most definitely, you might be interested in knowing whether eli's brother, "ERROR:ROOT", played that day too. You can always pass an array of values in for that third argument to json_path_query(), like so:
select json_path_query(
doc,
'$.* ? (#.players[*] == $player[*])',
'{"player": ["eli","ERROR:ROOT"]}'
) from TEAMS
Oh! That raises a really good possibility. Say you have a team, and you want to know what other days any of those players are up. You might do something like so:
select json_path_query(
doc,
'$.* ? (#.players[*] = $players[*])',
'{"num":3, "players": ["a", "eli", "x"], "color":"blue"'
) from TEAMS
If you want to know more, the PostgreSQL docs are expansive but fantastically detailed. There are tons of great examples in there, both at the page on JSON datatype and JSONpath, but also the page on JSON functions.
The JSON datatype: https://www.postgresql.org/docs/13/datatype-json.html
JSON functions: https://www.postgresql.org/docs/13/functions-json.html
Hiii,
I have a Database design as:
Table File (FileID, Name, Details)
Table Attributes (AttID, AttName, AttType)
Table AttValues (FileID, AttID, AttValue)
Till the runtime, it is not known how many Attributes are in a File and of what names.
And I want to display after insertion at Frontend in a manner like like:
FileID, FileName, Details, (Rows of Attribute Table as Column here).
i.e.,
Can anybody provide me a piece of code in Java or in MySQL to achieve this Pivoting Result.
Highly thanks full for your precious time.
Or is there any other better way to store data, So that I can get the desired result easily.
This requires two queries. First select the File:
SELECT * FROM File WHERE (...)
Then, fetch the Attributes:
SELECT *
FROM AttValues
JOIN Attributes ON (Attributes.AttId = AttValues.AttId)
WHERE FileId = $id
The latter query will provide you with one row per Attribute, which you can programmatically pivot for display on your frontend:
foreach(row in result) {
table.AddColumn(Header = row['AttName'], Value = row['AttValue']);
}
Adapt to your local programming environment as needed.
Of course, this only works for a single File or Files with the same attributes. If you want to display multiple files with different Attributes you can instead prefetch all AttNames:
SELECT Attributes.AttId, Attributes.AttName
FROM Attributes
JOIN AttValues ON (Attributes.AttId = AttValues.AttId)
WHERE FileId IN ( $list_of_ids )
Then load the values like this:
SELECT *
FROM AttValues
WHERE FileId IN ( $list_of_ids )
and use a local associative array to map from AttIds to column indexes.
As a final optimisation, you can combine the last two queries into an OUTER JOIN to avoid the third round trip. While this will probably increase the amount of data transferred, it also makes filling the table easier, if your class library supports named columns.
I answered a similar question recently: How to pivot a MySQL entity-attribute-value schema. The answer is MySQL-specific, but I guess that's OK as the question is tagged with mysql.