unpack all outer-most keys in json object as columns - json

I have a postgres DB called sales with a json-object, data containing around 100 outer-keys, lets name them k1,k2,k3..,k100.
I want to write a query
select * from sales some_function(data)
which simply returns something like
k1 | k2 | .. | k100
--------------------
"foo" | "bar" | .. | 2
"fizz"| "buzz"| .. | 10
ie. just unpacks the keys as columsn and their values as row.
Note, k1,k2..k100 is not their real name thus I can't do a
data->> key loop

That's not possible. One restriction of the SQL language is, that all columns (and their data types) must be known to the database when parsing the statement - so before it is actually run.
You will have to write each one separately:
select data ->> 'k1' as k1, data ->> 'k2' as k2, ...
from sales
One way to make this easier, is to generate a view dynamically by extracting all JSON keys from the column, then using dynamic SQL to create the view. You will however need to re-create that view each time the number of keys change.
Something along the lines (not tested!)
do
$$
declare
l_columns text;
l_sql text;
begin
select string_agg(distinct format('data ->> %L as %I', t.key, t.key), ', ')
into l_columns
from sales s
cross join jsonb_each(s.data) as t(key, value);
-- l_columns now contains something like:
-- data ->> 'k1' as k1, data ->> 'k2' as k2
-- now create a view from that
l_sql := 'create view sales_keys as select '||l_columns||' from sales';
execute l_sql;
end;
$$
;
You probably want to add e.g. the primary key column(s) to the view, so that you can match the JSON values back to the original row(s).

Related

JSON_VALUE for Nameless JSON payload

Just started playing with JSON_VALUE in SQL Server. I am able to pull values from name/value pairs of JSON but I happen to have an object that looks like this:
["first.last#domain.com"]
When I attempt what works for name/value pairs:
SELECT TOP 1
jsonemail,
JSON_VALUE(jsonemail, '$') as pleaseWorky
FROM MyTable
I get back the full input, not first.last#domain.com. Am I out of luck? I don't control the upstream source of the data. I think its a sting collection being converted into a json payload. If it was name: first.last#domain.com I would be able to get it with $.name.
Thanks in advance.
It is a JSON array. So you just need to specify its index, i.e 0.
Please try the following solution.
SQL
-- DDL and sample data population, start
DECLARE #tbl TABLE (ID INT IDENTITY PRIMARY KEY, jsonemail NVARCHAR(MAX));
INSERT INTO #tbl (jsonemail) VALUES
('["first.last#domain.com"]');
-- DDL and sample data population, end
SELECT ID
, jsonemail AS [Before]
, JSON_VALUE(jsonemail, '$[0]') as [After]
FROM #tbl;
Output
+----+---------------------------+-----------------------+
| ID | Before | After |
+----+---------------------------+-----------------------+
| 1 | ["first.last#domain.com"] | first.last#domain.com |
+----+---------------------------+-----------------------+
From the docs:
Array elements. For example, $.product[3]. Arrays are zero-based.
So you need JSON_VALUE(..., '$[0]') when the root is an array and you want the first value.
To break it out into rows, you would need OPENJSON:
SELECT TOP 1
jsonemail
,j.[value] as pleaseWorky
FROM MyTable
CROSS APPLY OPENJSON(jsonemail) j

How do I return a JSON updated document in Oracle?

From the docs I see an example:
SELECT json_mergepatch(po_document, '{"Special Instructions":null}'
RETURNING CLOB PRETTY)
FROM j_purchaseorder;
But When I try this code in SQL Developer I get a squiggly line under CLOB and an error when I run the query?
It works in Oracle 18c:
SELECT json_mergepatch(
po_document,
'{"Special Instructions":null}'
RETURNING CLOB PRETTY
) AS updated_po_document
FROM j_purchaseorder;
Which for the test data:
CREATE TABLE j_purchaseorder( po_document CLOB CHECK ( po_document IS JSON ) );
INSERT INTO j_purchaseorder ( po_document )
VALUES ( '{"existing":"value", "Special Instructions": 42}' );
Outputs:
| UPDATED_PO_DOCUMENT |
| :------------------------------- |
| {<br> "existing" : "value"<br>} |
Removing the Special Instructions attribute as per the documentation you linked to:
When merging object members that have the same field:
If the patch field value is null then the field is dropped from the source — it is not included in the result.
Otherwise, the field is kept in the result, but its value is the result of merging the source field value with the patch field value. That is, the merging operation in this case is recursive — it dives down into fields whose values are themselves objects.
db<>fiddle here

How to JSON extract from dynamic key value pair in MySQL?

I have the user table with user_id and user_details. it contains the JSON data in string format as shown below:
[{"name":"question-1","value":"sachin","label":"Enter your name?"},
{"name":"question-2","value":"abc#example.com","label":"Enter your email?"},
{"name":"question-3","value":"xyz","label":"Enter your city?"}]
I have tried with the json_extract but it return the result if json has data as shown below:
{"name":"question-1","value":"sachin","label":"Enter your name?"}
then it return the result as,
Name | Label
question-1 | Enter your name?
Expected Result :
I want to extract all name and label from json in sql query.
Example-1:
Consider that we have the following data in user_details column,
[{"name":"question-1","value":"sachin","label":"Enter your name?"},
{"name":"question-2","value":"abc#example.com","label":"Enter your email?"},
{"name":"question-3","value":"xyz","label":"Enter your city?"}]
then the sql query should return the result in following format ,
Name | Label
question-1 | Enter your name?
question-2 | Enter your email?
question-3 | Enter your city?
How to get this using JSON_EXTRACT in MySQL?
I assume that you are not using a table.
SET #data = '[{"name":"question-1","value":"sachin","label":"Enter your name?"},
{"name":"question-2","value":"abc#example.com","label":"Enter your email?"},
{"name":"question-3","value":"xyz","label":"Enter your city?"}]';
SELECT JSON_EXTRACT(#data,'$[*].name') AS "name", JSON_EXTRACT(#data,'$[*].label') AS "label";
it will return
name | label
["question-1", "question-2", "question-3"] | ["Enter your name?", "Enter your email?", "Enter your city?"]
SQL should be like below according to your table and column name:
SELECT JSON_EXTRACT(user_details,'$[*].name') AS "name", JSON_EXTRACT(user_details,'$[*].label') AS "label" FROM user;
you can match them by using some loops for arrays. I do not know if this is the best way but it satisfy my needs.
Another answer given by How to extract rows from a json array using the mysql udf json_extract 0.4.0? is to parse yourself the JSON with common_schema. Pretty tricky if you are not used to complex SQL.
You could create an own aggregated table as proposed in topic List all array elements of a MySQL JSON field if you know how many elements will be given by the field but I guess this is not your case.
However, it seems better, as mentioned in both answers, not to store such json lists in your SQL database. Maybe could you make a related table containing one line per each dictionary and then link it to your main table with a foreign key.
I was working in a report where there was a big json array list in one column. I modified the datamodel to store the relationship 1 to * instead of storing everything in one single column. For doing this process, I had to use a while in a stored procedure since I do not know the maximum size:
DROP PROCEDURE IF EXISTS `test`;
DELIMITER #
CREATE PROCEDURE `test`()
PROC_MAIN:BEGIN
DECLARE numNotes int;
DECLARE c int;
DECLARE pos varchar(10);
SET c = 0;
SET numNotes = (SELECT
ROUND (
(
LENGTH(debtor_master_notes)
- LENGTH( REPLACE ( debtor_master_notes, "Id", "") )
) / LENGTH("Id")
) AS countt FROM debtor_master
order by countt desc Limit 1);
DROP TEMPORARY TABLE IF EXISTS debtorTable;
CREATE TEMPORARY TABLE debtorTable(debtor_master_id int(11), json longtext, note int);
WHILE(c <numNotes) DO
SET pos = CONCAT('$[', c, ']');
INSERT INTO debtorTable(debtor_master_id, json, note)
SELECT debtor_master_id, JSON_EXTRACT(debtor_master_notes, pos), c+1
FROM debtor_master
WHERE debtor_master_notes IS NOT NULL AND debtor_master_notes like '%[%' AND JSON_EXTRACT(debtor_master_notes, pos) IS NOT NULL AND JSON_EXTRACT(debtor_master_notes, pos) IS NOT NULL;
SET c = c + 1;
END WHILE;
SELECT * FROM debtorTable;
END proc_main #
DELIMITER ;
You don't use JSON_EXTRACT(). You use JSON_TABLE():
mysql> create table mytable ( id serial primary key, data json);
Query OK, 0 rows affected (0.01 sec)
mysql> insert into mytable set data = '[{"name":"question-1","value":"sachin","label":"Enter your name?"},
'> {"name":"question-2","value":"abc#example.com","label":"Enter your email?"},
'> {"name":"question-3","value":"xyz","label":"Enter your city?"}]';
Query OK, 1 row affected (0.00 sec)
mysql> SELECT j.* FROM mytable,
JSON_TABLE(data, '$[*]' COLUMNS (
name VARCHAR(20) PATH '$.name',
label VARCHAR(50) PATH '$.label'
)) AS j;
+------------+-------------------+
| name | label |
+------------+-------------------+
| question-1 | Enter your name? |
| question-2 | Enter your email? |
| question-3 | Enter your city? |
+------------+-------------------+
JSON_TABLE() requires MySQL 8.0.4 or later. If you aren't running at least that version, you will have to upgrade.
Honestly, if you need to access the individual fields, it's less work to store your data in normal columns, and avoid using JSON.

Iterate through a table and replace values in another table using a stored procedure or function

Hopelessly stuck at the following and up until now none of my programming speed dial buddies has been able to help out (most of them not MySQL experts):
I have different tables where the column names and datatypes are auto generated from the 'import table data wizard' using a CSV file, and the table does not contain an AUTO INCREMENT column (yet). This particular table consists of approx: 30.000 rows It starts at row=id(1) from a table that looks like this:
I am trying to correct values in one column that are comma delimited using one 'corrections' table. And to do this I am writing a stored procedure containing a WHILE loop to interate through the corrections table row for row, and check wheter or not an Alias is found in the table that was imported.
| id | material | alias01 | alias02 | alias03 | *up to 12
1 Katoen Cotton Supima Pima
2 Polyester Polyster
3 Lyocell Lycocell Lyocel
4 Linnen Linen
5 Viscose Visose Viskose Viscoe Voscose
6 Scheerwol
7 Polyamide
8 Nylon
9 Leer Leder Lamsleder Varkensleder
10 Polyurethaan Polyurethan PU Polyuretaan
For testing purposes to test any kind of results i am only using alias01 for now ( it needs to check alias01, then 02 etc... but i'll try to solve that at a later time).
It needs to compare `Length' ( alias_string_length = found_string_length) to make sure that a string that consist of 'wo' is not found in 'wool' or 'wol'.
The values from the column that need corrections look like this (the comma's dont need to be there it's just what i was given to work with):
| material |
,Katoen,Elastaan,Voering,Acetaat,Polyester
,Nylon,Polyester,Elastaan
,Katoen
,Leder,in,Leder,Loopzool,Leder
,Polyester
,Polyester,Elastaan,Voering,Polyester
Update
Thanks to Drew's tip i changed the procedure. I added a tmp table that holds materials AND a unique id for each row, and iterate through each one with the alias01. It takes around 11 seconds to do 9000 rows but 0 row(s) affected,. Any tips on increasing speed are most welcome, but insight in what might be the issue would help alot more.
CREATE DEFINER=`root`#`localhost` PROCEDURE `replace_materials`()
BEGIN
set #rownumber = 1;
set #totalrows = 28;
set #um ='';
set #cm ='';
set #corrected ='';
set #correctme ='';
TRUNCATE TABLE tmp;
INSERT INTO tmp (material) SELECT material FROM vantilburgonline.productinfo;
WHILE (#rownumber < #totalrows) DO
SET #um = (SELECT alias01 FROM vantilburgonline.materials WHERE id=#rownumber);
-- gives 'um' value from column alias01, from table materials, row(X)
SET #cm = (SELECT material FROM vantilburgonline.materials WHERE id=#rownumber);
-- gives 'cm' value from column material, from table materials, row(X)
set #tmprow = 1;
set #totaltmprow =9000;
WHILE (#tmprow < #totaltmprow) DO
SET #correctme = (SELECT material FROM vantilburgonline.tmp WHERE id = #tmprow);
-- gives the value from column material from table tmp to correctme(X).
SET #correctme = REPLACE(#correctme,#um,#cm);
-- should run through column material from table productinfo and replace 'alias01' with correct 'material'.
SET #tmprow = #tmprow +1;
END WHILE;
SET #rownumber = #rownumber +1;
END WHILE;
END
though i'm certain alias01 contains strings it should've found in the materials. Also Workbench was using 9GB at this point and i was only able to counter that by restarting..
I would recommend an alteration from your materials table which is unwieldy with multiple columns (alias01 .. alias12). A transition to a normalized, extensible system. It would have a materials table and a materials_alias table. As it sits alongside your current table that you created, I named them with a 2.
Schema
drop table if exists materials2;
create table materials2
( material varchar(100) primary key, -- let's go with a natural key
active bool not null -- turn it LIVE and ON for string replacement of alias back to material name
-- so active is TRUE for ones to do replacement, or FALSE for skip
-- facilitates your testing of your synonyms, translations, slangs, etc
)engine=INNODB;
insert materials2 (material,active) values
('KARTON',true),
('Polyester',false),
('Lyocell',false),
('Linnen',true),
('Viscose',true),
('Scheerwol',false),
('Nylon',false),
('Leer',true),
('Polyurethaan',true),
('Polyacryl',true),
('Acryl',false),
('Modal',true),
('Acetaat',true),
('Papier',false),
('Wol',true),
('Zijde',true),
('Temcal',false),
('Polyamide',true),
('Wol-Merino',true),
('Elastan',true),
('Elastomultiester',true);
-- 21 rows
-- a few rows were skipped. The intent of them read as gibberish to me. Please review.
-- we need to restructure the materials2_alias table (after the first attempt)
-- 1. it might need special handling when `alias` is a legitimate substring of `material` (those 2 columns)
-- 2. it needs a unique composite index
drop table if exists materials2_alias;
create table materials2_alias
( id int auto_increment primary key,
material varchar(100) not null,
alias varchar(100) not null,
ais bool not null, -- Alias is Substring (alias is a legitimate substring of material, like Wo and Wol, respectively)
unique key(material,alias), -- Composite Index, do not allow dupe combos (only 1 row per combo)
foreign key `m2alias_m2` (material) references materials2(material)
)engine=INNODB;
insert materials2_alias (material,alias,ais) values
('KARTON','Cotton',false),('KARTON','Katoen',false),('KARTON','Pima',false),
('Polyester','Polyster',false),
('Lyocell','Lycocell',false),('Lyocell','Lyocel',false),
('Linnen','Linen',false),
('Viscose','Visose',false),('Viscose','Viskose',false),('Viscose','Viscoe',false),('Viscose','Voscose',false),
('Leer','Leder',false),('Leer','Lamsleder',false),('Leer','Varkensleder',false),('Leer','Schapenleder',false),('Leer','Geitenleder',false),
('Polyurethaan','Polyurethan',false),('Polyurethaan','PU',false),('Polyurethaan','Polyuretaan',false),('Polyurethaan','Polyurathane',false),('Polyurethaan','Polyurtaan',false),('Polyurethaan','Polyueretaan',false),
('Polyacryl','Polyacrylic',false),
('Acetaat','Leder',false),('Acetaat','Lamsleder',false),
('Wol','Schuurwol',false),('Wol','Wool',false),('Wol','WO',false),('Wol','Scheerwol',false),
('Zijde','Silk',false),('Zijde','Sede',false),
('Polyamide','Polyamie',false),('Polyamide','Polyamid',false),('Polyamide','Poliamide',false),
('Wol-Merino','Merino',false),
('Elastan','Elastaan',false),('Elastan','Spandex',false),('Elastan','Elataan',false),('Elastan','Elastane',false),
('Elastomultiester','elastomutltiester',false),('Elastomultiester','Elasomultiester',false);
-- this cleans up the above, where false should have been true
update materials2_alias
set ais=true
where instr(material,alias)>0;
-- 4 rows
There are several alter table statements and other things. I will try to document them or link to them. I am merely trying to capture something to share considering it is several hundred lines of code from you. But mine comes down to a simple chunk of code you would put in a loop.
The Update put in a loop:
UPDATE productinfo pi
join materials2_alias ma
on instr( pi.material, concat(',',ma.alias,',') )>0
join materials2 m
on m.material=ma.material and m.active=true
set pi.material=replace(lower(pi.material),lower(ma.alias),lower(ma.material)),
pi.touchCount=pi.touchCount+1;
A few notes on the update:
-- Note, pi.material starts and ends with a comma.
-- I forced that during the ETL. But `ma.alias` does not contain commas.
-- So add the commas with a concat() within the "Update with a Join" pattern shown
--
-- Note that the commas solved the problem with the Wol - Wo
Well, the following 4 in particular.
select * from materials2_alias
where ais=true
order by material,alias;
+----+------------+----------+-----+
| id | material | alias | ais |
+----+------------+----------+-----+
| 6 | Lyocell | Lyocel | 1 |
| 33 | Polyamide | Polyamid | 1 |
| 28 | Wol | WO | 1 |
| 35 | Wol-Merino | Merino | 1 |
+----+------------+----------+-----+
-- instr() is not case sensitive except for binary strings
-- REPLACE(str,from_str,to_str); -- case sensitive
-- http://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_replace
--
-- so the update uses lower() or this won't work due to replace() case sensitivity
--
Stored Procedure:
DROP PROCEDURE if exists touchCounts;
DELIMITER $$
CREATE PROCEDURE touchCounts()
BEGIN
select touchCount,count(*) as rowCount
from productinfo
group by touchCount
order by touchCount;
END $$
DELIMITER ;
When that stored procedure returns the same count of rows on a successive call (the next call), you are done modifying the material column via the update.
That stored procedure could naturally return an out parameter for the rowcount. But it is late and time to sleep.
For your last data set from your side, the update statement would need to be called 4 times. That is like 13 seconds on my mediocre laptop. The idea is naturally flexible, for hundreds of aliases per material if you want.
I parked it up on github as it is too much otherwise.

Is it possible to transform a settings table with unique names into a json hash object in postgresql?

I have a settings table with two columns - name and value. Names are unique. I can easily read it into memory and then create a dictionary using the entry names as the keys.
I was wondering whether this can be done entirely from the SQL using some postgresql functions and applying the row_to_json function at the end.
I have version 9.2
Is it possible? It should be.
I think what you'd have to do is create a function for pulling a record in (as an argument) and transforming it to a record of arbitrary type and turning that into JSON.
This was done on 9.1 with the json extension.
create or replace function to_json(test) returns json language plpgsql
as $$
declare t_row record;
retval json;
begin
EXECUTE $E$ SELECT $1 AS $E$ || quote_ident($1.name) INTO t_row
USING $1.value;
RETURN row_to_json(t_row);
end;
$$;
Then I can:
select * from test;
name | value
-------+--------
test1 | foo
test2 | foobar
(2 rows)
SELECT to_json(test) from test;
to_json
--------------------
{"test1":"foo"}
{"test2":"foobar"}
Now if you want to merge these all into one object you have a little more work to do but it could be done using the same basic tools.
This should work in postgres-9.3. (untested, since I don't have 9.3 available here yet)
DROP SCHEMA tmp CASCADE;
CREATE SCHEMA tmp ;
SET search_path=tmp;
CREATE table pipo (name varchar NOT NULL PRIMARY KEY
, value varchar);
INSERT INTO pipo (name, value ) VALUES
('memory' , '10Mb'), ('disk' , '1Gb'), ('video' , '10Mpix/sec'), ('sound' , '100dB');
SELECT row_to_json( ROW(p.name,p.value) )
FROM pipo p ;