Substring search on JSON data (ERROR: cannot extract element from a scalar) - json

I've been looking at this tutorial and this example trying to find a performant way to query by substring with JSON (<9.5)/JSONB (*9.5) data.
For example, I have this data:
CREATE TABLE public.foo
(
row_id SERIAL PRIMARY KEY,
data json
);
INSERT INTO foo VALUES (1,
'{ "name": "Book the First", "author": "Bob", "text_entry": "White Cow Walked Over the Moon" } ');
INSERT INTO foo VALUES (2,
'{ "name": "Book the Second", "author": "Charles", "text_entry": "Humptey Dumptey sat on the Moon" } ');
INSERT INTO foo VALUES (3,
'{ "name": "Book the Third", "author": "Jim", "text_entry": "Red Fox jumped over Brown Dog" } ');
I'm looking for a way to search ONLY "text_entry" and return any case that has the sub-string "the Moon" (in this case, it would be id = 1 & 2). Expected Return:
text_entry
"White Cow Walked Over the Moon" ## has the substring "the Moon"
"Humptey Dumptey sat on the Moon" ## has the substring "the Moon"
So far my query looks like this:
SELECT data->'text_entry'->'%the Moon%' AS query FROM foo;
ERROR: cannot extract element from a scalar
********** Error **********
Is there any elegant way to query substrings in JSON/B?

I'm looking for a way to search ONLY "text_entry" and return any case that has the sub-string "the Moon"
SELECT data->>'text_entry' as query
FROM foo
WHERE data->>'text_entry' LIKE '%the Moon%';
->
query
---------------------------------
White Cow Walked Over the Moon
Humptey Dumptey sat on the Moon
(2 rows)

Related

is there a way to insert json multiple times into the same record?

Let's say I have inserted a record like this:
id | u_id | the_data
-------+------+--------------------------------------------------
1 | 2863 |[{name: a, body: lorem}]
using this command:
CREATE TABLE users (
id SERIAL PRIMARY KEY,
u_id INT,
the_data JSON
);
INSERT INTO users (u_id, the_data) VALUES (2863, '[{"name": "a", "body": "lorem"}]');
But now, I want to insert some more data into the same record without losing the old array of json. How to do this type of insertion?
id | u_id | the_data
-------+------+------------------------------------------------------------------------------
1 | 2863 |[ {name: a, body: lorem}, {name: b, body: ipsum} ]
Please note: Below command creates a new record which I don't want.
INSERT INTO users (u_id, the_data)
VALUES (2863, '[{"name": "b", "body": "ipsum"}]');
Not looking for solutions like below since they all insert at the same time:
INSERT INTO users (u_id, the_data)
VALUES (2863, '[{"name": "a", "body": "lorem"}, {"name": "b", "body": "ipsum"}]');
INSERT INTO users (u_id, the_data)
VALUES (2863, '[{"name": "a", "body": "lorem"}]'), (2863, '[{"name": "b", "body": "ipsum"}]');
As the top level JSON object is an array, you can use the standard concatenation operator || to append an element to the array:
update users
set the_data = the_data || '{"name": "b", "body": "ipsum"}'
where u_id = 2863;
You should change your column definition to jsonb as that offers a lot more possibilities for querying or changing the value. Otherwise you will be forced to cast the column to jsonb every time you want to do something more interesting with it.
If you can't or don't want to change the data type you need to cast it:
set the_data = the_data::jsonb || '....'
You can create list of object and parse it to loop :
For Example:
var data = {
Id : Id
Name : Name
}
Json Request :
Data: data
Well, that's not a simple json object. You're trying to add a object to an array of values that is saved as json field.
So it's not about keeping the old array, but rather keeping the objects that were already present on the array saved in the json field and adding the new one.
I tried this on Postgres 12, it works, basically as someone else said, you will need to cast the jsonb type if you've json and use pipes operator to concatenate the new value.
UPDATE users
SET the_data = the_data::jsonb || '{"name": "b", "body": "ipsum"}'
WHERE id = 1;
Taken from here:
https://stackoverflow.com/a/69630521/9231145

Split JSON into columns in a dynamic way in Big Query

I have the following JSON:
{
"rewards": {
"reward_1": {
"type": "type 1",
"amount": "amount 1"
},
"reward_2": {
"type": "type 2",
"amount": "amount 2"
},
"reward_3": {
"type": "type 3",
"amount": "amount 3"
},
"reward_4": {
"type": "type 4",
"amount": "amount 4"
}
}
}
This JSON is dynamic and I don't necessarily know how many rewards it will get, here it's 4 but it can be 2 or 8 etc.
I want to write a query in Big Query that will parse those values dynamically without knowing how many of them exist, and then split them into column, like this:
Thank you!
Hope these are helpful.
since a JSON data is dynamic, first step is to find a max reward sequence. (I've used a regular expression and max_reward UDF.)
and then, extract each reward from a json rewards field in an iterative way.
lastly, make the result to be a wide form using PIVOT query.
If you want a more generic solution, you need to use BigQuery dynamic SQL to generate PIVOT columns. I've hard-coded them in the query.
('reward_1', 'reward_2', 'reward_3', 'reward_4')
query:
CREATE TEMP TABLE sample AS
SELECT 1 AS id, '{"rewards": { "reward_1": { ... ' AS json -- put your json here
UNION ALL
SELECT 2 AS id, '{"rewards": { "reward_1": { ... ' AS json -- put your another json here
;
CREATE TEMP FUNCTION extract_reward(json STRING, seq INT64)
RETURNS STRUCT<type STRING, amount STRING>
LANGUAGE js AS """
return JSON.parse(json)['reward_' + seq];
""";
CREATE TEMP FUNCTION max_reward(arr ARRAY<STRING>) AS ((
SELECT MAX(CAST(v AS INT64)) FROM UNNEST(arr) v
));
SELECT * FROM (
SELECT id,
'reward_' || seq AS reward,
extract_reward(FORMAT('%t', JSON_QUERY(json, '$.rewards')), seq) AS value
FROM sample, UNNEST(GENERATE_ARRAY(1, max_reward(REGEXP_EXTRACT_ALL(json, r'"reward_([0-9]+)"')))) seq
) PIVOT (ANY_VALUE(value) FOR reward IN ('reward_1', 'reward_2', 'reward_3', 'reward_4'));
output:
▶ Split a reward STRUCT column into separate columns
SELECT * FROM (
SELECT id,
'reward_' || seq || '_' || IF (offset = 0, 'type', 'amount') AS reward,
value
FROM sample,
UNNEST(GENERATE_ARRAY(1, max_reward(REGEXP_EXTRACT_ALL(json, r'"reward_([0-9]+)"')))) seq,
UNNEST([extract_reward(FORMAT('%t', JSON_QUERY(json, '$.rewards')), seq)]) pair,
UNNEST([pair.type, pair.amount]) value WITH OFFSET
) PIVOT (ANY_VALUE(value) FOR reward IN ('reward_1_type', 'reward_2_type', 'reward_3_type', 'reward_4_type', 'reward_1_amount', 'reward_2_amount', 'reward_3_amount', 'reward_4_amount'));
output:

Subset a JSON Object with MySQL Query

I have a MySQL database and one of the tables is called 'my_table'. In this table, one of the columns is called 'my_json_column' and this column is stored as a JSON object in MySQL. The JSON object has about 17 key:value pairs (see below). I simply want to return a "slimmed-down" JSON Object from a MySQL query that returns 4 of the 17 fields.
I have tried many different MySQL queries, see below, but I can't seem to get a returned subset JSON Object. I am sure it is simple, but I have been unsuccessful.
Something like this:
SELECT
json_extract(my_json_column, '$.X'),
json_extract(my_json_column, '$.Y'),
json_extract(my_json_column, '$.KB'),
json_extract(my_json_column, '$.Name')
FROM my_table;
yields:
5990.510000 90313.550000 5990.510000 "Operator 1"
I want to get this result instead (a returned JSON Object) with key value pairs:
[ { X: 5990.510000, Y: 90313.550, KB: 2105, Name: "Well 1" } ]
Sample data:
{
"Comment" : "No Comment",
"Country" : "USA",
"County" : "County 1",
"Field" : "Field 1",
"GroundElevation" : "5400",
"Identifier" : "11435358700000",
"Interpreter" : "Interpreter 1",
"KB" : 2105,
"Name" : "Well 1",
"Operator" : "Operator 1",
"Owner" : "me",
"SpudDate" : "NA",
"State" : "MI",
"Status" : "ACTIVE",
"TotalDepth" : 5678,
"X" : 5990.510000,
"Y" : 90313.550
}
Thank you in advance.
Use JSON_OBJECT(), available since MySQL 5.6:
Evaluates a (possibly empty) list of key-value pairs and returns a JSON object containing those pairs
SELECT
JSON_OBJECT(
'X', json_extract(my_json_column, '$.X'),
'Y', json_extract(my_json_column, '$.Y'),
'KB', json_extract(my_json_column, '$.KB'),
'Name', json_extract(my_json_column, '$.Name')
) my_new_json
FROM my_table;
This demo on DB Fiddle with your sample data returns:
| my_new_json |
| ----------------------------------------------------------- |
| {"X": 5990.51, "Y": 90313.55, "KB": 2105, "Name": "Well 1"} |

Convert flattened key/value table into hierarchical JSON in PostgreSQL

I have a PostgreSQL table with unique key/value pairs, which were originally in a JSON format, but have been normalized and melted:
key | value
-----------------------------
name | Bob
address.city | Vancouver
address.country | Canada
I need to turn this into a hierarchical JSON:
{
"name": "Bob",
"address": {
"city": "Vancouver",
"country": "Canada"
}
}
Is there a way to do this easily within SQL?
jsonb_set() almost does everything for you, but unfortunately it can only create missing leafs (i.e. missing last keys on a path), but not whole missing branches. To overcome this, here is a modified version of it, which can set values on any missing levels:
create function jsonb_set_rec(jsonb, jsonb, text[])
returns jsonb
language sql
as $$
select case
when array_length($3, 1) > 1 and ($1 #> $3[:array_upper($3, 1) - 1]) is null
then jsonb_set_rec($1, jsonb_build_object($3[array_upper($3, 1)], $2), $3[:array_upper($3, 1) - 1])
else jsonb_set($1, $3, $2, true)
end
$$;
Now you only need to apply this function one-by-one to your rows, starting with an empty json object: {}. You can do this with either recursive CTEs:
with recursive props as (
(select distinct on (grp)
pk, grp, jsonb_set_rec('{}', to_jsonb(value), string_to_array(key, '.')) json_object
from eav_tbl
order by grp, pk)
union all
(select distinct on (grp)
eav_tbl.pk, grp, jsonb_set_rec(json_object, to_jsonb(value), string_to_array(key, '.'))
from props
join eav_tbl using (grp)
where eav_tbl.pk > props.pk
order by grp, eav_tbl.pk)
)
select distinct on (grp)
grp, json_object
from props
order by grp, pk desc;
Or, with a custom aggregate defined as:
create aggregate jsonb_set_agg(jsonb, text[]) (
sfunc = jsonb_set_rec,
stype = jsonb,
initcond = '{}'
);
your query could became as simple as:
select grp, jsonb_set_agg(to_jsonb(value), string_to_array(key, '.'))
from eav_tbl
group by grp;
https://rextester.com/TULNU73750
There are no ready to use tools for this. The function generates a hierarchical json object based on a path:
create or replace function jsonb_build_object_from_path(path text, value text)
returns jsonb language plpgsql as $$
declare
obj jsonb;
keys text[] := string_to_array(path, '.');
level int := cardinality(keys);
begin
obj := jsonb_build_object(keys[level], value);
while level > 1 loop
level := level- 1;
obj := jsonb_build_object(keys[level], obj);
end loop;
return obj;
end $$;
You also need the aggregate function jsonb_merge_agg(jsonb) described in this answer. The query:
with my_table (path, value) as (
values
('name', 'Bob'),
('address.city', 'Vancouver'),
('address.country', 'Canada'),
('first.second.third', 'value')
)
select jsonb_merge_agg(jsonb_build_object_from_path(path, value))
from my_table;
gives this object:
{
"name": "Bob",
"first":
{
"second":
{
"third": "value"
}
},
"address":
{
"city": "Vancouver",
"country": "Canada"
}
}
The function do not recognize json arrays.
I can't really think of something simpler, although I think there should be an easier way.
I assume there is some additional column that can be used to bring the keys that belong to one "person" together, I used p_id for that in my example.
select p_id,
jsonb_object_agg(k, case level when 1 then v -> k else v end)
from (
select p_id,
elements[1] k,
jsonb_object_agg(case cardinality(elements) when 1 then ky else elements[2] end, value) v,
max(cardinality(elements)) as level
from (
select p_id,
"key" as ky,
string_to_array("key", '.') as elements, value
from kv
) t1
group by p_id, k
) t2
group by p_id;
The innermost query just converts the dot notation to an array for easier access later.
The next level then builds JSON objects depending on the "key". For the "single level" keys, it just uses key/value, for the others it uses the second element + the value and then aggregates those that belong together.
The second query level returns the following:
p_id | k | v | level
-----+---------+--------------------------------------------+------
1 | address | {"city": "Vancouver", "country": "Canada"} | 2
1 | name | {"name": "Bob"} | 1
2 | address | {"city": "Munich", "country": "Germany"} | 2
2 | name | {"name": "John"} | 1
The aggregation done in the second step, leaves one level too much for the "single element" keys, and that's what we need level for.
If that distinction wasn't made, the final aggregation would return {"name": {"name": "Bob"}, "address": {"city": "Vancouver", "country": "Canada"}} instead of the wanted: {"name": "Bob", "address": {"city": "Vancouver", "country": "Canada"}}.
The expression case level when 1 then v -> k else v end essentially turns {"name": "Bob"} back to "Bob".
So, with the following sample data:
create table kv (p_id integer, "key" text, value text);
insert into kv
values
(1, 'name','Bob'),
(1, 'address.city','Vancouver'),
(1, 'address.country','Canada'),
(2, 'name','John'),
(2, 'address.city','Munich'),
(2, 'address.country','Germany');
then query returns:
p_id | jsonb_object_agg
-----+-----------------------------------------------------------------------
1 | {"name": "Bob", "address": {"city": "Vancouver", "country": "Canada"}}
2 | {"name": "John", "address": {"city": "Munich", "country": "Germany"}}
Online example: https://rextester.com/SJOTCD7977
create table kv (key text, value text);
insert into kv
values
('name','Bob'),
('address.city','Vancouver'),
('address.country','Canada'),
('name','John'),
('address.city','Munich'),
('address.country','Germany');
create view v_kv as select row_number() over() as nRec, key, value from kv;
create view v_datos as
select k1.nrec, k1.value as name, k2.value as address_city, k3.value as address_country
from v_kv k1 inner join v_kv k2 on (k1.nrec + 1 = k2.nrec)
inner join v_kv k3 on ((k1.nrec + 2= k3.nrec) and (k2.nrec + 1 = k3.nrec))
where mod(k1.nrec, 3) = 1;
select json_agg(json_build_object('name',name, 'address', json_build_object('city',address_city, 'country', address_country)))
from v_datos;

How to update jsonb string with PostgreSQL?

I'm using PostgreSQL 9.4.5. I'd like to update a jsonb column.
My table is structured this way:
CREATE TABLE my_table (
gid serial PRIMARY KEY,
"data" jsonb
);
JSON strings are like this:
{"files": [], "ident": {"id": 1, "country": null, "type ": "20"}}
The following SQL doesn't do the job (syntax error - SQL state = 42601):
UPDATE my_table SET "data" -> 'ident' -> 'country' = 'Belgium';
Is there a way to achieve that?
Ok there are two functions:
create or replace function set_jsonb_value(p_j jsonb, p_key text, p_value jsonb) returns jsonb as $$
select jsonb_object_agg(t.key, t.value) from (
select
key,
case
when jsonb_typeof(value) = 'object' then set_jsonb_value(value, p_key, p_value)
when key = p_key then p_value
else value
end as value from jsonb_each(p_j)) as t;
$$ language sql immutable;
First one just changes the value of the existing key regardless of the key path:
postgres=# select set_jsonb_value(
'{"files": [], "country": null, "ident": {"id": 1, "country": null, "type ": "20"}}',
'country',
'"foo"');
set_jsonb_value
--------------------------------------------------------------------------------------
{"files": [], "ident": {"id": 1, "type ": "20", "country": "foo"}, "country": "foo"}
(1 row)
create or replace function set_jsonb_value(p_j jsonb, p_path text[], p_value jsonb) returns jsonb as $$
select jsonb_object_agg(t.key, t.value) from (
select
key,
case
when jsonb_typeof(value) = 'object' then set_jsonb_value(value, p_path[2:1000], p_value)
when key = p_path[1] then p_value
else value
end as value from jsonb_each(p_j)
union all
select
p_path[1],
case
when array_length(p_path,1) = 1 then p_value
else set_jsonb_value('{}', p_path[2:1000], p_value) end
where not p_j ? p_path[1]) as t;
$$ language sql immutable;
Second one changes the value of the existing key using the path specified or creates it if the path does not exists:
postgres=# select set_jsonb_value(
'{"files": [], "country": null, "ident": {"id": 1, "type ": "20"}}',
'{ident,country}'::text[],
'"foo"');
set_jsonb_value
-------------------------------------------------------------------------------------
{"files": [], "ident": {"id": 1, "type ": "20", "country": "foo"}, "country": null}
(1 row)
postgres=# select set_jsonb_value(
'{"files": [], "country": null, "ident": {"id": 1, "type ": "20"}}',
'{ident,foo,bar,country}'::text[],
'"foo"');
set_jsonb_value
-------------------------------------------------------------------------------------------------------
{"files": [], "ident": {"id": 1, "foo": {"bar": {"country": "foo"}}, "type ": "20"}, "country": null}
(1 row)
Hope it will help to someone who uses the PostgreSQL < 9.5
Disclaimer: Tested on PostgreSQL 9.5
In PG 9.4 you are out of luck with "easy" solutions like jsonb_set() (9.5). Your only option is to unpack the JSON object, make the changes and re-build the object. That sounds very cumbersome and it is indeed: JSON is horrible to manipulate, no matter how advanced or elaborate the built-in functions.
CREATE TYPE data_ident AS (id integer, country text, "type" integer);
UPDATE my_table
SET "data" = json_build_object('files', "data"->'files', 'ident', ident.j)::jsonb
FROM (
SELECT gid, json_build_object('id', j.id, 'country', 'Belgium', 'type', j."type") AS j
FROM my_table
JOIN LATERAL jsonb_populate_record(null::data_ident, "data"->'ident') j ON true) ident
WHERE my_table.gid = ident.gid;
In the SELECT clause "data"->'ident' is unpacked into a record (for which you need to CREATE TYPE a structure). Then it is built right back into a JSON object with the new country name. In the UPDATE that "ident" object is re-joined with the "files" object and the whole thing cast to a jsonb.
A pure thing of beauty -- just so long as speed is not your thing...
My previous solution relied on 9.5 functionality.
I would recommend instead either going with abelisto's solutions below or using pl/perlu, plpythonu, or plv8js to write json mutators in a language that has better support for them.