Related
I am using JSON_MODIFY to build complex JSON. Moving from MySQL I am struggling with the JSON functions provided by SQL Server. The issue I'm having is that SQL Server seems to construct all JSON objects in an array. There is the WITHOUT_ARRAY_WRAPPER mechanism, which seems like it should do what I want, however; there are two undesirable consequences.
It only returns one result depending on how it is used
The result is a single string with escape characters
I have constructed a simple query which illustrates my needs and the issue.
QUERY 1
SELECT JSON_MODIFY(
JSON_QUERY('{"definitions": {"id": "INT", "name": "VARCHAR(23)"}}'),
'append $.data',
(
SELECT * FROM (
SELECT 1 AS id, '123abc' AS "name" UNION
SELECT 2 AS id, '234bcd' AS "name"
) AS "data"
FOR JSON PATH, WITHOUT_ARRAY_WRAPPER
)
) AS "data";
OUTPUT 1
{
"definitions":{
"id":"INT",
"name":"VARCHAR(23)"
},
"data":[
"{\"id\":1,\"name\":\"123abc\"},{\"id\":2,\"name\":\"234bcd\"}"
]
}
QUERY 2
SELECT JSON_MODIFY(
JSON_QUERY('{"definitions": {"id": "INT", "name": "VARCHAR(23)"}}'),
'append $.data',
(
SELECT * FROM (
SELECT 1 AS id, '123abc' AS "name" UNION
SELECT 2 AS id, '234bcd' AS "name"
) AS "data"
FOR JSON PATH
)
) AS "data";
OUTPUT 2
{
"definitions":{
"id":"INT",
"name":"VARCHAR(23)"
},
"data":[
[
{"id":1, "name":"123abc"},
{"id":2, "name":"234bcd"}
]
]
}
QUERY 1
The data object is an array (which is expected), but the problem is what is in the array... A single string with escape characters.
QUERY 2
The data object is an array, which contains an array. In order to access the actual array of data, I would use something like for each obj in data[0].... The problem this poses is, for anyone consuming the JSON object, I would have to tell them:
"In this particular object the data element is an array of
arrays--You'll want to use the first and only the first
element to access the actual array of data."
I've naively tried many different combinations of JSON_MODIFY, JSON_QUERY, and CONCAT to no avail. How can I properly use JSON_MODIFY to get the following output, without the double array in data?
{
"definitions":{
"id":"INT",
"name":"VARCHAR(23)"
},
"data":[
{"id":1, "name":"123abc"},
{"id":2, "name":"234bcd"}
]
}
You are over-thinking this by trying to JSON_MODIFY an existing object.
Construct the definitions and data properties that you need, inside a subquery if necessary.
Then use FOR JSON a second time to get the outer object.
SELECT
definitions = JSON_QUERY('{"id": "INT", "name": "VARCHAR(23)"}'),
data =
(
SELECT id, name
FROM (VALUES
(1, '123abc'),
(2, '234bcd')
) v(id, name)
FOR JSON PATH
)
FOR JSON PATH;
SQL Fiddle
By trial and error, I found the solution.
Removed the append keyword from the path parameter in the JSON_MODIFY statement
Removed the WITHOUT_ARRAY_WRAPPER parameter from the FOR JSON statement.
Now the results are as expected and I don't need to explain to any consumers to "Just use data[0]"
The Query
SELECT JSON_MODIFY(
JSON_QUERY('{"definitions": {"id": "INT", "name": "VARCHAR(23)"}}'),
'$.data',
(
SELECT * FROM (
SELECT 1 AS id, '123abc' AS "name" UNION
SELECT 2 AS id, '234bcd' AS "name"
) AS "data"
FOR JSON PATH
)
) AS "data";
Produces the following output
{
"definitions":{
"id":"INT",
"name":"VARCHAR(23)"
},
"data":[
{"id":1, "name":"123abc"},
{"id":2, "name":"234bcd"}
]
}
I've made a simple function to update a jsonb with new values:
CREATE OR REPLACE FUNCTION jsonupdate(
IN "pJson" jsonb, IN "pNewValues" jsonb)
RETURNS jsonb AS
$BODY$
DECLARE
jsonreturn jsonb;
BEGIN
jsonreturn := (SELECT json_object_agg(keyval.key, keyval.value::jsonb)
FROM (SELECT key,
CASE WHEN "pNewValues" ? key THEN
(SELECT "pNewValues" ->> key)
ELSE
value
END
FROM jsonb_each_text("pJson")) keyval);
RETURN jsonreturn;
END; $BODY$
LANGUAGE plpgsql IMMUTABLE
COST 100;
Sample inputs and outputs:
IN: SELECT jsonupdate('{"a" : "1", "b" : "2"}', '{"a": "3"}');
OUT: {"a": 3, "b": 2}
IN: SELECT jsonupdate('{"a" : "3", "b" : { "c": "text", "d": 1 }}', '{"b": { "c": "another text" }}');
OUT: {"a": 3, "b": {"c": "another text"}}
IN: SELECT jsonupdate('{"a" : "1", "b" : "2", "c": 3, "d": 4}', '{"a": "5", "d": 6}');
OUT: {"a": 5, "b": 2, "c": 3, "d": 6}
The problem happens when using inputs like this one: SELECT jsonupdate('{"a" : "1", "b" : ""}', '{"a": "5"}') or this one: SELECT jsonupdate('{"a" : "1", "b" : "2"}', '{"a": "."}') or this one: SELECT jsonupdate('{"a" : "1", "b" : "2"}', '{"a": ""}') it gives me an error
ERROR: invalid input syntax for type json
DETAIL: The input string ended unexpectedly.
CONTEXT: JSON data, line 1:
What's wrong here?
You sould use the jsonb_each() function (instead of jsonb_each_text()). Also, the -> operator (instead of ->>):
CREATE OR REPLACE FUNCTION jsonupdate(IN "pJson" jsonb, IN "pNewValues" jsonb)
RETURNS jsonb
LANGUAGE sql
IMMUTABLE AS
$BODY$
SELECT json_object_agg(key, CASE
WHEN "pNewValues" ? key THEN "pNewValues" -> key
ELSE value
END)
FROM jsonb_each("pJson")
$BODY$;
jsonb_each_text() and the ->> operator converts any non-string JSON value to their string representation. Converting those back to JSON will modify your data in a way you probably don't want to.
But I have to admit, what you are trying to achieve is almost the || (concatenation) operator. I.e.
SELECT jsonb '{"a" : "1", "b" : "2"}' || jsonb '{"a": "3"}'
will give you your desired output. The only difference between || and your function is when pNewValues contains key(s), which are not in pJson: || will append those too, while your function does not append them (it only modifies existing ones).
Update: for simulating the || operator on 9.4, you can use the following function:
CREATE OR REPLACE FUNCTION jsonb_merge_objects(jsonb, jsonb)
RETURNS jsonb
LANGUAGE sql
IMMUTABLE AS
$func$
SELECT json_object_agg(key, COALESCE(b.value, a.value))
FROM jsonb_each($1) a
LEFT JOIN jsonb_each($2) b USING (key)
$func$;
Using the || operator yields the following result:
select '{"a":{"b":2}}'::jsonb || '{"a":{"c":3}}'::jsonb ;
?column?
-----------------
{"a": {"c": 3}}
(1 row)
I would like to be able to do achieve the following result (?? just a placeholder for the operator):
select '{"a":{"b":2}}'::jsonb ?? '{"a":{"c":3}}'::jsonb ;
?column?
-----------------
{"a": {"b": 2, "c": 3}}
(1 row)
So, you can see the top-level a key has its child values "merged" such that the result contains both b and c.
How do you "deep" merge two JSONB values in Postgres?
Is this possible, if so how?
A more complex test case:
select '{"a":{"b":{"c":3},"z":true}}'::jsonb ?? '{"a":{"b":{"d":4},"z":false}}'::jsonb ;
?column?
-----------------
{"a": {"b": {"c": 3, "d": 4}, "z": false}}
(1 row)
Another test case where a primitive "merges over" and object:
select '{"a":{"b":{"c":3},"z":true}}'::jsonb ?? '{"a":{"b":false,"z":false}}'::jsonb ;
?column?
-----------------
{"a": {"b": false, "z": false}}
(1 row)
You should merge unnested elements using jsonb_each() for both values. Doing this in a non-trivial query may be uncomfortable, so I would prefer a custom function like this one:
create or replace function jsonb_my_merge(a jsonb, b jsonb)
returns jsonb language sql as $$
select
jsonb_object_agg(
coalesce(ka, kb),
case
when va isnull then vb
when vb isnull then va
else va || vb
end
)
from jsonb_each(a) e1(ka, va)
full join jsonb_each(b) e2(kb, vb) on ka = kb
$$;
Use:
select jsonb_my_merge(
'{"a":{"b":2}, "d": {"e": 10}, "x": 1}'::jsonb,
'{"a":{"c":3}, "d": {"f": 11}, "y": 2}'::jsonb
)
jsonb_my_merge
------------------------------------------------------------------
{"a": {"b": 2, "c": 3}, "d": {"e": 10, "f": 11}, "x": 1, "y": 2}
(1 row)
You can slightly modify the function using recursion to get a solution working on any level of nesting:
create or replace function jsonb_recursive_merge(a jsonb, b jsonb)
returns jsonb language sql as $$
select
jsonb_object_agg(
coalesce(ka, kb),
case
when va isnull then vb
when vb isnull then va
when jsonb_typeof(va) <> 'object' then va || vb
else jsonb_recursive_merge(va, vb)
end
)
from jsonb_each(a) e1(ka, va)
full join jsonb_each(b) e2(kb, vb) on ka = kb
$$;
Examples:
select jsonb_recursive_merge(
'{"a":{"b":{"c":3},"x":5}}'::jsonb,
'{"a":{"b":{"d":4},"y":6}}'::jsonb);
jsonb_recursive_merge
------------------------------------------------
{"a": {"b": {"c": 3, "d": 4}, "x": 5, "y": 6}}
(1 row)
select jsonb_recursive_merge(
'{"a":{"b":{"c":{"d":{"e":1}}}}}'::jsonb,
'{"a":{"b":{"c":{"d":{"f":2}}}}}'::jsonb)
jsonb_recursive_merge
----------------------------------------------
{"a": {"b": {"c": {"d": {"e": 1, "f": 2}}}}}
(1 row)
Finally, the variant of the function with changes proposed by OP (see comments below):
create or replace function jsonb_recursive_merge(a jsonb, b jsonb)
returns jsonb language sql as $$
select
jsonb_object_agg(
coalesce(ka, kb),
case
when va isnull then vb
when vb isnull then va
when jsonb_typeof(va) <> 'object' or jsonb_typeof(vb) <> 'object' then vb
else jsonb_recursive_merge(va, vb) end
)
from jsonb_each(a) e1(ka, va)
full join jsonb_each(b) e2(kb, vb) on ka = kb
$$;
This kind of "deep merge" can be interpreted quite differently, depending on your use case. For completeness, my intuition usually dictates the following rules:
object + object: Every property survives from each object, which is not in the other object (JSON's null value is considered to be in the object, if it's explicitly mentioned). When a property is in both objects, the merge continues recursively with the same rules (this point is usually agreed on).
array + array: The result is the concatenation of the two arrays.
array + primitive/object: the result is the first array, with the second JSON value appended to it.
any other cases: The result is the second JSON value (so f.ex. primitives or incompatible types override each other).
create or replace function jsonb_merge_deep(jsonb, jsonb)
returns jsonb
language sql
immutable
as $func$
select case jsonb_typeof($1)
when 'object' then case jsonb_typeof($2)
when 'object' then (
select jsonb_object_agg(k, case
when e2.v is null then e1.v
when e1.v is null then e2.v
else jsonb_merge_deep(e1.v, e2.v)
end)
from jsonb_each($1) e1(k, v)
full join jsonb_each($2) e2(k, v) using (k)
)
else $2
end
when 'array' then $1 || $2
else $2
end
$func$;
This function's added bonus is that it can be called with literally any type of JSON values: always produces a result & never complains about JSON value types.
http://rextester.com/FAC95623
After PostgreSQL 9.5 you can use jsonb_set function:
'{a,c}' looking into path if it is not there, it will created.
'{"a":{"c":3}}'::jsonb#>'{a,c}' this will get the value of c
new_value added if create_missing is true ( default is true)
Hier is document jsonb -functions
select jsonb_set('{"a":{"b":2}}', '{a,c}','{"a":{"c":3}}'::jsonb#>'{a,c}' )
Result: {"a":{"c":3,"b":2}}
Merge more attribute at once:
with jsonb_paths(main_part,missing_part) as (
values ('{"a":{"b":2}}','{"a":{"c":3,"d":4}}')
)
select jsonb_object_agg(t.k,t.v||t2.v)
from jsonb_paths,
jsonb_each(main_part::jsonb) t(k,v),
jsonb_each(missing_part::jsonb) t2(k,v);
result: {"a":{"c":3,"b":2,"d":4}}
As #lightSouls say, after PostgreSQL 9.5 you can use jsonb_set() function... But you must to learn how to use it!
jsonb_set can merge or destroy...
Supposing j:='{"a":{"x":1},"b":2}'::jsonb.
jsonb_set(j, '{a,y}', '1'::jsonb); will merge object {"y":1} with object {"x":1}. Result: {"a": {"x": 1, "y": 1}, "b": 2}
jsonb_set(j, '{a}', '{"x":1}'::jsonb); will destroy! replacing full old object by the new one.Result: {"a": {"x": 1}, "b": 2}
Combining (merging :-D) answers from #klin, #pozs and comment from #Arman Khubezhov while also actually merging arrays instead of concatenating (which resulted in duplicates otherwise), came up with the following function:
create or replace function jsonb_merge_deep(jsonb, jsonb)
returns jsonb
language sql
immutable
as $func$
select case jsonb_typeof($1)
when 'object' then
case jsonb_typeof($2)
when 'object' then (
select jsonb_object_agg(k,
case
when e2.v is null then e1.v
when e1.v is null then e2.v
else jsonb_merge_deep(e1.v, e2.v)
end
)
from jsonb_each($1) e1(k, v)
full join jsonb_each($2) e2(k, v) using (k)
)
else COALESCE($2, $1)
end
when 'array' then
(
SELECT jsonb_agg(items.val)
FROM (
SELECT jsonb_array_elements($1) AS val
UNION
SELECT jsonb_array_elements($2) AS val
) AS items
)
else $2
end
$func$;
Based on comment from #Arman Khubezhov, enhanced the case when any of $1 or $2 is null with:
else COALESCE($2, $1)
And added real merge (no duplicate) of the 2 arrays values with:
when 'array' then
(
SELECT jsonb_agg(items.val)
FROM (
SELECT jsonb_array_elements($1) AS val
UNION
SELECT jsonb_array_elements($2) AS val
) AS items
)
Glad if one can come up with a enhanced code for this one - like an existing PostreSQL function I am not aware of?
Pros: no data loss when combining 2 JSONB values or updating a JSONB field in an UPDATE query like.
UPDATE my_table
SET my_jsonb_field = jsonb_merge_deep(my_jsonb_field, '{ "a": { "aa" : { "aaa" : [6, 4, 7] } } }'::jsonb)
Cons: removing a key/value or array value requires a dedicated query.
I was testing some queries at pg9.4 in "JSON mode", and now I am checking if pg9.5 will bring all same JSONB functionality... But there are no row_to_jsonb() function (!). (why it is not orthogonal instruction set in the basic parameters?)
The guide only says "the to_jsonb function supplies much the same functionality". Where we can check "how much"? There are other specific JSONB guide about this details?
((Year 2022 update and pg upgrade))
The phrase "supplies much the same functionality" was removed on the version 13. The current Guide does not use the phrase neither the word "much".
Now row_to_json is an alias for to_json except when the optional boolean parameter is true — the result will be the inclusion of line feeds like in jsonb_pretty().
Now the functions to_jsonb and to_json are orthogonal (!), and typical use is the same:
SELECT t.a, t.b, to_jsonb(r) json_info
-- or to_json(r)
FROM t, LATERAL (SELECT t.c,t.d,t.f) r;
-- or SELECT to_jsonb(r) FROM (SELECT c,d,f FROM t) r;
You can just use to_jsonb() instead of row_to_json(), example:
with the_table(a, b, c) as (
select 1, 'alfa', '2016-01-01'::date
)
select to_jsonb(t), row_to_json(t)
from the_table t;
to_jsonb | row_to_json
------------------------------------------+-------------------------------------
{"a": 1, "b": "alfa", "c": "2016-01-01"} | {"a":1,"b":"alfa","c":"2016-01-01"}
(1 row)
The first has a wider application than the other because of the type of arguments (anyelement versus record). For example, you can convert a Postgres array to json array using to_jsonb(), that cannot be done with row_to_json():
select to_jsonb(array['a', 'b', 'c']);
to_jsonb
-----------------
["a", "b", "c"]
(1 row)
In case of the use of two arguments in row_to_json() you should additionally use jsonb_pretty():
with the_table(a, b, c) as (
select 1, 'alfa', '2016-01-01'::date
)
select jsonb_pretty(to_jsonb(t)), row_to_json(t, true)
from the_table t;
jsonb_pretty | row_to_json
-----------------------+--------------------
{ +| {"a":1, +
"a": 1, +| "b":"alfa", +
"b": "alfa", +| "c":"2016-01-01"}
"c": "2016-01-01"+|
} |
(1 row)
You can use to_jsonb as a drop-in replacement for row_to_json.
SELECT to_jsonb(rows) FROM (SELECT * FROM table) rows;
you can cast json to jsonb row_to_json(...)::jsonb, not ideal but often does the trick
I have two Postgres SQL queries returning JSON arrays:
q1:
[
{"id": 1, "a": "text1a", "b": "text1b"},
{"id": 2, "a": "text2a", "b": "text2b"},
{"id": 2, "a": "text3a", "b": "text3b"},
...
]
q2:
[
{"id": 1, "percent": 12.50},
{"id": 2, "percent": 75.00},
{"id": 3, "percent": 12.50}
...
]
I want the result to be a union of both array unique elements:
[
{"id": 1, "a": "text1a", "b": "text1b", "percent": 12.50},
{"id": 2, "a": "text2a", "b": "text2b", "percent": 75.00},
{"id": 3, "a": "text3a", "b": "text3b", "percent": 12.50},
...
]
How can this be done with SQL in Postgres 9.4?
Assuming data type jsonb and that you want to merge records of each JSON array that share the same 'id' value.
Postgres 9.5
makes it simpler with the new concatenate operator || for jsonb values:
SELECT json_agg(elem1 || elem2) AS result
FROM (
SELECT elem1->>'id' AS id, elem1
FROM (
SELECT '[
{"id":1, "percent":12.50},
{"id":2, "percent":75.00},
{"id":3, "percent":12.50}
]'::jsonb AS js
) t, jsonb_array_elements(t.js) elem1
) t1
FULL JOIN (
SELECT elem2->>'id' AS id, elem2
FROM (
SELECT '[
{"id": 1, "a": "text1a", "b": "text1b", "percent":12.50},
{"id": 2, "a": "text2a", "b": "text2b", "percent":75.00},
{"id": 3, "a": "text3a", "b": "text3b", "percent":12.50}]'::jsonb AS js
) t, jsonb_array_elements(t.js) elem2
) t2 USING (id);
The FULL [OUTER] JOIN makes sure you don't lose records without match in the other array.
The type jsonb has the convenient property to only keep the latest value for each key in the record. Hence, the duplicate 'id' key in the result is merged automatically.
The Postgres 9.5 manual also advises:
Note: The || operator concatenates the elements at the top level of
each of its operands. It does not operate recursively. For example, if
both operands are objects with a common key field name, the value of
the field in the result will just be the value from the right hand operand.
Postgres 9.4
Is a bit less convenient. My idea would be to extract array elements, then extract all key/value pairs, UNION both results, aggregate into a single new jsonb values per id value and finally aggregate into a single array.
SELECT json_agg(j) -- ::jsonb
FROM (
SELECT json_object_agg(key, value)::jsonb AS j
FROM (
SELECT elem->>'id' AS id, x.*
FROM (
SELECT '[
{"id":1, "percent":12.50},
{"id":2, "percent":75.00},
{"id":3, "percent":12.50}]'::jsonb AS js
) t, jsonb_array_elements(t.js) elem, jsonb_each(elem) x
UNION ALL -- or UNION, see below
SELECT elem->>'id' AS id, x.*
FROM (
SELECT '[
{"id": 1, "a": "text1a", "b": "text1b", "percent":12.50},
{"id": 2, "a": "text2a", "b": "text2b", "percent":75.00},
{"id": 3, "a": "text3a", "b": "text3b", "percent":12.50}]'::jsonb AS js
) t, jsonb_array_elements(t.js) elem, jsonb_each(elem) x
) t
GROUP BY id
) t;
The cast to jsonb removes duplicate keys. Alternatively you could use UNION to fold duplicates (for instance if you want json as result). Test which is faster for your case.
Related:
How to turn json array into postgres array?
Merging Concatenating JSON(B) columns in query
For any single jsonb element this use of the concat || operator works well for me with strip_nulls and another trick to cast the result back to jsonb (not an array).
select jsonb_array_elements(jsonb_strip_nulls(jsonb_agg(
'{
"a" : "unchanged value",
"b" : "old value",
"d" : "delete me"
}'::jsonb
|| -- The concat operator works as merge on jsonb, the right operand takes precedence
-- NOTE: it only works one JSON level deep
'{
"b" : "NEW value",
"c" : "NEW field",
"d" : null
}'::jsonb
)));
This gives the result
{"a": "unchanged value", "b": "NEW value", "c": "NEW field"}
which is properly typed jsonb