I'm struggling with JSONB_PATH_EXISTS Postgres function
I'm using PG 12 and following this documentation : https://www.postgresql.org/docs/12/functions-json.html
With the following request (test it on DBFiddle: https://dbfiddle.uk/?rdbms=postgres_12&fiddle=d5aa984182852438c6f71cf5fa70324e) :
select
json
from (
select '{
"fields": {
"foo": true,
"number": 3,
"listnb": [3, 4],
"listenb2": ["3", "4"],
"txt": "hello how are you",
"listtxt": ["hello","how","are", "you", "3"],
"nullval": null
}
}'::jsonb as json
) t
where 1=1
-- Works with 'strict'
AND JSONB_PATH_EXISTS(json -> 'fields' -> 'listtxt', 'strict $ ? (#.type() == "array")')
-- Doesn't work without 'strict'. Why ?
--AND JSONB_PATH_EXISTS(json -> 'fields' -> 'listtxt', '$ ? (#.type() == "array")')
-- Can't add a nested condition on an array element value (syntax error)
--AND JSONB_PATH_EXISTS(json -> 'fields' -> 'listtxt', 'strict $ ? (#.type() == "array" && #[*] ? (# == "how"))')
;
#1 - I can't get type() function work without strict mode
It could be related to the lax mode unwrapping arrays automatically, but the documentation explicitly states that it is not done when type() function is called :
The lax mode facilitates matching of a JSON document structure and path expression if the JSON data does not conform to the expected schema. [...] Automatic unwrapping is not performed only when:
The path expression contains type() or size() methods that return the type and the number of elements in the array, respectively.
[...]
So I don't understand why we have a difference in the result
#2 I can't get the nested condition work (3rd AND in the sample request)
According to the examples in the documentation, the syntax looks OK but I have a syntax error that I don't understand.
Thank you for your help
If you pass the complete JSON value to the function, then the following works:
where jsonb_path_exists(json, '$ ? (#.fields.listtxt.type() == "array")')
However I would probably simply use jsonb_typeof() without a path query
where jsonb_typeof(json -> 'fields' -> 'listtxt') = 'array'
Related
I'm struggling with JSONB_PATH_EXISTS Postgres function
I'm using PG 12 and following this documentation : https://www.postgresql.org/docs/12/functions-json.html
With the following request (test it on DBFiddle: https://dbfiddle.uk/?rdbms=postgres_12&fiddle=d5aa984182852438c6f71cf5fa70324e) :
select
json
from (
select '{
"fields": {
"foo": true,
"number": 3,
"listnb": [3, 4],
"listenb2": ["3", "4"],
"txt": "hello how are you",
"listtxt": ["hello","how","are", "you", "3"],
"nullval": null
}
}'::jsonb as json
) t
where 1=1
-- Works with 'strict'
AND JSONB_PATH_EXISTS(json -> 'fields' -> 'listtxt', 'strict $ ? (#.type() == "array")')
-- Doesn't work without 'strict'. Why ?
--AND JSONB_PATH_EXISTS(json -> 'fields' -> 'listtxt', '$ ? (#.type() == "array")')
-- Can't add a nested condition on an array element value (syntax error)
--AND JSONB_PATH_EXISTS(json -> 'fields' -> 'listtxt', 'strict $ ? (#.type() == "array" && #[*] ? (# == "how"))')
;
#1 - I can't get type() function work without strict mode
It could be related to the lax mode unwrapping arrays automatically, but the documentation explicitly states that it is not done when type() function is called :
The lax mode facilitates matching of a JSON document structure and path expression if the JSON data does not conform to the expected schema. [...] Automatic unwrapping is not performed only when:
The path expression contains type() or size() methods that return the type and the number of elements in the array, respectively.
[...]
So I don't understand why we have a difference in the result
#2 I can't get the nested condition work (3rd AND in the sample request)
According to the examples in the documentation, the syntax looks OK but I have a syntax error that I don't understand.
Thank you for your help
If you pass the complete JSON value to the function, then the following works:
where jsonb_path_exists(json, '$ ? (#.fields.listtxt.type() == "array")')
However I would probably simply use jsonb_typeof() without a path query
where jsonb_typeof(json -> 'fields' -> 'listtxt') = 'array'
I've wrote a program which process JSON objects. Now I want to verify if I've missed something.
Is there an JSON-example of all allowed JSON structure combinations? Something like this:
{
"key1" : "value",
"key2" : 1,
"key3" : {"key1" : "value"},
"key4" : [
[
"string1",
"string2"
],
[
1,
2
],
...
],
"key5" : true,
"key6" : false,
"key7" : null,
...
}
As you can see at http://json.org/ on the right hand side the grammar of JSON isn't quite difficult, but I've got several exceptions because I've forgotten to handles some structure combinations which are possible. E.g. inside an array there can be "string, number, object, array, true, false, null" but my program couldn't handle arrays inside an array until I ran into an exception. So everything was fine until I got this valid JSON object with arrays inside an array.
I want to test my program with a JSON object (which I'm looking for). After this test I want to be feel certain that my program handle every possible valid JSON structure on earth without an exception.
I don't need nesting in depth 5 or so. I only need something in nested depth 2 or max 3. With all base types which nested all allowed base types, inside this base type.
Have you thought of escaped characters and objects within an object?
{
"key1" : {
"key1" : "value",
"key2" : [
"String1",
"String2"
],
},
"key2" : "\"This is a quote\"",
"key3" : "This contains an escaped slash: \\",
"key4" : "This contains accent charachters: \u00eb \u00ef",
}
Note: \u00eb and \u00ef are resp. charachters ë and ï
Choose a programming language that support json.
Try to load your json, on fail the exception's message is descriptive.
Example:
Python:
import json, sys;
json.loads(open(sys.argv[1]).read())
Generate:
import random, json, os, string
def json_null(depth = 0):
return None
def json_int(depth = 0):
return random.randint(-999, 999)
def json_float(depth = 0):
return random.uniform(-999, 999)
def json_string(depth = 0):
return ''.join(random.sample(string.printable, random.randrange(10, 40)))
def json_bool(depth = 0):
return random.randint(0, 1) == 1
def json_list(depth):
lst = []
if depth:
for i in range(random.randrange(8)):
lst.append(gen_json(random.randrange(depth)))
return lst
def json_object(depth):
obj = {}
if depth:
for i in range(random.randrange(8)):
obj[json_string()] = gen_json(random.randrange(depth))
return obj
def gen_json(depth = 8):
if depth:
return random.choice([json_list, json_object])(depth)
else:
return random.choice([json_null, json_int, json_float, json_string, json_bool])(depth)
print(json.dumps(gen_json(), indent = 2))
I am a total beginner with Erlang and functional programming in general. For fun, to get me started, I am converting an existing Ruby Sinatra REST(ish) API that queries PostgreSQL and returns JSON.
On the Erlang side I am using Cowboy, Epgsql and Jiffy as the JSON library.
Epgsql returns results in the following format:
{ok, [{column,<<"column_name">>,int4,4,-1,0}], [{<<"value">>}]}
But Jiffy expects the following format when encoding to JSON:
{[{<<"column_name">>,<<"value">>}]}
The following code works to convert epgsql output into suitable input for jiffy:
Assuming Data is the Epgsql output and Key is the name of the JSON object being created:
{_, C, R} = Data,
Columns = [X || {_, X, _, _, _, _} <- C,
Rows = tuple_to_list(hd(R)),
Result = {[{atom_to_binary(Key, utf8), {lists:zip(Columns, Rows)}}]}.
However, I am wondering if this is efficient Erlang?
I've looked into the documentation for Epgsql and Jiffy and can't see any more obvious ways to perform the conversion.
Thank you.
Yes, need parse it.
For example function parse result
parse_result({error, #error{ code = <<"23505">>, extra = Extra }}) ->
{match, [Column]} =
re:run(proplists:get_value(detail, Extra),
"Key \\(([^\\)]+)\\)", [{capture, all_but_first, binary}]),
throw({error, {non_unique, Column}});
parse_result({error, #error{ message = Msg }}) ->
throw({error, Msg});
parse_result({ok, Cols, Rows}) ->
to_map(Cols, Rows);
parse_result({ok, Counts, Cols, Rows}) ->
{ok, Counts, to_map(Cols, Rows)};
parse_result(Result) ->
Result.
And function convert result to map
to_map(Cols, Rows) ->
[ maps:from_list(lists:zipwith(fun(#column{name = N}, V) -> {N, V} end,
Cols, tuple_to_list(Row))) || Row <- Rows ].
And encode it to json. You can change my code and make output as proplist.
I am using Postgres' json data type but want to do a query/ordering with data that is nested within the json.
I want to order or query with .where on the json data type. For example, I want to query for users that have a follower count > 500 or I want to order by follower or following count.
Thanks!
Example:
model User
data: {
"photos"=>[
{"type"=>"facebook", "type_id"=>"facebook", "type_name"=>"Facebook", "url"=>"facebook.com"}
],
"social_profiles"=>[
{"type"=>"vimeo", "type_id"=>"vimeo", "type_name"=>"Vimeo", "url"=>"http://vimeo.com/", "username"=>"v", "id"=>"1"},
{"bio"=>"I am not a person, but a series of plants", "followers"=>1500, "following"=>240, "type"=>"twitter", "type_id"=>"twitter", "type_name"=>"Twitter", "url"=>"http://www.twitter.com/", "username"=>"123", "id"=>"123"}
]
}
For any who stumbles upon this. I have come up with a list of queries using ActiveRecord and Postgres' JSON data type. Feel free to edit this to make it more clear.
Documentation to the JSON operators used below: https://www.postgresql.org/docs/current/functions-json.html.
# Sort based on the Hstore data:
Post.order("data->'hello' DESC")
=> #<ActiveRecord::Relation [
#<Post id: 4, data: {"hi"=>"23", "hello"=>"22"}>,
#<Post id: 3, data: {"hi"=>"13", "hello"=>"21"}>,
#<Post id: 2, data: {"hi"=>"3", "hello"=>"2"}>,
#<Post id: 1, data: {"hi"=>"2", "hello"=>"1"}>]>
# Where inside a JSON object:
Record.where("data ->> 'likelihood' = '0.89'")
# Example json object:
r.column_data
=> {"data1"=>[1, 2, 3],
"data2"=>"data2-3",
"array"=>[{"hello"=>1}, {"hi"=>2}],
"nest"=>{"nest1"=>"yes"}}
# Nested search:
Record.where("column_data -> 'nest' ->> 'nest1' = 'yes' ")
# Search within array:
Record.where("column_data #>> '{data1,1}' = '2' ")
# Search within a value that's an array:
Record.where("column_data #> '{array,0}' ->> 'hello' = '1' ")
# this only find for one element of the array.
# All elements:
Record.where("column_data ->> 'array' LIKE '%hello%' ") # bad
Record.where("column_data ->> 'array' LIKE ?", "%hello%") # good
According to this http://edgeguides.rubyonrails.org/active_record_postgresql.html#json
there's a difference in using -> and ->>:
# db/migrate/20131220144913_create_events.rb
create_table :events do |t|
t.json 'payload'
end
# app/models/event.rb
class Event < ActiveRecord::Base
end
# Usage
Event.create(payload: { kind: "user_renamed", change: ["jack", "john"]})
event = Event.first
event.payload # => {"kind"=>"user_renamed", "change"=>["jack", "john"]}
## Query based on JSON document
# The -> operator returns the original JSON type (which might be an object), whereas ->> returns text
Event.where("payload->>'kind' = ?", "user_renamed")
So you should try Record.where("data ->> 'status' = 200 ") or the operator that suits your query (http://www.postgresql.org/docs/current/static/functions-json.html).
Your question doesn't seem to correspond to the data you've shown, but if your table is named users and data is a field in that table with JSON like {count:123}, then the query
SELECT * WHERE data->'count' > 500 FROM users
will work. Take a look at your database schema to make sure you understand the layout and check that the query works before complicating it with Rails conventions.
JSON filtering in Rails
Event.create( payload: [{ "name": 'Jack', "age": 12 },
{ "name": 'John', "age": 13 },
{ "name": 'Dohn', "age": 24 }]
Event.where('payload #> ?', '[{"age": 12}]')
#You can also filter by name key
Event.where('payload #> ?', '[{"name": "John"}]')
#You can also filter by {"name":"Jack", "age":12}
Event.where('payload #> ?', {"name":"Jack", "age":12}.to_json)
You can find more about this here
We can access any JSON element in PostgreSQL 9.3 using the -> and ->> operators. Seems to me that the #> along with #>> only provide a shorter form of writing the JSON path. Or is there a bigger picture behind the #> operator? Does it serve a special purpose/provide any advantage over the arrow notation? Which one is the preffered method of writing paths?
It all comes to the question: why should I use the #> and #>> operator over the -> and ->>?
The docs are a bit enigmatic about this.
Both queries below give the same result:
=> select '{"a":[1,2,3],"b":[4,5,6]}'::json#>'{a,2}';
?column?
----------
3
=> select '{"a":[1,2,3],"b":[4,5,6]}'::json->'a'->>2;
?column?
----------
3
Consider nesting.
{
"a" : {
"b" : {
"c" : 1,
"d" : 2
}
}
}
Imagine that you have a json document, and you don't know in advance how it will be nested. If you knew you needed a three level path, you could write:
SELECT '{
"a" : {
"b" : {
"c" : 1,
"d" : 2
}
}
}'::json -> 'a' -> 'b' -> 'c';
but what if you want to write a query that doesn't know that in advance? That's where the path-based operators are useful; the path can be supplied along with the document, and there's no longer any assumption about the document structure in the query.
SELECT '{
"a" : {
"b" : {
"c" : 1,
"d" : 2
}
}
}'::json #>> ARRAY['a','b','c']