Postgres json type inner Query [duplicate] - json

I am looking for some docs and/or examples for the new JSON functions in PostgreSQL 9.2.
Specifically, given a series of JSON records:
[
{name: "Toby", occupation: "Software Engineer"},
{name: "Zaphod", occupation: "Galactic President"}
]
How would I write the SQL to find a record by name?
In vanilla SQL:
SELECT * from json_data WHERE "name" = "Toby"
The official dev manual is quite sparse:
http://www.postgresql.org/docs/devel/static/datatype-json.html
http://www.postgresql.org/docs/devel/static/functions-json.html
Update I
I've put together a gist detailing what is currently possible with PostgreSQL 9.2.
Using some custom functions, it is possible to do things like:
SELECT id, json_string(data,'name') FROM things
WHERE json_string(data,'name') LIKE 'G%';
Update II
I've now moved my JSON functions into their own project:
PostSQL - a set of functions for transforming PostgreSQL and PL/v8 into a totally awesome JSON document store

Postgres 9.2
I quote Andrew Dunstan on the pgsql-hackers list:
At some stage there will possibly be some json-processing (as opposed
to json-producing) functions, but not in 9.2.
Doesn't prevent him from providing an example implementation in PLV8 that should solve your problem. (Link is dead now, see modern PLV8 instead.)
Postgres 9.3
Offers an arsenal of new functions and operators to add "json-processing".
The manual on new JSON functionality.
The Postgres Wiki on new features in pg 9.3.
The answer to the original question in Postgres 9.3:
SELECT *
FROM json_array_elements(
'[{"name": "Toby", "occupation": "Software Engineer"},
{"name": "Zaphod", "occupation": "Galactic President"} ]'
) AS elem
WHERE elem->>'name' = 'Toby';
Advanced example:
Query combinations with nested array of records in JSON datatype
For bigger tables you may want to add an expression index to increase performance:
Index for finding an element in a JSON array
Postgres 9.4
Adds jsonb (b for "binary", values are stored as native Postgres types) and yet more functionality for both types. In addition to expression indexes mentioned above, jsonb also supports GIN, btree and hash indexes, GIN being the most potent of these.
The manual on json and jsonb data types and functions.
The Postgres Wiki on JSONB in pg 9.4
The manual goes as far as suggesting:
In general, most applications should prefer to store JSON data as
jsonb, unless there are quite specialized needs, such as legacy
assumptions about ordering of object keys.
Bold emphasis mine.
Performance benefits from general improvements to GIN indexes.
Postgres 9.5
Complete jsonb functions and operators. Add more functions to manipulate jsonb in place and for display.
Major good news in the release notes of Postgres 9.5.

With Postgres 9.3+, just use the -> operator. For example,
SELECT data->'images'->'thumbnail'->'url' AS thumb FROM instagram;
see http://clarkdave.net/2013/06/what-can-you-do-with-postgresql-and-json/ for some nice examples and a tutorial.

With postgres 9.3 use -> for object access. 4 example
seed.rb
se = SmartElement.new
se.data =
{
params:
[
{
type: 1,
code: 1,
value: 2012,
description: 'year of producction'
},
{
type: 1,
code: 2,
value: 30,
description: 'length'
}
]
}
se.save
rails c
SELECT data->'params'->0 as data FROM smart_elements;
returns
data
----------------------------------------------------------------------
{"type":1,"code":1,"value":2012,"description":"year of producction"}
(1 row)
You can continue nesting
SELECT data->'params'->0->'type' as data FROM smart_elements;
return
data
------
1
(1 row)

Related

Entity Framework Queries For Complicated JSON Documents (npgsql)

I am handling legacy (old) JSON files that we are now uploading to a database that was built using code-first EF Core (with the JSON elements saved as a jsonb field in a postgresql db, represented as JsonDocument properties in the EF classes). We want to be able to query these massive documents against any of the JSON's many properties. I've been very interested in the excellent docs here https://www.npgsql.org/efcore/mapping/json.html?tabs=data-annotations%2Cpoco, but the problem in our case is that our JSON has incredibly complicated hierarchies.
According to the npgsql/EF doc above, a way to do this for "shallow" json hierarchies would be something like:
myDbContext.MyClass
.Where(e => e.JsonDocumentField.RootElement.GetProperty("FieldToSearch").GetString() == "SearchTerm")
.ToList();
But that only works if is directly under the root of the JSONDocument. If the doc is structed like, say
{"A": {
...
"B": {
...
"C": {
...
"FieldToSearch":
<snip>
Then the above query won't work. There is an alternative to map our JSON to an actual POCO model, but this JSON structure (a) may change and (b) is truly massive and would result in some ridiculously complicated objects.
Right now, I'm building SQL strings using field configurations where I save strings to find the fields I want using psql's JSON querying language
Example:
"(JSONDocumentField->'A'->'B'->'C'->>'FieldToSearch')"
and then using that sql against the DB using
myDbContext.MyClass.FromSqlRaw(sql).ToList();
This is hacky and I'd much rather do it in a method call. Is there a way to force JsonDocument's GetProperty call to drill down into the hierarchy to find the first/any instance of the property name in question (or another method I'm not aware of)?
Thanks!

How to return an array of JSON objects rather the a collection of rows using JOOQ & PostgreSQL

Having read this post suggesting that it's sometimes a good trade-off to use JSON operators to return JSON directly from the database; I'm exploring this idea using PostgreSQL and JOOQ.
I'm able to write SQL queries returning a JSON array of JSON objects rather than rows using the following pattern:
select jsonb_pretty(array_to_json(array_agg(row_to_json(r)))::jsonb)
from (
select [subquery]
) as r;
However, I failed to translate this pattern in JOOQ.
Any help regarding how to translate a collection of rows (fields being of "usual" SQL type or already mapped as json(b)) using JOOQ would be appreciated.
SQL Server FOR JSON semantics
That's precisely what the SQL Server FOR JSON clause does, which jOOQ supports and which it can emulate for you on other dialects as well:
ctx.select(T.A, T.B)
.from(T)
.forJSON().path()
.fetch();
PostgreSQL native functionality
If you prefer using native functions directly, you will have to do with plain SQL templating for now, as some of these functions aren't yet supported by jOOQ, including:
JSONB_PRETTY (no plans of supporting it yet)
ARRAY_TO_JSON (https://github.com/jOOQ/jOOQ/issues/12841)
ROW_TO_JSON (https://github.com/jOOQ/jOOQ/issues/10685)
It seems quite simple to write a utility that does precisely this:
public static ResultQuery<Record1<JSONB>> json(Select<?> subquery) {
return ctx
.select(field(
"jsonb_pretty(array_to_json(array_agg(row_to_json(r)))::jsonb)",
JSONB.class
))
.from(subquery.asTable("r"))
}
And now, you can execute any query in your desired form:
JSONB result = ctx.fetchValue(json(select(T.A, T.B).from(T)));
Converting between PG arrays and JSON arrays
A note on performance. It seems that you're converting between data types a bit often. Specifically, I'd suggest you avoid aggregating a PostgreSQL array and turning that into a JSON array, but to use JSONB_AGG() directly. I haven't tested this, but it seems to me that the extra data structure seems unnecessary.

Postgres 11 Jsonpath support

I am evaluating whether Postgres would be suited to parsing a number of quite complicated JSON documents to extract and semantically match entities, eventually filling a relational schema with high integrity.
I have found the use of jsonpath to be very helpful working with these documents and found this article which suggested that Postgres 11 would have support of sorts. However, the docs do not mention this at all
My question is then will support be forthcoming? Also, is this kind of processing suited to Postgres at all? (possibly use Lucene, MongoDb for the parsing and matching then importing into Postgres relational tables somehow?)
An example of the data could be:
```
{
"event_classes": [
{
"name": "American Football",
"url": "/sportsapi/v2/american-football",
"id": 27
},
{
"name": "Athletics",
"url": "/sportsapi/v2/athletics",
"id": 48
},
{
"name": "Aussie Rules",
"url": "/sportsapi/v2/aussie-rules",
"id": 10000062
},
{
"name": "Badminton",
"url": "/sportsapi/v2/badminton",
"id": 10000069
},
{
"name": "Baseball",
"url": "/sportsapi/v2/baseball",
"id": 5000026
}
]
}
```
SQL/JSON support didn't make it in v11.
It is available from PostgreSQL v12 on.
Your use case is a little vague, but I think that PostgreSQL would be well suited for this kind of processing, particularly if the data should end up in a relational schema.
As a general point on SQL/Path will be a more terse method to query a JSONB data-structure. It will compile down to traditional methods of querying JSONB. That makes it a parser feature, providing a standard syntax.
Imho, that standard syntax is substantially better and gives room for future optimizations, however any query on JSON can be done with the PostgreSQL operators you've linked to, it's just not always pretty.
Finding out if that array contains {"foo":2} is simple.
WITH t(jsonb) AS ( VALUES ('[{"foo":2, "qux":42},{"bar":2},{"baz":4}]'::jsonb) )
SELECT *
FROM t
WHERE jsonb #> '[{"foo":2}]';
However, it's substantially harder to do get the value of qux given the above.
WITH t(jsonb) AS ( VALUES ('[{"foo":2, "qux":42},{"bar":2},{"baz":4}]'::jsonb) )
SELECT e->'qux'
FROM t
CROSS JOIN LATERAL jsonb_array_elements(jsonb) AS a(e)
WHERE t.jsonb #> '[{"foo":2}]'
AND e #> '{"foo":2}';
But, that's not the end of the world. That's actually a really nice SQL syntax. It's just not JavaScript. With JSON PATH you'll be able to do something,
SELECT t.json _ '$.[#.foo == 2].qux'
FROM t
WHERE t.json _ '$.[#.foo == 2]';
Where _ is some kind of JSONPATH operator. As an aside, you can always create an actual JavaScript stored procedure on the server and run it with node. It's really dirt simple with pl/v8.

How to bind dynamic JSON objects to PostgreSQL, using mongodb_fdw?

The Foreign Data Wrapper for MongoDB is pretty awesome! I've gotten it to work using these instructions, apart from:
an object with dynamic fields within it - which PostgreSQL type to use for such?
{
"key1": some,
...
}
an array of objects - which PostgreSQL type to use for such? The length of the array may vary, but the objects are uniform in their inner structure.
[ { "a": 1 }, { "a": 2 }, { "a": 3 } ]
I found these slides on JSON capabilities in recent PostgreSQL versions. Neat. But BSON, JSON or JSONB don't seem to be recognized by the FDW as SQL data types.
If I use:
CREATE FOREIGN TABLE t6
(
"aaa.bbb" JSON -- 'bbb' is an array of JSON objects
)
SERVER mongo_server OPTIONS(...);
SELECT "aaa.bbb" AS bbb FROM t6;
I get:
psql:6.sql:152: ERROR: cannot convert bson type to column type
HINT: Column type: 114
The normal types TEXT, FLOAT etc. work.
The EnterpriseDB fork does it, as #pozs was pointing out. Just mark your data as JSON type.
However, the build system is rather bizarre to my taste, and does not really give you right errors for missing build components (it's obviously Linux-based and simply expects you to have a bunch of tools without properly checking for them).
Here's how I managed to build it on OS X + Homebrew:
$ brew install libtool libbson autoconf automake
$ ./autogen.sh --with-legacy
Note that the --with-meta variant does not provide JSON support, which was the reason I went for this fork anyways.
ref. https://github.com/EnterpriseDB/mongo_fdw/issues/20

partial update json field in postgres

In Postgres I have a table like this:
CREATE TABLE storehouse
(
user_id bigint NOT NULL,
capacity integer NOT NULL,
storehouse json NOT NULL,
last_modified timestamp without time zone NOT NULL,
CONSTRAINT storehouse_pkey PRIMARY KEY (user_id)
)
And storehouse.storehouse is storing data like this:
{
"slots":[
{
"slot" : 1,
"id" : 938
},
{
"slot" : 2,
"id" : 127
},
]
}
The thing is, I want to update storehouse.storehouse.slots[2], but I do not have an idea on how to do it.
I know how to alter the entire storehouse.storehouse field, but I am wondering since Postgres supports json type, it should support partial modify, otherwise that would be no difference between json type and text type. (I know json type also has type validation which is differ to text)
JSON indexing and partial updates are not currently supported. The JSON support in PostgreSQL 9.2 is rudimentary, limited to validating JSON and to converting rows and arrays to JSON. Internally, json is indeed pretty much just text.
There's ongoing work for enhancements like partial updates,indexing, etc. No matter what, though, PostgreSQL won't be able to avoid rewriting the whole row when part of a JSON value changes, because that's inherent to the MVCC model of concurrency. The only way to make that possible would be to split JSON values out into multiple tuples in a side relation, like TOAST tables - something that's possible, but likely to perform poorly and that's very far from being considered at this point.
As Chris Travers points out, you can use PL/V8 functions or functions in other languages with json support like Perl or Python to extract values, then create expression indexes on those functions.
Since PostgreSQL 9.5, there a function called jsonb_set which takes as input parameters:
a JSON object
an array indicating the path (keys and subkeys)
the new value to be stored (also a JSON object)
Example:
# SELECT jsonb_set('{"name": "James", "contact": {"phone": "01234 567890", "fax": "01987 543210"}}'::jsonb,
'{contact,phone}',
'"07900 112233"'::jsonb);
jsonb_replace
--------------------------------------------------------------------------------
{"name": "James", "contact": {"fax": "01987 543210", "phone": "07900 112233"}}
(1 row)