Latest release of PostgreSQL have capabilities to work like document oriented databases (e.g. MongoDB). There Is promising benchmarks that says postgres x times faster then mongo. Can someone give me advice how to work with postgres as with MongoDB. I'm seeking for step by step simple example concerned on
1) How to create simpliest table that contain JSON/JSONB objects like documents in mongodb
2) How to make search on it at least by id, like I can do in mongodb with collection.find({id: 'objectId'}) for example
3) How to create new object or overwrite existing at least by id, like I can do in mongodb with
collection.update(
{id: objectId},
{$set: someSetObject, $unset: someUnsetObject}
{upsert: true, w: 1}
)
4) How to delete object if it exists at leas by id, like I can do in mongodb with collection.remove({id: 'objectId'})
It's too large topic to be covered in one answer. So there is just some examples as requested. For more information see documentation:
8.14. JSON Types
9.15. JSON Functions and Operators
Create table:
create table test(
id serial primary key,
data jsonb);
Search by id:
select * from test where id = 1;
Search by json value:
select * from test where data->>'a' = '1';
Insert and update data:
insert into test(id, data) values (1, '{"a": 1, "b": 2, "c": 3}');
update test set data = data - 'a' || '{"c": 5}' where id = 1;
Delete data by id:
delete from test where id = 1;
Delete data by json value:
delete from test where data->>'b' = '2';
Related
In my table I have a JSON field that has an object with key values where some values are arrays. Verifying if a single key exists or even if the key exists with a specified value is easy enough using JSON_CONTAINS_PATH and JSON_EXTRACT respectively, however I need to go one or two steps further.
I need to find an array element (an object) that has a specified key and value while at the same time checking if another specified key doesn't exist or if it exists and has a falsly value.
I've been trying combinations of JSON_CONTAINS_PATH and JSON_EXTRACT and JSON_CONTAINS
For example:
mysql> SET #j = '{"data": [{"a": "abc", "b": true}, {"a": "123"}]}';
mysql> SELECT * FROM db.table WHERE JSON_CONTAINS(#j->>"$.data", JSON_OBJECT('a', 'abc', 'b', true))
+--------------+
| RESULT: 1 |
+--------------+
The above works great, but if I try to apply the same to the second object {"a": "123"} it returns 0. 0 would normally be a valid response but our current usage of "b" is EXISTS && TRUE, EXISTS && FALSE and NOT EXISTS. The first two work with the example above, but I cannot find a way to verify if "a" = "abc" && "b" NOT EXISTS with the current JSON functions provided by MySQL
Also, I tried I also tried JSON_CONTAINS(#j->>"$.data", JSON_OBJECT('a', 'abc', 'b', NULL)) to no avail.
There is a single page application which sends POST HTTP requests with payload in JSON:
{"product": { "name": "product A", "quantity": 100 }}
There is a Postgres database which has tables and stored procedures:
create table product {
product_id serial primary key,
name text,
quantity numeric,
description text
}
create function insert_product (product product) returns product as $$
-- This function accepts a product type as an argument
Is there a solution in any language that would sit on a server, handle HTTP requests, call stored procedures and automatically convert JSON objects to proper Postgres row types?
In pseudo-Express.js
app.post('/product', (req, res) =>
db.query('select insert_product($1)', [convertToPostgresPlease(req.body.product)])
What I don't consider solutions:
destructuring JSON object and feeding Postgres every key by the teaspoon
or handling JSON inside stored procedure (SP should accept a row type)
duplicating information from Postgres schema (solution must use Postgres introspection capabilities)
Manually concatenating '(' + product.name + ',' + ...
I know stored procedures are often frowned upon, but for small projects I honestly think they're great. SQL is an amazing DSL for working with data and Postgres is advanced enough to handle any data-related task.
In any case, what is the most simple way to connect JSON HTTP request with a proper SQL RDBMS?
Found solutions (almost):
postgraphql (works, but too much magic)
Chicken Scheme (list-unparser, didn't try yet)
As Abelisto mentioned in the comments, you can convert from JSON/JSONB parameters within a database function to a specific table row, using json_populate_record/jsonb_populate_record. Another alternative is using the json variable directly using the -> and ->> operators to retrieve its contents. The disadvantage of this is that there is a fair amount of coding for the maintenance of each table.
You may also be able to benefit from RESTful interfaces (e.g. https://github.com/QBisConsult/psql-api).
Another option for a heavily JSON-based solution is simplify operations for the bulk of small tables that wouldn't grow beyond a few hundred records each. There would be a performance toll, but for a few rows it would likely be negligible.
The following exemplifies the power of the JSON datatype in PostgreSQL and the GIN indexes which support JSON operators. You can still use normal tables and specialised functions for data that requires maximum performance.
The example:
CREATE TABLE public.jtables (
table_id serial NOT NULL PRIMARY KEY,
table_name text NOT NULL UNIQUE,
fields jsonb
);
INSERT INTO public.jtables VALUES (default, 'product', '{"id": "number", "name": "string", "quantity": "number", "description": "string"}'::jsonb);
CREATE TABLE public.jdata (
table_id int NOT NULL REFERENCES jtables,
data jsonb NOT NULL
);
CREATE UNIQUE INDEX ON public.jdata USING BTREE (table_id, (data->>'id'));
CREATE INDEX ON public.jdata USING GIN (data);
You could create functions to manipulate data in a generic JSON way, e.g.:
CREATE FUNCTION public.jdata_insert(_table text, _data jsonb) RETURNS text AS
$BODY$
INSERT INTO public.jdata
SELECT table_id, $2
FROM public.jtables
WHERE table_name = $1
RETURNING (data)->>'id';
$BODY$ LANGUAGE sql;
CREATE FUNCTION public.jdata_update(_table text, _id text, _data jsonb) RETURNS text AS
$BODY$
UPDATE public.jdata d SET data = jsonb_strip_nulls(d.data || $3)
FROM public.jtables t
WHERE d.table_id = t.table_id AND t.table_name = $1 AND (d.data->>'id') = $2
RETURNING (d.data)->>'id';
$BODY$ LANGUAGE sql;
Rows can then be inserted using these generic functions:
SELECT public.jdata_insert('product', '{"id": 1, "name": "Product 1", "quantity": 10, "description": "no description"}'::jsonb);
SELECT public.jdata_insert('product', '{"id": 2, "name": "Product 2", "quantity": 5}'::jsonb);
SELECT public.jdata_update('product', '1', '{"description": "test product"}'::jsonb);
And their data can be queried in a variety of ways which make use of the existing indexes:
SELECT * FROM public.jdata WHERE table_id = 1 AND (data->>'id') = '1';
SELECT * FROM public.jdata WHERE table_id = 1 AND data #> '{"quantity": 5}'::jsonb;
SELECT * FROM public.jdata WHERE table_id = 1 AND data ? 'description';
Views can make make queries easier:
CREATE VIEW public.vjdata AS
SELECT d.table_id, t.table_name, (d.data->>'id') AS id, d.data
FROM jtables t
JOIN jdata d USING (table_id);
CREATE OR REPLACE FUNCTION public.vjdata_upsert() RETURNS trigger AS $$
BEGIN
IF TG_OP = 'INSERT' THEN
PERFORM public.jdata_insert(NEW.table_name, NEW.data);
ELSE
PERFORM public.jdata_update(NEW.table_name, NEW.id, NEW.data);
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER vjdata_upsert_trigger INSTEAD OF INSERT OR UPDATE
ON public.vjdata FOR EACH ROW EXECUTE PROCEDURE public.vjdata_upsert();
UPDATE public.vjdata SET
data = data || jsonb_build_object('quantity', (data->>'quantity')::int + 2)
WHERE table_name = 'product' AND id = '2'
SELECT * FROM public.vjdata WHERE table_name = 'product' AND id = '2';
Check out PostgREST? BTW, I don't know why anyone would frown on stored procs. The correct way to interact with the DB is through views and functions/procs. Having SQL in plain code is something that just happened over the last 15 years, really due simply to convenience and loss of SQL skills. It's harder for most people to do set operations than procedural processing.
I have found an answer here on how I can specify an AUTO_INCREMENT on a secondary column in a multiple column index in MySQL e.g.
CREATE TABLE foo (
id INTEGER NOT NULL AUTO_INCREMENT,
grp INTEGER NOT NULL ,
name VARCHAR(64),
PRIMARY KEY (id, grp)
) ENGINE = MyISAM;
How would I extend a slick.driver.MySQL.api.Table so that such a table is generated, if indeed it is at all possible? Difficulties I am having currently: (1) I don't know how to create a composite primary key in slick within the main create statement and (2) I don't know how to specify to use the MyISAM engine.
Update: Following #ulas' advice, I used slick.codegen to generate the slick data model from the (already created) SQL table. However, the data model cannot be used to then recreate the table - it generates two statements instead of one, and neither reference MyISAM. Regarding this, I have listed an issue in the slick github repos.
For now this leaves me with following #RickJames' advice, which I would rather do anyway since it doesn't rely on MyISAM, a non-default engine for the current version of MySQL.
So my question can now be collapsed to, how would I execute the following using slick?
BEGIN;
SELECT #id := IFNULL(MAX(id), -1) + 1 FROM foo WHERE grp = 1 FOR UPDATE;
INSERT INTO foo VALUES (#id, 1, 'bar');
COMMIT;
I have no idea how to do it using the 'higher-level' abstraction, so I tried following the Plain SQL Queries section of the slick manual. My attempt went something like:
val statements = DBIO.seq(
sqlu" SELECT #id := IFNULL(MAX(id), -1) + 1 FROM foo WHERE grp = 1 FOR UPDATE",
sqlu"INSERT INTO foo VALUES (#id, 1, 'bar')"
)
db.run(statements.transactionally)
But I got the error:
Exception in thread "main" slick.SlickException: Update statements should not return a ResultSet
Help appreciated.
I'm getting this error when trying to access data in a JSON object, does anybody know what it is causing it?
This is the query:
SELECT id, data FROM cities WHERE data->'location'->>'population' = '270816'
This is the JSON object:
location": {
"population": 270816,
"type": "city"
}
Any help would be really appreciate it. Thanks
I was able to get this SELECT to work in Postgres 9.3.1. Here's an sqlfiddle which illustrates that.
Here is the DDL and INSERT statement I used in the sqlfiddle:
create table cities
(
id serial,
data json
);
insert into cities (data) VALUES
('{
"location": {
"population": 270816,
"type": "city"
}
}'
);
What version of Postgres are you using? How are you inserting the JSON? What's the DDL for your cities table?
It suspect it may be an issue with the way you are inserting the JSON data. Try inserting it similar to the way I am doing in the sqlfiddle above and see if that works for you. i.e. as a pure SQL string, but one with valid JSON inside, into a column defined as json.
Just had what sounds like the same issue on Postgres 9.6.6. Improper string escaping caused mysterious JSONB behavior. Using pgAdmin4,
CREATE TABLE json_test (json_data JSONB);
INSERT INTO json_test (json_data) VALUES ('"{\"id\": \"test1\"}"');
INSERT INTO json_test (json_data) VALUES ('{"id": "test2"}');
SELECT json_data, json_data->>'id' as id FROM json_test;
returns the following pgAdmin4 output showing baffling failure to find id test2. Turns out the pgAdmin4 display is misleading. Situation becomes clear using text display from PSQL:
db=> CREATE TABLE json_test (json_data JSONB);
CREATE TABLE
db=> INSERT INTO json_test (json_data) VALUES ('"{\"id\": \"test1\"}"');
INSERT 0 1
db=> INSERT INTO json_test (json_data) VALUES ('{"id": "test2"}');
INSERT 0 1
db=> SELECT json_data, json_data->>'id' as id FROM json_test;
json_data | id
-----------------------+-------
"{\"id\": \"test1\"}" |
{"id": "test2"} | test2
(2 rows)
Where it is obvious that the first row was inserted as a string which just looks like JSON, not as a nested JSON object.
Does PostgreSQL provide any notation/method for putting a constraint on each element of a JSON array?
An example:
create table orders(data json);
insert into orders values ('
{
"order_id": 45,
"products": [
{
"product_id": 1,
"name": "Book"
},
{
"product_id": 2,
"name": "Painting"
}
]
}
');
I can easily add a constraint on the order_id field:
alter table orders add check ((data->>'order_id')::integer >= 1);
Now I need to do the same with product_id. I can put constraint on idividual array items:
alter table orders add check ((data->'products'->0->>'product_id')::integer >= 1);
alter table orders add check ((data->'products'->1->>'product_id')::integer >= 1);
-- etc.
So obviously what I'm looking for is some kind of wildcard operator for matching any JSON array element:
alter table orders add check ((data->'products'->*->>'product_id')::integer >= 1);
-- ^ like this
I know that this can be done by extracting products to a separate products table with a foreign key to orders. But I want to know if this is possible within single JSON column, so I can keep that in mind when designing a database schema.
So I asked this question on PostgreSQL mailing list, as suggested by Craig Ringer, and I've got the answer.
In short the solution is to write a procedure which materializes JSON array to PostgreSQL array:
create function data_product_ids(JSON) returns integer[] immutable as $$
select array_agg((a->>'product_id')::integer) from
json_array_elements($1->'products') as a $$ language sql ;
and use that procedure in CHECK statment:
alter table orders add check (1 <= ALL(data_product_ids(data)));
For more details on how this works se the answer on PostgreSQL mailing list. Credits to Joel Hoffman.
From one of the developers of JSON for Postgres
The path stuff does not support wildcards.