Querying a MySQL table with a JSON field and accessing JSON attributes - mysql

With MySQL 5.7 new features involving JSON has emerged. Among these features is the ability to query the fields in the JSON object as it is stored in the database.
My object looks like this.
{
"color": [
{"WHITE" :{ "size": [
{"S": [{"Price" : "31"},
{"discountPrice" : "13" }]}
]}},
{"BLACK" :{ "size": [
{"S": "69"},
{"M": "31"},
{"L": "55.666"}
]}}
]}
I want to query this as if it was regular tabular data, to this end I tried the following query to no avail.
select json_extract(Sku, '$.color[0]') from CRAWL.DAILYDATA;
I want to explode this into a format that looks more like a traditional RDBMS.
Any ideas?

In order to get data out of a json object as values, you need to get all the way down to the values. For instance, if you wanted to pull all of the values like they are regular RDBMS columns:
select json_extract(Sku, '$.color[0].WHITE.size[0].S[0].price') as price,
json_extract(Sku, '$.color[0].WHITE.size[0].S[0].discountPrice') as discountPrice
from CRAWL.DAILYDATA;
Of course, you need to know exactly what you're looking for in the object. This is the price of having a schema-less object like json. In principle, you could define a mysql function that would use combinations of
json_contains_path
and
json_extract
to make sure the path you are looking for exists, and otherwise it returns null. Personally though, if you want the RDBMS quality, why not just force it into a form where you can put the values directly into mysql tables? This is, of course, why RDBMS's exist. If you can't put it into such a form, you're going to be stuck with searching your json as above.

Related

Get Everything before and after a character/pattern in a string

I have a column from which i want to extract everything before and after a string. I have the following entry:
[{"model": "test.question", "pk": 123456789, "fields": {"status": "graded"}}]
[{"model": "test.question", "pk": 123456789, "fields": {"status": "answered"}}]
I want to extract the substring after "status": {" and before }}]
SQL's LIKE keyword will let you use % as a wildcard. The rest is just straight text matching. So, you should be able to use something like
WHERE columnName LIKE 'status":{"%}}]
(replace columnName with your column, of course).
However, if you have structured data in a table, you might want to reconsider your options. MySQL has a JSON data type (see https://dev.mysql.com/doc/refman/5.7/en/json.html) which may let you query more directly and correctly - for example, the approach I've described above will break if somehow a status exists that includes the string }}].
If you want the substring itself, MySQL has a substring function, detailed at MySQL has a SUBSTRING function, detailed at https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_substring , which you would use in your SELECT clause, probably using LOCATE to get the index to use in the substring. Without seeing your current SQL, it's tough to describe how that would have to be put together.

PostgreSQL json search

I'am thinking about storing some data in postgres jsonb data type. There would be a structure like
{"name": "Jhon Smith", "emails": ["smith#test.com", "agent#matrix.net"],
"phones": ["123456789", "987654321"]}.
I know, that i can search this structure like
where contact->'emails' #> '"smith#test.com"'::jsonb;
But what I need is to search my data with some LIKE operator, so
where contact->'emails' <SOME_JSON_LIKE_OPERATOR> "smith"';
I can't find if psql have something similar, maybe it does not. So, maybe I can convert contact->'emails' field to Text ('["smith#test.com", "agent#matrix.net"]') and then use simple LIKE.. How would you have solved this problem?
You can expand the json array into a recordset of text and search that in whatever manner you like:
where exists (
select 1 from json_array_elements_text(contact->'emails')
where
value like "%smith%"
)

Querying JSON Strings in AWS Redshift

I have a field varchar(65000) column in my AWS Redshift database which is used to store JSON strings. The JSON key/value pairs change frequently and I need to be able to run a daily report to retrieve all key/value data from the column.
For example:
create table test.json(json varchar(65000));
insert into test.json
select '{"animal_id": 1, "name": "harry", "animal_type": "cat", "age": 2, "location": "oakland"}' union
select '{"animal_id": 2, "name": "louie","animal_type": "dog", "age": 4}' union
select '{"animal_id": 3, "gender": "female"}' union
select '{"animal_id": 4, "size": "large"}' ;
With the above data I can write the below query to get the attributes I know are there however if a new attribute is added tomorrow, my report query will not pick up that new key/value pair. Is there any way to do a SELECT * type query on this table?
SELECT
json_extract_path_text(JSON,'animal_id') animal_id,
json_extract_path_text(JSON,'name') name,
json_extract_path_text(JSON,'animal_type') animal_type,
json_extract_path_text(JSON,'location') location,
json_extract_path_text(JSON,'age') age,
json_extract_path_text(JSON,'gender') gender,
json_extract_path_text(JSON,'size') size
FROM test.json
ORDER BY animal_id;
It is not possible to do what you want using your current schema with plain SQL.
If you can have application logic when creating your SQL query, you could dynamically create the SELECT statement.
Option A
Load the whole JSON in your app, parse it and obtain the required information this way.
Option B
When storing values in your database, parse the JSON object and add the discovered keys to another table. When querying your Redshift cluster, load this list of values and generate the appropriate SQL statement using this information.
Here's hoping these workarounds can be applied to your situation.

PostgreSQL: How to check if any of the elements in a JSON array match criteria

Assuming the following schema in PostgreSQL 9.3:
create table test (data json);
insert into test values ('{
"id" : 1,
"name" : "abc",
"values" : [{"id" : 2, "name" : "cde"}, {"id" : 3, "name" : "def"}]
}'::json);
insert into test values ('{
"id" : 4,
"name" : "efg",
"values" : [{"id" : 5, "name" : "fgh"}, {"id" : 6, "name" : "ghi"}]
}'::json);
What is the best way to query for documents where at least one of the objects in the "values" array satisfy a criteria? The only way I could come up with is this:
select data
from (select
data,
json_array_elements(data->'values')->>'name' as valueName
from test) a
where valueName = 'ghi';
Is there a way to do this without the nested query? In MongoDB I could simply say:
db.test.find({values : {$elemMatch : {name : "ghi"}}});
Well... you could do something like this if you prefer subqueries:
select value
from (
select json_array_elements(data -> 'values')
from test
) s(value)
where value ->> 'name' = 'ghi'
But beyond that there is no function available to do what you want. You could easily create your own operator or stored procedure to take care of this however.
Here's a fiddle btw: http://sqlfiddle.com/#!15/fb529/32
PostgreSQL is a object-relational database system, MongoDB is a NoSQL database, so there is no kind of similarity among them (except for the obvious part that they are both used to store data).
You cannot do your search without the nested query. You can create a postgres function or custom type (or both) to aid you, but still, internally a nested query is needed.
It is very important to understand that postgres structured data columns like json are not meant to be used this way. It is mostly a shortcut and a utility to convert data when inserting or selecting, so that your program that executes the sql queries will not need to do extra conversions.
However, you shouldn't be searching in the fields of structured data like this. It is extremely inefficient, you are putting a big load on your server for no reason and you can't use indexes as efficiently (corrected after Igor's comment). Your database will become unusable with a few thousand rows. (I could say more....). I strongly suggest you take the time rearrange your data in more columns and tables so you can easily select from them with the use of indexes and without nested queries. Then you can easily use to_json() to get your data in json format.
EDIT/NOTE:
This answer was written when the current version of Postgres was 9.3 and applies to 9.3 and prior. It is almost certain that Postgres will be able to completely support document store with fully indexed and efficient search in elements, in the (near) future. Each upgrade since 9.0 has been a step to that direction.

MySQL increment value in a text field

Say I have a text field with JSON data like this:
{
"id": {
"name": "value",
"votes": 0
}
}
Is there a way to write a query which would find id and then would increment votes value?
I know i could just retrieve the JSON data update what I need and reinsert updated version, but i wonder is there a way to do this without running two queries?
UPDATE `sometable`
SET `somefield` = JSON_REPLACE(`somefield`, '$.id.votes', JSON_EXTRACT(`somefield` , '$.id.votes')+1)
WHERE ...
Edit
As of MySQL 5.7.8, MySQL supports a native JSON data type that enables efficient access to data in JSON documents.
JSON_EXTRACT will allow you to access a particular JSON element in a JSON field, while JSON_REPLACE will allow you to update it.
To specify the JSON element you wish to access, use a string with the format
'$.[top element].[sub element].[...]'
So in your case, to access id.votes, use the string '$.id.votes'.
The SQL code above demonstrates putting all this together to increment the value of a JSON field by 1.
I think for a task like this you're stuck using a plain old SELECT followed by an UPDATE (after you parse the JSON, increment the value you want, and then serialize the JSON back).
You should wrap these operations in a single transaction, and if you're using InnoDB then you might also consider using SELECT ... FOR UPDATE : http://dev.mysql.com/doc/refman/5.0/en/innodb-locking-reads.html
This is sort of a tangent, but I thought I'd also mention that this is the type of operation that a NoSQL database like MongoDB is quite good at.