Bookshelf: query builder: how to use like operator with json column? - json

I'm using bookshelf with postgresql database
Information is a column of type json.
I want to retrieve all column that are like '%pattern%'
With sql query i use
select * from table where information::text like '%pattern%';
I want to do that with bookshelf query builder
model.query(function(qb) {
qb.where('information', 'LIKE', '%pattern%')
}).fetch()
But it didn't work and i can't find how to do it in bookshelf docs
Any idea?

The tricky part here is, although you might think that JSON (and JSONB) columns are text, they aren't! So there's no way to do a LIKE comparison on one. Well, there is, but you'd have to convert it to a string first:
SELECT * FROM wombats WHERE information #>> '{}' LIKE '%pattern%';
which is a really terrible idea, please don't do that! As #GMB points out in the comments, JSON is a structured format that is far more powerful. Postgres is great at handling JSON, so just ask it for what you need. Let's say your value is in a JSON property named description:
SELECT * FROM wombats
WHERE (information->'description')::TEXT
LIKE '%pattern%';
Here, even though we've identified the correct property in our JSON object, it comes out as type JSON: we still have to cast it to ::TEXT before comparing it with a string using LIKE. The Bookshelf/Knex version of all this would look like:
model
.query(function(qb) {
const keyword = "pattern";
qb.whereRaw(`(information->'description')::TEXT LIKE '%${keyword}%'`)
})
.fetch();
Apparently this part of the raw query cannot be parameterized (in Postgres, at least) so the string substitution in JavaScript is required. This means you should be extra careful with where that string comes from (ie only use a limited subset, or sanitise before use) as you're bypassing Knex's usual protections.

Related

SQL replace all specified keys

I have one column(varchar) containing only json string within one table. I want replace all keys with "" on that column. How can I do that using sql? My database is MySQL.
For example:
|--------------------------------------------------------------------|
| t_column |
|--------------------------------------------------------------------|
| {"name":"mike","email":"xxx#example.com","isManage":false,"age":22}|
|--------------------------------------------------------------------|
SELECT replace(t_column, regexp, "") FROM t_table
I expect:
mikexxx#example.comfalse22
How to write that regexp?
Start from
select t_column->'$.*' from test
This will return a JSON array of attribute values:
[22, "mike", "xxx#example.com", false]
This might be already all you need, and you can try something like
select *
from test
where t_column->'$.*' like '%mike%';
Unfortunately there seems to be no native way to join array values to a single string like JSON_ARRAY_CONCAT(). In MySQL 8.0 you can try REGEXP_REPLACE() and strip all JSON characters:
select regexp_replace(t_column->'$.*', '[" ,\\[\\]]', '') from test
which will return '22mikexxx#example.comfalse'.
If the values can contain one of those characters, they will also be removed.
Note: That isn't very reliable. But it's all I can do in a "simple" way.
See demo on db-fiddle.
I could be making it too simplistic, but this is just a mockup based on your comment. I can formalize it into a query if it fits your requirement.
Let's say you get your JSON string to this format where you replace all the double quotes and curly brackets and then add a comma at the end. After playing with replace and concat_ws, you are now left with:
name:mike,email:xxx#example.com,isManage:false,age:22,
With this format, every value is now preceded by a semicolon and followed by a comma, which is not true for the key. Let's say you now want to see if this JSON string has the value "mike" in it. This, you could achieve using
select * from your_table where json_col like '%:mike,%';
If you really want to solve the problem with your approach then the question becomes
What is the regex that selects all the undesired text from the string {"name":"mike","email":"xxx#example.com","isManage":false,"age":22} ?
Then the answer would be: {\"name\":\"|\"email\":\"|\",\"isManage\":|,\"age\":|}
But as others let you notice I would actually approach the problem parsing JSONs. Look up for functions json_value and json_query
Hope I helped
PS: Keep close attention on how I structured the bolded sentence. Any difference changes the problem.
EDIT:
If you want a more generic expression, something like select all the text that is not a value on a json-formatted string, you can use this one:
{|",|"\w+\":|"|,|}

MYSQL REGEXP with JSON array

I have an JSON string stored in the database and I need to SQL COUNT based on the WHERE condition that is in the JSON string. I need it to work on the MYSQL 5.5.
The only solution that I found and could work is to use the REGEXP function in the SQL query.
Here is my JSON string stored in the custom_data column:
{"language_display":["1","2","3"],"quantity":1500,"meta_display:":["1","2","3"]}
https://regex101.com/r/G8gfzj/1
I now need to create a SQL sentence:
SELECT COUNT(..) WHERE custom_data REGEXP '[HELP_HERE]'
The condition that I look for is that the language_display has to be either 1, 2 or 3... or whatever value I will define when I create the SQL sentence.
So far I came here with the REGEX expression, but it does not work:
(?:\"language_display\":\[(?:"1")\])
Where 1 is replaced with the value that I look for. I could in general look also for "1" (with quotes), but it will also be found in the meta_display array, that will have different values.
I am not good with REGEX! Any suggestions?
I used the following regex to get matches on your test string
\"language_display\":\[(:?\"[0-9]\"\,)*?\"3\"(:?\,\"[0-9]\")*?\]
https://regex101.com/ is a free online regex tester, it seems to work great. Start small and work big.
Sorry it doesn't work for you. It must be failing on the non greedy '*?' perhaps try without the '?'
Have a look at how to serialize this data, with an eye to serializing the language display fields.
How to store a list in a column of a database table
Even if you were to get your idea working it will be slow as fvck. Better off to process through each row once and generate something more easily searched via sql. Even a field containing the comma separated list would be better.

MySQL 5.7 - Query to set the value of a JSON key to a JSON Object

Using MySQL 5.7, how to set the value of a JSON key in a JSON column to a JSON object rather than a string.
I used this query:
SELECT json_set(profile, '$.twitter', '{"key1":"val1", "key2":"val2"}')
from account WHERE id=2
Output:
{"twitter": "{\"key1\":\"val1\", \"key2\":\"val2\"}", "facebook": "value", "googleplus": "google_val"}
But it seems like it considers it as a string since the output escapes the JSON characters in it. Is it possible to do that without using JSON_OBJECT()?
There's a couple of options that I know of:
Use the JSON_UNQUOTE function to unquote the output (ie not cast it to string) as documented here
Possibly use the ->> operator and select a specific path, documented here
Has a lot of implications, but you could disable backslashes as an escape character. I haven't tried this, so I don't even know if that works, but it's mentioned in the docs
On balance, I'd either use the ->> operator, or handle the conversion on the client side, depending on what you want to do.

Dynamic field name query using N1QL

I'm having a use case here which I can't seem to solve. Basically, I need to create a webservice where users may query the couchbase cluster "dynamically". Indeed, i'm storing metadata of different files, and the "creation" of this metadata is up to the user : I don't have specific fields in my Java POJO, i'm inserting a MAP which gets inserted as a nested object in couchbase.
Now the query I need is pretty simple on paper and looks something like this :
#Query("#{#n1ql.selectEntity} WHERE #{#n1ql.filter} AND $1 = $2")
List<FileMetadata> findListMetadata(String pKey, String pValue);
But it doesn't seem to work, $1 doesn't seem to ever get replaced by the pKey variable.
I'm using CouchBase 4.5 with the Spring Data connector.
Any ideas on how to solve that use case ?
You need to dynamically generate the query string, so that pKey is inserted into the query string and pValue is passed as a parameter (as you are doing).

Search in mysql database - unserialized data

Situation:
I have user model. attribute "meta_data" in db represents "text" type field.
In model it seriazized by custom class. ( serialize :meta_data, CustomJsonSerializer.new )
It means, when I have an instance of user, I can work with meta_data like with Hash.
User.first.meta_data['username']
Problem:
I need to write a search function, which will search users by given string. I can do it by manual building search query in rails ex. User.where("email LIKE '%#{string}%'")...
But what about meta_data ? Should I search in this field by LIKE statement too? If I will do so, it will decrease relevance of found record.
For example:
I have 2 users. One of them has username "patrick", another one is "sergio"
meta data in db will look like this:
1) {username: patrick}
2) {username: sergio}
I want to find sergio , I enter a search string "ser" => but I have 2 results, instead of one. This meta_data string "{uSERname: Patrick}" also has "ser", so it makes this record irrelevant.
Do you have any idea how to solve it?
That's really the problem with serialized data. In theory, the serialization could be an algorithm that is very unsearchable. It could do a Hoffman encoding, or other compression, and store the serialization in binary. You are relying on the assumption that the serialization uses JSON and your string will still be findable as a sub-string in the serialization.
Then the problem you are having is another issue. Other data in the serialization can mess up your results.
In general, if you serialize data, you are making a choice to not be searchable.
So a solution would be to add an additional field that you populate in a way that you control. Have a values field and store a pipe (|) delimited value that you can search. So if the data is {firstname: "Patrick", lastname: "Stern"}, your meta_values field might be "Patrick|Stern".
Also, don't use the where method with a string with #{} expansion of input values. The makes it vulnerable to SQL attacks. Instead use:
where("meta_values is like :pattern", pattern: "%#{string}%")
I know that may not look very different, but ActiveRecord will go through a sanitizing this way. If someone has a semi-colon in string, then ActiveRecord will escape the semi-colon in the search condition.